Page tested for compliance with INDUSTRY (not MS-only) standards, using the free, publicly accessible validator at validator.w3.org
Bookmark this on Delicious
Recommend to StumbleUpon
You can write to and read from SD memory cards with the datalogging shield for Arduinos from nuelectronics.com (£11, c. $16 at 7/10).
While you do not have all the features of a generalized SD card reader/ writer, you can do useful memory card- based work. You can write data, as logged by your Arduino + Shield, to a card. Then you can remove the card, plug it into a "big computer", and read all of the data logged. Not bad! (I.e., you have a "sneakernet", if you want one.)
If you are disappointed when you learn the limitations mentioned below, remember that if you want the system to do more, you have to give up the simplicity (compared to what fancier features would require!) and small size of the current provisions. More is not always better!
In matters of writing to SD cards I am very much a novice. I hope some experienced SD card users will scan this, particularly the "Where We're Going" stuff at the bottom, and send comments. I don't need details of the code debugged, but comments on the general ideas here would be very welcome, especially comments like "You're working along the right lines." I'm not even sure of that, at the moment!!
This page is a work in progress. I may be wrong... but SO FAR I'm finding that I DO have to issue a "close file/ unmount card" command to avoid data loss. I'd hoped... probably too much to hope... that I could write to the SD card almost as if it were a piece of paper in an old fashioned typewriter. I've also, I think, encountered a little, unimportant(?), discrepancy between the Petit FAT File System (PFFS) Module as described on it's web page and how it works in the nuelectonics shield, if you use the low level interface/ driver files from nuelectronics. The discrepancy: when you close the file, the unused bytes in the last sector are filled with with "A"s, not the "0"s promised at the PFFS page. (This is a HELP, if the PFFS page is talking about the number 0, vs the character "0".) The code presented below, while rough and possibly imperfect, does TRY to address the issues mentioned here. It SEEMS to work, but hasn't been rigorously tested, and is not robust.
First an overview. If you find any of these pages useful, by the way, I'd be grateful if you clicked on the "StumbleUpon" button at the top of the page. Such "votes" will be part of what determines which pages get further attention. "Vaporware"? I hope not. I hope these pages are already good for something. As good as they could be? Does anything ever become perfect?
In a "big" computer, the SD card will be formatted, (use the "FAT" structure, not "FAT32")
Still on the "big" computer, a file full of spaces (or some other "blanks") will be created and saved on the card, with a specific name.
That card will then be put into the SD card interface on the datalogging shield.
When the Arduino program runs, it will either immediately, or in response to a command, "mount" the card, and "open" the file for random write.
Then, under the control of your Arduino program, new data will be written into the file on the SD card, overwriting whatever was on that part of the file previously. (The system in use here does not allow you to create a new file, or extend an existing one. Annoying, but not crippling.) You would normally overwrite what was there previously from the start of the file forward, but you can write code which lets you jump around. (In technical terms: There is a "seek" command.)
I think you can write to the card one byte at a time. Details to be determined later! If that isn't exactly right, the system will at least allow something along those lines, I think.
I think that if you have a power failure, or remove the card from the socket at a "bad" moment, that you will lose data. I think you need to issue a program command to dismount the card before you physically remove it. (In Windows terms: You need to execute the "safe to remove" procedure which you SHOULD be using with thumb-drives. Floppies spoiled us!)
That's the overview. First, a detail...
The "get_line" function has been modified from what is in the nuelectronics demos because, I think, I needed to do that to get it to work with the Arduino serial monitor. For more details on that, and other general details, see my page about my working environment.
The following is imperfect, but does, seem to work, at least as far as it goes. You have to prepare an SD card on a separate computer. You need to format it with a plain "FAT" file system, not "FAT32". (Windows terms, sorry.) Also, on the big computer, save a big file on the card. That file can be full of anything you like... I use a bunch of periods ("......."). Call the file "MySDfile.txt". (You can call the file what you like, but if you use something other than "MySDfile.txt", remember to use your name everywhere you see me using "MySDfile.txt") You then insert the card in the datalogging shield, run the program, issue the following commands: m,f,s (enter each in the serial monitor, and press enter after each one. (3 "enter"s)). You should get messages after each. If you get an error message, usually repeating the command is sufficient. Then wait about 15 seconds, letting the Arduino "log data".... you should see dots appearing on the serial monitor. (The "data", sadly, is just "123xxxx" over and over again... at this stage.) Press "s" (and enter) again to stop the "datalogging" process. (The first "s" started it, now you are stopping it.) Press "u" (and enter) to "unmount, etc", and the Arduino will write buffered things to the card, close the file, and unmount the card. You can then safely take the card out, and you should find "123xxxxx" written on it many times, followed by "!EOF!". Anything after that is "undefined". You may get a bunch of "A"s, you may get a bunch of "0"s, you may have gibberish left over from previous experiments. (You don't have to restore the file all to "......"s between experiments.) (The file has to be bigger than the amount of data you plan to write). You shouldn't enter "s" to start the "datalogging" until after a successful "m" and "f". The only problem I've had is that once in a while, my first "m" is not successful (an error message arises). I just say "m" again, and I've always been successful on the second try, except the time that I had a FAT32 formatted SD card in the shield.
The code doesn't (yet) do anything useful! :-( When it is "logging data", it is merely writing "123xxxxx" to the SD card over and over again. If you want to log some data, you "only" have to add the necessary code at "//Put ReadRTC and ReadSensor(s) here!", and then replace the "123xxxxx" after the PSTR with what the clock and sensor(s) said.
The code below is, as I said, poor... but does, I think "work". Before I do much more with the code, I need to make some progress with the "Where I am Going" issues covered further down this page.
You need to install the sensor_pff library from the nuelectronics site to use the SD reader/ writer. I've done a tutorial on software libraries in case you don't already know about them.
/*WriteToSD
ver 18Jly10 */
#include <avr/pgmspace.h>
#include <sensor_pff.h>
SENSOR_PFF sd;
byte sensor_start = 0;
char buff[128]; /* i/o buffer */
static void get_line (char *buff, byte len)
{
byte c;
int idx = 0;
for (;;) {//1 Start infinite loop... will only exit via "break"
if(Serial.available()){//2
c = Serial.read();
if (((byte)c >= ' ') && (idx < len - 1)) {
buff[idx++] = c; Serial.print(c);
}//3
}else break;//2 TKB ADDED "else break"
}//1 (ends "for...")
buff[idx] = 0;
//Serial.print(c);
Serial.print('\n');
}
// === END OF GET_LINE ===
void setup()
{
byte res;
delay(300);//let system settle
Serial.begin(9600);
sprintf_P(buff, PSTR("\nTKB Write to SD demo\n"));
Serial.println(buff);
sprintf_P(buff, PSTR("Derived from nulectonics.com prgm 'datalog.pde'\n"));
Serial.println(buff);
sprintf_P(buff, PSTR("See http://sheepdogguides.com/ardino/ar3ne1sd.htm for latest details\n"));
Serial.println(buff);
Serial.print('>');
}
void loop()
{
char *ptr;
long p1, p2;
byte res;
unsigned short w;
if(Serial.available() == FALSE){//1
if(sensor_start == TRUE){//2
//Put ReadSensor(s) here!
/* The following is a scrap of something complex which might work...
if certain sensor and RTC driving software were present, and if calls to
"read sensor", "read RTC" had been made!....
sprintf_P(buff, PSTR("%u/%u/%u, %u, %02u:%02u:%02u,
hum= %02u.%02u, temp = %02u.%02u\n"), rtc.year,
rtc.month, rtc.mday, rtc.wday, rtc.hour, rtc.min,
rtc.sec, hum_int, hum_dec, temp_int, temp_dec);*/
sprintf_P(buff, PSTR("123xxxxx"));//tkb demo: Fill buffer with "data"
res = pf_write(buff, strlen(buff), &w);// Write data to the file
if (res != FR_OK) Serial.println("write err\n");
else Serial.print('.');
delay(500);
}//2
}//1
else
{//no cmnds waiting
get_line(buff, sizeof(buff));
ptr = buff;
switch (*ptr++) {//switch
case 'm': // mount SD card
res = disk_initialize();
if (!res) {//1.. the "fact" that !res true when init succeeded not confirmed
res = pf_mount(&sd.fs);
if (res==0) {//2 Mount DOES return 0 when all well.
sprintf_P(buff, PSTR("SD card mounted OK"));
}
else {
sprintf_P(buff, PSTR("SD card failed to mount"));
};//2
}//1
else {//1
sprintf_P(buff, PSTR("SD Init Failed"));
};//1
Serial.println(buff);
break;//end "m"
case 'f': // open a file. Card must be mounted, first
Serial.print("\nOpening ssd.txt\n");
sprintf_P(buff, PSTR(" MySDfile.txt"));//N.B. Space needed in
//front of file's name. I tried removing it here, and
//removing next line...
while(*ptr == ' ') ptr++;//(see prev).... but no joy! "Broke it"
res = pf_open(ptr);
if(res!=0)
sprintf_P(buff, PSTR("Error opening file on SD"));
else//opened... now put pointer at start of file... this added by TKB
{ res=pf_lseek(0);
if (res==0){
sprintf_P(buff, PSTR("File on SD opened OK"));}
else sprintf_P(buff, PSTR("Error doing initial seek into file"));
}//else
Serial.println(buff);
break;//end "f"
case 's': //start/stop datalog
//needs work... must not be allowed until file opened.
//Also, provide for messages "Logging Started" and "Stopped"
Serial.print("Starting or Stopping Logging");
if(sensor_start == FALSE) sensor_start = TRUE;
else sensor_start = FALSE;
break;//end "s"
case 'u': // finalize writing (flush/ close) and Unmount
//needs work!!
Serial.print("\nClosing file.\n");
sprintf_P(buff, PSTR("!EOF!"));//tkb
res = pf_write(buff, strlen(buff), &w); /* Write data to the file */
if (res != FR_OK) { Serial.println("write err\n"); }
res=pf_write(0,0,&w);
if (res==0){
sprintf_P(buff, PSTR("SD CARD unmounted OK"));}
else sprintf_P(buff, PSTR("SD CARD unmount failed"));
Serial.println(buff);
break;//end "u"
}//end "switch..."
Serial.print('>');
}//else, no serial waiting
}//end "loop()"
So! There's some "working" code. A visit to the Petit FAT File System Module site will help readers who have some file systems programming experience.
Is the code above good? No! But it's a start!
In building software, and the hardward for it to run in, as in life, it always pays to keep in mind where you are going while you are moving. You want to know you are moving forward, towards your goal, after all!
Most of the work I've done so far with the nuelectronics.com datalogging shield, and the sensors and actuators (effectors) for it has been amenable to my usual practice of looking at something in a tightly constrainted context. For instance, for the temperature sensor, I was quite happy with a little program which just read the sensor every 5 seconds, and printed the result out on the serial monitor. The fact that I've worked with many sensors over the years helped, too, of course.
The datalogging shield's ability to write to, (and read from), an SD card is a wonderful thing... but using it will never be as simple as using a temperature sensor!
While my messy software (above) breaks the back of finding out how the pieces of the puzzle work, it is nothing like a "good answer". Before we can move towards that "good answer", we need a plan, and my plan is: Build a "box" with the following. Some of the things on the box could be conflated. For instance, the "mount, etc" button could do double duty as a "unmount, etc" button. Maybe a second generation "box" might be built that used fewer buttons and LEDs. But for getting started, I want all of the following:
Here's how using the box should go. Besides the "messages" implied by the state of the LEDs, the box should also send more verbose messages to the serial monitor, but it should not be necessary for the user to see those messages in order to use the box, and no input will be via the serial monitor. (There will even be times when the box will not be plugged into the big computer while the box is running. Happily, this will not cause a problem, with our box or in general... messages sent with Serial.print do not have to be acknowledged. With Serial.print, the Arduino "speaks", but doesn't care if "no one is listening".)
Switch on box. Card can be in or not. No LEDs should be on after the Arduino boots up.
The SD card should have a file on it, bigger than the biggest you want to write. It should be called "MySDfile.txt". You may want to make the file be full of some character or pattern, so changes are easier to see, but the system really won't care.
Put SD card into the datalogging shield. Press the "Mount, Etc" button. This should cause the Arduino to...
If everything went according to plan. At each stage, the result returned from the file system software should be checked, and it the process failed, the attempt to "Mount, Etc" should be terminated, and the "Unable" LED turned on. I wouldn't be surprized if this happened from time to time. I would be surprized if, with a good SD card which did have the required file, it would take more that pressing the "Mount, Etc" button again to get success on the second (or maybe third) attempt.
Once the "Mounted and Ready" LED was on, I would press the "Start Logging" button. The "Logging Data" LED should show that data is being logged. (There are variations to consider on this theme!)
For our purposes here, "logging data" would still consist merely of mindlessly writing "123xxxxx" to the file, over and over again... but you would do it in a way that made altering that part of the code easy.
The "Stop Logging" button would be more like a "pause" button than anything else. It would stop the logging, but not do much more. The "Mounted and Ready" LED should stay on. Pressing the "Start Logging" button again should cause the logging of data to resume... not "start again". (Old data should not be lost).
(Any button that should not do anything at a given moment should be ignored, of course. E.g. the "Start Logging" button shouldn't do anything when the system is already logging.)
Once the logging has been stopped, the "Flush buffer/ Unmount" button should cause the box to...
Looking at those in detail....
"So- far- merely- buffered". At some (higher) levels, it may SEEM as if you have "written" something to the SD card when in fact you haven't actually done so yet. The deeper levels of the software may be writing to the SD card in "blocks".
"mark end of data". If the data you were recording consisted entirely of digits, i.e. 0,1,2...7,8,9, then you could write an "E" after the last digit to say "ignore everything beyond here." The way you mark the end of the file depends on the data you are saving, how you are encoding it, how you want to read it. For simple tests: Stick to printable ASCII, and use a text editor to look at what you've achieved. I like Textpad, for all sorts of reasons. Your file will normally be bigger than the part of the file holding meaningful data. The only time this will not be true will be when the file was just big enough to hold what you wanted to save.
The "unmount" doesn't include pushing it physically out of the shield. It is "unmounted" only as far the software is concerned.
A "frill": While the file system can't extend a file, it can determine the size of the file. You really "need" to build into "Mount, Etc" the things that are necessary to read from the SD card the size of the file, and then through everything else that happens, you need to watch that you are not trying to write past the end of the file, and that you are leaving room for your "end of data" marker.
You might choose to build the following into the "Flush, Etc" button: It might be desirable to have the response to that button see if data is currently being logged, and it it was, then the processing of "Flush, Etc" would start with whatever is done when you press "Stop Logging". Or you could just disable the "Flush, Etc" button any time data was being logged.
So! There you have it! That's "all" we need. I hope someone will write that, and post it at the Arduino Playground. After that "little" task is out of the way, the rest of us can get on with the "hard" part... building the rest of a datalogging application, building the system around the "save the data" backbone. The full system will have sensors and ways to input user requirements such as sampling rate.
By the way... story for another day... the system sketched above is not "power failure friendly". If the power to the system fails unexpectedly, or if a user pulls the card out without pressing the "Stop Logging" and "Flush, Etc" buttons first, at least some data will probably be lost. You could make it more robust with a "write at once" system, probably based on frequent "flushs"... but that would give rise to wear and tear on the SD card, and I don't want to "go there"... for now.
Just in closing let me share some helpful, I hope, information that someone from Jackson Ohio USA kindly emailled to me. He was also working with the nuelectronics datalogging shield. He said.... (edited)...
I have successfully written to an SD card, a 2 GB Kingston, FAT formatted. I used the file and material from Adafruit Industries.... www.adafruit.com They also sell a datalogging shield. They recommend the SDFATlib by Mr. Bill Greiman.... http://code.google.com/p/sdfatlib/ You will have to change the #define SPI_SS_PIN from 10 (normal) to 5 for the nuElectronics shield, in the SDcard.H file. He has a lot of good examples, and information in the zip file. All of the examples that I tried worked with the nuElectronics Shield.... Wrote to SD card with no file on card, created a file, appended to it, erased it, time stamped it, copied it. The library is for Fat16 SDs, and there is a library for Fat32 SDs, but not Fat12. He covers the differences in some of his text files and pdf file.
So! I hope you have enough ideas to "play" with, for the moment?
Further to the Arduino ideas the page you are reading now will take you to, I have posted a series of essays which try to help you become a better Arduino programmer and engineer... but, for the best result, you will have to buckle down and work your way through them in sequence. The "How To's" here can be accessed in whatever order you like.
Feel free to use this information in programming courses, etc, but a credit of the source would be appreciated. If you simply copy the pages to other web pages you will do your readers a disservice: Your copies won't stay current. Far better to link to these pages, and then your readers see up-to-date versions. For those who care- thank you- I have posted a page with more information on what copyright waivers I extend, and suggestions for those who wish to put this material on CDs, etc.
See the discussion near the bottom of the "top level" page covering the bulk of my Arduino contributions. There is information there, too, about things like "May I copy your material?", and the system of file names I am trying to work to.
If you visit 1&1's site from here, it helps me. They host my website, and I wouldn't put this link up for them if I wasn't happy with their service... although I was less than pleased the other day to have what I was doing interrupted by a telephone call from their sales team, trying to get me to extend my involvement. Sigh. Hardly a rare event, but I'd thought 1&1 were a bit classier that some of the people who have my telephone number.
Click here to visit editor's Sheepdog Software (tm) freeware, shareware pages.
Click here to visit the homepage of my biggest site.
Click here to visit the homepage of Sheepdogsoftware.co.uk. Apologies if the "?Frmar3ne1sd" I added to that link causes your browser problems. Please let me know, if so?
Page tested for compliance with INDUSTRY (not MS-only) standards, using the free, publicly accessible validator at validator.w3.org
....... P a g e . . . E n d s .....