Save a Web File to the Arduino SD Card

Created on: 10 June 2015

This article shows how to use an Arduino to fetch a file from the Internet and save it to an SD card.

An Arduino and Ethernet shield is needed to connect to the Internet, or an Arduino with built-in Ethernet such as the Arduino Ethernet board.

The Ethernet shield provides a wired connection to the Internet as well as a micro SD card socket for connecting a micro SD card to the Arduino.

An Arduino sketch is used to configure the Arduino as an Ethernet client that fetches the web file. The web file can be an HTML page, XML file, text file, image or any other web resource file.

Two sketches are provided below. The first sketch saves the HTTP header and requested file to the SD card in case further information from the HTTP header is required by a project. The second sketch saves only the requested file and strips off the HTTP header.

Hardware Setup

The sketches were tested using an Arduino Uno and Ethernet shield, with 2GB micro SD card inserted into the micro SD card slot of the Ethernet shield.

Other Arduino boards that are compatible with the Ethernet shield should work, such as the Arduino MEGA 2560 and others.

Saving the HTTP Header and Web File

The web2SD sketch below saves the HTTP header and requested file to the SD card.

Change the server that the file is being requested from, the requested file name and name of the file to save on the SD card in the sketch where commented in the sketch.

web2SD Sketch

// Gets a file from the Internet and saves it to the SD card
// Also saves the incoming HTTP header to the file
// This example gets the XML cricket score from synd.cricbuzz.com
//
// References - Arduino Example sketches from IDE by David A. Mellis et al.:
// -- File --> Examples --> Ethernet --> WebClient
// http://www.arduino.cc/en/Tutorial/WebClient
// -- File --> Examples --> SD --> ReadWrite
// http://www.arduino.cc/en/Tutorial/ReadWrite
// 
// Author: W.A. Smith    Date: 10 June 2015
// http://startingelectronics.org/software/arduino/save-web-file-to-SD-card/

#include <SPI.h>
#include <Ethernet.h>
#include <SD.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
char server[] = "synd.cricbuzz.com";    // change server to get file from here
IPAddress ip(192, 168, 0, 50);
EthernetClient client;
File theFile;

void setup() {
  // disable Ethernet chip
  pinMode(10, OUTPUT);
  digitalWrite(10, HIGH);
  
  Serial.begin(115200);
  
  Serial.print(F("Initializing SD card..."));

  if (!SD.begin(4)) {
    Serial.println(F(" initialization failed!"));
    return;
  }
  Serial.println(F(" initialization done."));
  
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println(F("Failed to configure Ethernet using DHCP"));
    // no point in carrying on, so do nothing forevermore:
    // try to congifure using IP address instead of DHCP:
    Ethernet.begin(mac, ip);
  }
  // give the Ethernet shield a second to initialize:
  delay(1000);
  
  Serial.println("connecting...");

  // if you get a connection, report back via serial:
  if (client.connect(server, 80)) {
    Serial.println("connected");
    // Make a HTTP request:
    client.println("GET /j2me/1.0/livematches.xml HTTP/1.1");  // change resource to get here
    client.println("Host: synd.cricbuzz.com");                 // change resource host here
    client.println("Connection: close");
    client.println();
  }
  else {
    // didn't get a connection to the server:
    Serial.println("connection failed");
  }
  // open the file for writing
  Serial.println("Creating file.");
  theFile = SD.open("results.xml", FILE_WRITE);  // change file name to write to here
  if (!theFile) {
    Serial.println("Could not create file");
    while (1);
  }
}

void loop() {
  // if there are incoming bytes available
  // from the server, read them and print them:
  if (client.available()) {
    char c = client.read();
    theFile.print(c);
  }

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();
    theFile.close();
    Serial.println("Finished writing to file");
    
    // do nothing forevermore:
    while (true);
  }
}

web2SD Sketch Resources

This sketch is based on these two Arduino example sketches for configuring an Arduino as a web client and writing to the SD card:

• WebClient – configure the Arduino as a web client.
• ReadWrite – write to the SD card.

Saving the Web File without HTTP Header

The sketch below called WebFile2SD saves only the requested file to the SD card and does not include the HTTP header in the file.

WebFile2SD Sketch

// Gets a file from the Internet and saves it to the SD card
// Removes the incoming HTTP header, saves the file only
// This example gets the XML cricket score from synd.cricbuzz.com
//
// References - Arduino Example sketches from IDE by David A. Mellis et al.:
// -- File --> Examples --> Ethernet --> WebClient
// http://www.arduino.cc/en/Tutorial/WebClient
// -- File --> Examples --> SD --> ReadWrite
// http://www.arduino.cc/en/Tutorial/ReadWrite
// 
// Author: W.A. Smith    Date: 10 June 2015
// http://startingelectronics.org/software/arduino/save-web-file-to-SD-card/

#include <SPI.h>
#include <Ethernet.h>
#include <SD.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
char server[] = "synd.cricbuzz.com";    // change server to get file from here
IPAddress ip(192, 168, 0, 50);
EthernetClient client;

boolean currentLineIsBlank = true;
File theFile;

void setup() {
  // disable Ethernet chip
  pinMode(10, OUTPUT);
  digitalWrite(10, HIGH);
  
  Serial.begin(115200);
  
  Serial.print(F("Initializing SD card..."));

  if (!SD.begin(4)) {
    Serial.println(F(" initialization failed!"));
    return;
  }
  Serial.println(F(" initialization done."));
  
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println(F("Failed to configure Ethernet using DHCP"));
    // no point in carrying on, so do nothing forevermore:
    // try to congifure using IP address instead of DHCP:
    Ethernet.begin(mac, ip);
  }
  // give the Ethernet shield a second to initialize:
  delay(1000);
  
  Serial.println("connecting...");

  // if you get a connection, report back via serial:
  if (client.connect(server, 80)) {
    Serial.println("connected");
    // Make a HTTP request:
    client.println("GET /j2me/1.0/livematches.xml HTTP/1.1");  // change resource to get here
    client.println("Host: synd.cricbuzz.com");                 // change resource host here
    client.println("Connection: close");
    client.println();
  }
  else {
    // didn't get a connection to the server:
    Serial.println("connection failed");
  }
  // open the file for writing
  Serial.println("Creating file.");
  theFile = SD.open("crscore.xml", FILE_WRITE);  // change file name to write to here
  if (!theFile) {
    Serial.println("Could not create file");
    while (1);
  }
}

void loop() {
  // if there are incoming bytes available
  // from the server, read them and print them:
  if (client.available()) {
    char c = client.read();
    if (c == '\n' && currentLineIsBlank) {
      // end of HTTP header, now save requested file
      while (client.connected()) {
        // stay in this loop until the file has been received
        if (client.available()) {
          c = client.read();  // get file byte
          theFile.print(c);   // save file byte
        }
      }
    }
    // detect the end of the incoming HTTP header
    if (c == '\n') {
      // starting a new line
      currentLineIsBlank = true;
    }
    else if (c != '\r') {
      // got a character on the current line
      currentLineIsBlank = false;
    }
  }

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();
    theFile.close();
    Serial.println("Finished writing to file");
    
    // do nothing forevermore:
    while (true);
  }
}

WebFile2SD Sketch Resources

This sketch is based on these three Arduino example sketches for configuring an Arduino as a web client, writing to the SD card and detecting the end of the HTTP header:

• WebClient – configure the Arduino as a web client.
• ReadWrite – write to the SD card.
• WebServer – detect end of HTTP request using currentLineIsBlank flag.

Running the Sketches

After one of the above sketches is loaded to the Arduino, it will connect to the server and fetch the file that has been specified in the sketch and save it to SD card.

The SD card can be removed from the Arduino and inserted into an SD card reader attached to a PC to view the requested file.

Viewing the Sketch Serial Messages

The Serial Monitor window must be opened to view the messages output by the Arduino sketch. The Serial Monitor window baud rate must be set to 115200 at the bottom of the window.

If the Serial Monitor window is opened immediately after loading the sketch, the Arduino will be reset and then the text messages from the Arduino will be seen. If the Serial monitor window is not opened immediately after loading the sketch, but after a while, then the sketch will run initially and then again when the Arduino is reset by the Serial monitor window being opened. This results in the file data being written to the SD card file twice because the file is not overwritten, but added to if it already exists.

Preventing the SD Card File Being Written to More than Once

To prevent the file being added to each time that the sketch runs, first check if the file exists in the setup() part of the sketch after initializing the SD card and then delete it if it exists as shown in the code below.

if (SD.exists("results.xml")) {
  SD.remove("results.xml");
}

Sketch Serial Output and SD Card File Examples

This image shows the output from the Serial Monitor window of the Arduino IDE after running one of the sketches:

The image below shows the SD card file opened in the file manager of a PC and opened in a text editor: