Arduino Web Server Data Logger

Created on: 11 July 2015

An Arduino and Ethernet shield are used as an Arduino web server data logger that periodically logs data to a file on the SD card. The logged data can be viewed on a web page.

In the Arduino sketch for this project, the value from analog input A5 is logged to file together with the time in milliseconds from the millis() function. Any other data could be logged in the same way, such as temperature, pressure, voltage, etc.

Data can be logged at the interval set in the sketch. The example sketch logs data every 5000 milliseconds or 5 seconds.

This video shows the sketch running. The logged data is viewed on the web page hosted by the Arduino and can also be viewed by inserting the micro SD card into a card reader. Clicking a link on the web page refreshes the web page with the latest data that has been logged.

Arduino Web Server Data Logger Hardware

An Arduino Uno, MEGA 2560 or similar Arduino board that is compatible with the Arduino Ethernet shield can be used in this project.

The following hardware is needed to build the Arduino web server data logger project:

• Arduino board
• Arduino Ethernet shield
• Micro SD card – inserted into card socket of Ethernet shield, hosts web page and other web files
• USB cable for programming and powering the Arduino
• Ethernet patch cable for connecting the Arduino to the local network

Arduino Web Server Data Logger Code and Web Files

The Arduino sketch code is based on the basic Arduino web server code. As shown below, the basic web server sketch has been modified to add logging capabilities.

After loading the web files to the micro SD card, insert it into the socket on the Ethernet shield. Load the sketch to the Arduino and use a web browser to surf to the IP address set in the sketch. The web page hosted by the Arduino will be displayed in the web browser. Click the link on the web page to load the log file to the web page, click the link periodically to refresh the log file on the web page.

Arduino Sketch

This sketch implements the Arduino web server data logger and must be used in conjunction with the web files found below that are to be copied to the micro SD card.

Change the MAC address in the sketch to the MAC printed on the bottom of the Ethernet shield. Change the IP address to suit your own network.

// Basic Arduino Web Server version 0.1 modified to include logging
// http://startingelectronics.org/software/arduino/web-server/basic-01/
//
// More details and web files available from:
// http://startingelectronics.org/software/arduino/web-server/01-log-data/
//
// Date: 11 July 2015
//
#include <SPI.h>
#include <Ethernet.h>
#include <SD.h>

// maximum length of file name including path
#define FILE_NAME_LEN  20
// HTTP request type
#define HTTP_invalid   0
#define HTTP_GET       1
#define HTTP_POST      2

// file types
#define FT_HTML       0
#define FT_ICON       1
#define FT_CSS        2
#define FT_JAVASCRIPT 3
#define FT_JPG        4
#define FT_PNG        5
#define FT_GIF        6
#define FT_TEXT       7
#define FT_INVALID    8

// pin used for Ethernet chip SPI chip select
#define PIN_ETH_SPI   10

long log_time_ms = 5000; // how often to log data in milliseconds
long prev_log_time = 0;   // previous time log occurred

// the media access control (ethernet hardware) address for the shield:
const byte mac[] PROGMEM = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
//the IP address for the shield:
const byte ip[] = { 192, 168, 0, 20 };  // does not work if this is put into Flash
// the router's gateway address:
const byte gateway[] PROGMEM = { 192, 168, 0, 1 };
// the subnet:
const byte subnet[] PROGMEM = { 255, 255, 255, 0 };
EthernetServer server(80);

void setup() {
  // deselect Ethernet chip on SPI bus
  pinMode(PIN_ETH_SPI, OUTPUT);
  digitalWrite(PIN_ETH_SPI, HIGH);
  
  Serial.begin(115200);       // for debugging
  
  if (!SD.begin(4)) {
    return;  // SD card initialization failed
  }

  Ethernet.begin((uint8_t*)mac, ip, gateway, subnet);
  server.begin();  // start listening for clients
}

void loop() {
  // if an incoming client connects, there will be bytes available to read:
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (ServiceClient(&client)) {
        // received request from client and finished responding
        break;
      }
    }  // while (client.connected())
    delay(1);
    client.stop();
  }  // if (client)
  // log the data
  LogData();
}// void loop()

void LogData(void)
{
  unsigned long current_time;   // current millisecond time
  File logFile;                 // file to log data to

  current_time = millis();  // get the current time in milliseconds
  if ((current_time - prev_log_time) > log_time_ms) {
    prev_log_time = current_time;
    // the log time has elapsed, so log another set of data
    logFile = SD.open("log.txt", FILE_WRITE);
    if (logFile) {
      logFile.print(current_time);
      logFile.print(" = ");
      logFile.println(analogRead(5)); // log the analog pin value
      logFile.close();
    }
  }
}

bool ServiceClient(EthernetClient *client)
{
  static boolean currentLineIsBlank = true;
  char cl_char;
  File webFile;
  // file name from request including path + 1 of null terminator
  char file_name[FILE_NAME_LEN + 1] = {0};  // requested file name
  char http_req_type = 0;
  char req_file_type = FT_INVALID;
  const char *file_types[] = {"text/html", "image/x-icon", "text/css", "application/javascript", "image/jpeg", "image/png", "image/gif", "text/plain"};
  
  static char req_line_1[40] = {0};  // stores the first line of the HTTP request
  static unsigned char req_line_index = 0;
  static bool got_line_1 = false;

  if (client->available()) {   // client data available to read
    cl_char = client->read();
    
    if ((req_line_index < 39) && (got_line_1 == false)) {
      if ((cl_char != '\r') && (cl_char != '\n')) {
        req_line_1[req_line_index] = cl_char;
        req_line_index++;
      }
      else {
        got_line_1 = true;
        req_line_1[39] = 0;
      }
    }
    
    if ((cl_char == '\n') && currentLineIsBlank) {
      // get HTTP request type, file name and file extension type index
      http_req_type = GetRequestedHttpResource(req_line_1, file_name, &req_file_type);
      if (http_req_type == HTTP_GET) {         // HTTP GET request
        if (req_file_type < FT_INVALID) {      // valid file type
          webFile = SD.open(file_name);        // open requested file
          if (webFile) {
            // send a standard http response header
            client->println(F("HTTP/1.1 200 OK"));
            client->print(F("Content-Type: "));
            client->println(file_types[req_file_type]);
            client->println(F("Connection: close"));
            client->println();
            // send web page
            while(webFile.available()) {
              int num_bytes_read;
              char byte_buffer[64];
              // get bytes from requested file
              num_bytes_read = webFile.read(byte_buffer, 64);
              // send the file bytes to the client
              client->write(byte_buffer, num_bytes_read);
            }
            webFile.close();
          }
          else {
            // failed to open file
          }
        }
        else {
          // invalid file type
        }
      }
      else if (http_req_type == HTTP_POST) {
        // a POST HTTP request was received
      }
      else {
        // unsupported HTTP request received
      }
      req_line_1[0] = 0;
      req_line_index = 0;
      got_line_1 = false;
      // finished sending response and web page
      return 1;
    }
    if (cl_char == '\n') {
      currentLineIsBlank = true;
    }
    else if (cl_char != '\r') {
      currentLineIsBlank = false;
    }
  }  // if (client.available())
  return 0;
}

// extract file name from first line of HTTP request
char GetRequestedHttpResource(char *req_line, char *file_name, char *file_type)
{
  char request_type = HTTP_invalid;  // 1 = GET, 2 = POST. 0 = invalid
  char *str_token;
  
  *file_type = FT_INVALID;
  
  str_token =  strtok(req_line, " ");    // get the request type
  if (strcmp(str_token, "GET") == 0) {
    request_type = HTTP_GET;
    str_token =  strtok(NULL, " ");      // get the file name
    if (strcmp(str_token, "/") == 0) {
      strcpy(file_name, "index.htm");
      *file_type = FT_HTML;
    }
    else if (strlen(str_token) <= FILE_NAME_LEN) {
      // file name is within allowed length
      strcpy(file_name, str_token);
      // get the file extension
      str_token = strtok(str_token, ".");
      str_token = strtok(NULL, ".");
      
      if      (strcmp(str_token, "htm") == 0) {*file_type = 0;}
      else if (strcmp(str_token, "ico") == 0) {*file_type = 1;}
      else if (strcmp(str_token, "css") == 0) {*file_type = 2;}
      else if (strcmp(str_token, "js")  == 0) {*file_type = 3;}
      else if (strcmp(str_token, "jpg") == 0) {*file_type = 4;}
      else if (strcmp(str_token, "png") == 0) {*file_type = 5;}
      else if (strcmp(str_token, "gif") == 0) {*file_type = 6;}
      else if (strcmp(str_token, "txt") == 0) {*file_type = 7;}
      else {*file_type = 8;}
    }
    else {
      // file name too long
    }
  }
  else if (strcmp(str_token, "POST") == 0) {
    request_type = HTTP_POST;
  }

  return request_type;
}

SD Card Web Files

The following files must be copied to the micro SD card, although the favicon.ico and robots.txt are optional.

index.htm

This is the main web page displayed when a web browser connects to the Arduino web server.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Arduino Web Server Logging</title>
        
        <link rel="stylesheet" href="styles.css">
        <script src="script.js"></script>
    </head>

    <body>
        <h1>Arduino Web Server Logging</h1>
        <p><a href="log.txt">Fetch log file</a> to refresh log data.</p>
        <p>Log values:</p>
        <iframe src="default.htm" name="icontent" id="icontent"></iframe>
        
    </body>

</html>

default.htm

This web page is the default page displayed in the iframe of the index.htm page before the logged data is displayed.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Load Log File</title>
    </head>

    <body>
        <p>Click link to load log file.</p>
    </body>

</html>

styles.css

A simple style sheet straight from the basic web server version 0.1 web files. It only changes the background colour of the index.htm web page, but can be used to add custom styles to the project.

body {
    background-color: #DFEBED;
}

script.js

This script is used to load the log file to the iframe window when the link on the page is clicked.

window.onload = initLinks;

function initLinks() {
    for (var i=0; i<document.links.length; i++) {
        document.links[i].target = "icontent";
    }
}

robots.txt

Used to keep web spiders / bots from indexing the web page if the Arduino web server is put on the Internet instead of on a local network.

User-agent: *
Disallow: /

favicon.ico

Use your own favicon.ico file, or copy the Arduino favicon file (right click and choose Save Link As or similar menu item, depending on browser).

How the Arduino Web Server Data Logger Works

The web server part of the code can be understood by studying the code from the basic web server version 0.1. Web server principles and technology is explained in the Arduino web server tutorial series.

Adding Logging Capabilities to the Web Server

The basic web server code was modified to add the LogData() which logs the analog data to a file on the SD card.

Logging Period

A variable near the top of the web server sketch can be changed to set the interval period that data is logged in milliseconds:

long log_time_ms = 5000; // how often to log data in milliseconds

In the sketch, data is sampled every 5 seconds, or 5000 milliseconds.

Logging the Data

Calling the LogData() function periodically in the main loop of the sketch, ensures that the time between each logging event is checked. If the time period since the last log has expired and the log file can be opened, the analog input A5 is read and saved to the log file together with the current millisecond time.

LogData can be modified to log the desired values, whether they are additional analog values or data from a sensor.

After at least one value has been logged, the link in the Arduino hosted web page can be clicked to display the contents of the log file.

Limitations of the Sketch

The sketch uses a fairly long delay between logging data and is intended for logging values at longer intervals, preferably minutes apart.

Long intervals are necessary for logging as logging can not take place while the Arduino is busy serving up a web page to a browser. The bigger the web page, the longer it will take to serve up the web page and the longer the period will be that data can not be logged.

If faster or more accurate logging periods are needed, it will be necessary to use interrupts. Some buffering would need to be used in conjunction with the interrupt so that the SD card is not written to in the interrupt service routine (ISR).

For projects that are logging data such as temperature throughout the day and only need sample rates of 5 or 10 minutes, the example sketch should be sufficient.

References

The web server is based on the basic web server code.

Calculating the sample intervals using the millis() function was taken from the BlinkWithoutDelay example sketch.

Logging analog values to an SD card file was taken from the Datalogger example sketch.