A Modern Micro Web Stack For The ESP32

esp32uiThere are a lot of different ways to serve up web content on the popular ESP32 platform.  As a part of building some prototype devices we need a good user interface for configuring our devices to connect, and we want to do that via a web interface running on the device itself.  We’re prototyping with the Arduino tooling, which makes it really simple to prove out an idea.  Most of the Arduino examples out there use some variant of printing content out to the browser from within the sketch itself to serve up a page, SVG, whatever:

ESP32WebServer server ( 80 );

String message = “open tags here”;
message += “Some content goes here”;
message += “close tags here”;
server.send ( 200, “text/html”, message );

Look familiar?  The idea is pretty simple.  Start up a web server, wait for a connection, write out some HTML to the browser.  It works, but this is a huge pain and I’m not a big fan of building up HTML in strings inside of a sketch.  Yeah, it works, but it just feels wrong.  For our project, I started wondering what it would take to get a more modern front end working with an ESP32 based Arduino. Building that out would require a few things:

  1. The frameworks need to be small.  The ESP32 boards I’m working with have 4MiB of flash available.  That’s not a lot of space.
  2. The frameworks need to be simple.  Anything that requires maintaining a complex component tree is going to be tough.
  3. I’d like something with a modern build process
  4. It would be great to have something that doesn’t require embedding anything in the sketch itself, and allows me to modify the UI without modifying the sketch at all.

After a little digging, I settled on Preact and Milligram for a proof of concept.  Preact is a simple, lightweight, React inspired (and mostly compatible) JS framework.  Milligram is a similarly tiny and lightweight css framework.  Together, uncompressed, both frameworks weigh in at less than 30k combined.  Great!  That’s item number one checked off the list.  Preact comes with a handy CLI that sets up a project with all the build tools ready to go.  One of the project templates that the CLI can generate is the “simple template” that creates the simplest possible Preact project structure, and all the build scripts are automatically built as well.  That’s item number two and three checked off as well.  So far, so good, so let’s get started!

First, there are a few tools to install.  Follow the instructions for installing the Preact CLI, and make sure you have a recent version of the Arduino IDE.  This is also a good time to install the Arduino ESP32 filesystem plugin, and the ESP32 HTTPS Server library if you want to follow along.  To start, create a simple Arduino sketch.  Do this the same way you usually would, and save the sketch in a new location.  This will create a folder for your project, and in that folder is a sketch with the same name:

my-awesome-project
|—- my-awesome-project.ino

I’d like to keep all of this self contained, so within the Arduino project directory I’ve created a folder for the Preact UI project, using the Preact CLI.  From within the Arduino sketch folder, use the CLI to create a project from the “simple” template:

preact create simple my-project-ui

Since I wanted to use Milligram as a CSS framework, I simply replaced the default style.css that the Preact CLI added to my project with milligram.min.css from the Milligram distribution bundle.  You can drop it in and name it style.css, or change the import statement in index.js to point to milligram.min.css.  Either should work.

Now that we have a simple UI project to start with, we need a way to serve it up on the ESP32 that doesn’t involve building strings in a sketch.  To do that the UI files need to get moved to the Arduino.  On the ESP32 we can use the flash storage, and we can use the SPIFFS filesystem to make things a little easier.  SPIFFS isn’t like most of the filesystems used on larger systems.  It doesn’t currently support directories, for example, so we need a flat structure for our UI project.  There are multiple ways to upload files to SPIFFS, including the official Espressif ESP-IDF, and a handy Arduino plugin that can write files to the device storage.  Since I’m prototyping on Arduino, I chose the latter option.  This plugin will look in your sketch folder for a “data” folder that contains the data to be uploaded to the ESP32 flash, mounted to “/data” for SPIFFS.  So, go back into your sketch folder and create a folder called “data”.  Once you are done, your Arduino project structure should look more or less like this:

my-awesome-project
|—- my-awesome-project.ino
|—- my-project-ui
|—- data

Pretty simple so far.  UI development takes place in the my-project-ui directory, and can be locally tested.  When the UI is ready to test on the device, we can use “preact build” to bundle up a production version and output it to the data directory.  Once it is there, the ESP32FS Arduino plugin provides the tools to copy it to the device flash.  Our sketch will contain a simple web server, configured to serve content off of the flash storage via SPIFFS.  Let’s start with the UI build.

Preact uses the typical JS build tools, a quick look at the build scripts reveals Babel and Webpack, but for this simple case you probably won’t need to worry about it.  After building your components and testing the app locally, use the Preact CLI to prepare a production build and drop it into the data directory:

preact build –production –dest ../data

The build only takes a moment, and once it is complete the data directory of your Arduino project should contain a copy of the UI.  If you haven’t already done so, install the ESP32FS plugin using the instructions provided by the project.  Then, use the plugin to push the data directory to the flash on the ESP32.  The plugin can be found on the tools menu.  Make sure that the serial monitor is closed or the IDE will complain.

Screen Shot 2018-09-23 at 4.36.09 PM

At this point in the process, we’re halfway there.  We have a modern UI framework that is small enough to be practical on a device with extremely limited resources, a way to build it, and a way to get it on the device.  As an aside, at the ESP32 Sketch Data Upload tool reports that my base UI took up 78K compressed on the flash storage.  Not bad!  Now, how can we serve it up?

The sketch is where it starts to get a little tricky.  There are several different HTTP server implementations available for the ESP32, and each has pros / cons.  The first one I tried was a port of the web server library for the ESP8266, which preceded the ESP32.  This library has built in support for SPIFFS, which is what drew me to it in the first place.  Serving up a file stored on SPIFFS is as easy as initializing the filesystem with a mount point and then mapping that file to a URL:

ESP32WebServer server(80);
SPIFFS.begin(true, “/data”, 10);
server.serveStatic(“/index.html”, SPIFFS, “/index.html”);

This worked well, but created a few problems.  First, I couldn’t get directory mappings to work with the serveStatic function.  Each file would need to be mapped individually like in the example above.  That means I can’t change the project structure without changing the sketch.  No bueno.  Also, Preact’s default build adds a hash to filenames.  I assume this is for cache busting.  Handy, but if the filename changes with every build then the static mappings shown in the code snippet above becomes even more onerous.  I decided to test it anyway, and also found that the ESP32WebServer doesn’t cope well with multiple concurrent connections.  I was getting lots of resets, partial loads, etc.  The combination of these two shortcomings sent me looking for other options.

Enter the (so far) excellent ESP32 HTTPS Server.  This library looks a lot more full featured, and uses a programming model that should be familiar to most people.  Map a URL, define a handler that receives the request and response objects.  If you’ve written Java Servlets or used a framework like Express the model this library uses should feel immediately familiar.  The ESP32 HTTPS Server doesn’t seem to have a direct way to map to SPIFFS storage to serve files though, so we need to do a little work to make that happen.  Most of the examples for this library show individual URLs and HTTP methods mapped to specific handlers.  However there is a special default handler.  Perhaps we can use that to serve up whatever we ask for on the filesystem?

// set up the HTTPServer instance
HTTPServer server = HTTPServer();
// define the file handler function
void handleFile(HTTPRequest * req, HTTPResponse * res);
// create a resource node for the default handler
ResourceNode * nodeDefault = new ResourceNode(“”, “GET”, &handleFile);
// set this resource node as the default handler
server.setDefaultNode(nodeDefault);

The code above creates the HTTPServer instance, defines a handler function, and tells the server to use that function for any request that doesn’t have another mapping.  Here’s what the handleFile function looks like:

void handleFile(HTTPRequest * req, HTTPResponse * res) {
std::string reqString = req->getRequestString();

static uint8_t buf[512];
size_t len = 0;

File file = SPIFFS.open(reqString.c_str());

// if we can’t find the requested file, send the 404 page
if (!file) {
file = SPIFFS.open(“/404.html”);
res->setStatusCode(404);
}else {
res->setStatusCode(200);
}

len = file.size();

res->setHeader(“Content-Length”, “” + len);
res->setHeader(“Content-Type”, getContentType(reqString.c_str()));
size_t flen = len;
size_t i = 0;
while (len) {
size_t toRead = len;
if (toRead > 512) {
toRead = 512;
}
file.read(buf, toRead);
len -= toRead;
res->write(buf, toRead);
}
}

This function is pretty simple.  It uses the request string to get the path that the browser is asking for and checks to see if a file exists with that name.  If it does, the proper response code is set and the file is sent.  If the requested file does not exist, then a 404 code is set and a simple “not found” message is sent instead.  Nice!  Now that shiny new Preact + Milligram UI can be served up via the ESP32.  No ugly string concatenation in a sketch.  No having to hard code every file we are serving up either.  When I start to build out the REST API for this device, I can create additional handlers for those routes.  As an added bonus, this library supports HTTPS so it should be straightforward to bake in secure connections.

If you’d like to try out the ESP32 server sketch, I’ve uploaded it to Github.  This should run pretty much out of the box on the Heltec ESP32 dev boards with the OLED displays.  I think this presents a great way to build out a web UI for your next ESP32 project!

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s