How much power does gzip save on IoT web access?

Intro

I've recently been working on an optimized library for deflate/zip/gzip decompression. I released it to Github and the Arduino Library Manager as zlib_turbo. With zlib_turbo, I've been exploring HTTP GET requests on the ESP32 and enabling gzip compression to reduce the payload size. I assumed that asking the server to compress the content and sending it as gzip would reduce the connection time. I decided to test how much time and energy this could save. When using the ESP32, or other network-capable MCUs, how much power can we save by requesting compressed content?

What is gzip compression?

Before getting into the project details, a prerequisite is a quick overview of gzip. If you're already familiar with it, you can skip to the next section. gzip is an ancient (relative to the age of the internet) standard based on the deflate compression algorithm. gzip is a file container used to store a single 'deflated' file. There are simple GUI and command line tools to compress and decompress gzip files on every operating system.  gzip files have a header with a magic number followed by fixed and optional data such as the original filename and/or the original file timestamp. The gzip file ends with a 32-bit integer which tells you the uncompressed file size. The deflate compression algorithm uses both statistics to compress with Huffman coding and repeated pattern recognition to replace repeated groups of up to 258 bytes with a single symbol. The end result is that human-readable text files tend to get compressed by 2 or 3:1 while computerish files like html/xml tend to get compressed by 5 or 6:1. The deflate algorithm will have mostly no effect on files which are already compressed. Image files (JPEG/PNG/GIF) fall into this group because those files are already well compressed with algorithms specifically tailored to that type of data. If you request an image file from a web server and opt for gzip content compression, the server will ignore that option and send the file as-is.

The Test Rig

For this test, I'll be using an Unexpected Maker FeatherS2 connected to a Nordic Power Profiler Kit II through the battery connector (to measure actual battery power usage).

The Test Code

I created a simple Arduino sketch which sends a GET request to TomsHardware RSS feed. This returns a XML file. For the first test, I request the data as uncompressed (~224K) and on the second pass, I request it as gzip compressed (~47K).

After the begin() method, we optionally ask for gzip compressed data by setting the "Accept-Encoding:" header value. Not all websites support this, but most should. In the past I would have used the getString() method to get the payload into a String object with a single call. I recently found that this method fails to return the entire payload when it's past a certain size. It doesn't return any error, just less data than you're expecting. I switched to using the stream and adding a pause to wait for more data to arrive instead of giving up when available() returns 0. This seems to work reliably: 

I use an arbitrary timeout value of 4 seconds for the data to arrive. If you know you've got huge payloads for the ESP32, this will probably have to increase.

The HTTPClient library (part of Arduino's ESP32 board support and not in the library manager) doesn't expose any methods to verify the response headers, so we can only tell if the payload is gzip by looking for the magic numbers in the header:

Decompressing the payload is quite easy with the API I created for zlib_turbo:

gzip_info() looks at the gzip header and the uncompressed size value at its end; if everything checks out okay it will return the uncompressed data size. zlib_turbo contains no external dependencies, not even malloc(), so it's up to the caller to manager the buffers. For this example I used ps_malloc() to allocate a potentially huge block of memory from PSRAM (external pseudo-static RAM). You can manage the memory how you like, but I wanted this example to take advantage of the large external RAM available on many ESP32 devices in case the uncompressed data is too big to fit in the internal static RAM. Next, the gunzip() method uncompresses the data. A return code of ZT_SUCCESS (0) means that the deflated data was valid and no decompression errors occurred.

The Results

As you can see in the Arduino serial terminal output above, the time to receive the compressed data is about 1/6th the time compared to the uncompressed data. Decompressing the data after it's received takes a small fraction of that time. The decompression speed of zlib_turbo seems to consistently be in the range of 8-10K bytes per millisecond.

The values in milliseconds don't really help us visualize how much power is being saved. On the PPK chart, the power saved is easier to see:

In the screenshot above, the highlighted area shows the power used to download uncompressed data. The time was over 2 seconds and the total energy consumed was 206mC.

In the screenshot above, we can see the highlighted area for the second part of my example - after a 1 second pause, it leaves the WiFi connection active and downloads the compressed data. This portion lasts for 428ms (the last 31ms is decompressing the data) and uses a total of 40.67mC of energy. If your project runs on batteries, this quantity of energy saved is quite significant. The energy use can be reduced a bit more by turning off WiFi before decompressing the data, but the main concept is well proven by the current code.

As you can see in both screenshots, the ESP32-S2 uses around 30mA for idle tasks, 100mA while connected to WiFi and around 200mA while actively receiving data over WiFi. The S2 has a single CPU core. The original ESP32 and the ESP32-S3 both use more power because they have 2 CPU cores. The lesson is this - the faster you can receive the data and shut down the connection, the more energy you can save.

Final Thoughts

I assumed that web data compression would be beneficial for battery powered ESP32 applications, but my assumptions are only sufficient to be proven or disproven by experimentation. This experiment recorded some hard numbers which show just how beneficial it can be. Before running this power test, I also tested my zlib_turbo library against miniz and found that my code decompresses 50-100% faster than miniz (and zlib) (depending on the data). This is because I profiled and optimized the code (on my Mac) to speed it up.

By releasing my optimized (and simple) gzip solution, and writing this blog post, I hope this:

- Saves you frustration when you need to work with compressed gzip data
- Shows how easy it is to enable compressed payloads with ESP32's HTTPClient
- Extends the battery life of your ESP32 projects
- Helps spread the word about enabling gzip compression to save network bandwidth, time and power

As always, please feel free to leave comments and suggestions below.