An embedded-friendly PNG decoder

Intro

I wrote my own imaging codecs many years ago for all of the 'standard' file formats available. Over the last couple of years I've been dusting off that code to give it a new life as open source libraries for embedded/Arduino. I wrote blog posts about my JPEG and GIF decoders so I thought it would be useful to write about my new PNG decoding library.

What is PNG?

The (Portable Network Graphic) specification was created not too long after Unisys started enforcing their LZW patent to collect licensing fees from the use of GIF images. PNG was specifically designed to not infringe any patents and offered the benefit of supporting many more pixel types and an alpha channel. This offered a new feature not found in JPEG and much more functional than GIF's single transparent color. PNG uses the ZIP/FLATE compression scheme which is open source and without any patents attached to it. The FLATE compression by itself isn't drastically better than the LZW compression used in GIF files. What allows PNG files to compress images so well is the addition of a pre-filtering step which takes advantage of symmetries in the data to make it more compressible. For several years GIF hasn't been hampered by any active patents and could take advantage of the same trick to shrink its files, but alas it's been marked "deprecated" in the eyes of many. It's highly unlikely GIF will see any updates to its spec.

PNG Filters

There are 4 filter types (not including 'none') used to predict pixels on the current line from neighboring pixels. Filters basically replace the pixels with the difference between the pixel and its best matching neighbor. For areas of constant color or constant gradients, this will replace the data with 0's (or the gradient delta). The compression scheme used on the filtered data can do a better job if there are fewer unique symbols (aka colors) to compress. Imagine an image composed of vertical stripes of colors. If the "up" filter is applied, The entire image would be converted to 0's except for the first line. LZW or FLATE would still do a decent job with the original color bars, but could do a much better job if most of the data were 0's. The filter type can be unique on each line of the image and is stored as the first byte of each line of the compressed data. If a 8-bit palette image is 320 pixels wide, then there will be 321 bytes per line of data in the file. PNG encoders that try to compress the data as small as possible will test each filter on each line and choose the one that results in the lowest 'entropy'. Some encoders simply choose the 'paeth' filter for every line since it generally performs the best. To decode a PNG image (non-interlaced) you need to buffer at least the previous line of pixels in order to be able to de-filter the image data. This sets the minimum amount of memory that my library uses - zlib (32K + 7K), file buffer (2K), current and previous lines (5K by default).

The MCU Challenge

Working with images on constrained memory devices is always challenging; pixels take a lot of space. Consider the case that you have an ATSAMD51 (192K internal RAM) and want to display a 320x240x32-bpp PNG image. If you were to decode the whole image into RAM, you would require 320x240x4 = 300K bytes, plus the 39K needed by the zlib inflate() code. This would normally be impossible to do if not for displays designed specifically for "Arduino" devices. These low cost displays have an internal frame buffer and receive commands+data over an SPI bus. One of the design goals of this library was to handle this use case. PNG decoding is already designed to be done a line at a time and so I added an optional callback function so that only a single line of pixels needs to be buffered and can then be transmitted to an external LCD. You can also decode the full image into memory in one shot if you have enough space. Part 2 of this challenge is managing all of the possible pixel types. PNG supports 1/2/4/8-bit palette images and grayscale or truecolor, all with optional alpha channel. I decided it would be better to deliver the native pixel format to the callback function (faster than forcibly converting it to another format) and then provide a separate function to convert any of the pixel types into RGB565 for those inexpensive LCD displays. This design gives the best speed and flexibility.

The second challenge of PNGdec was to remove dynamic memory calls (malloc/free). The reason I like to make my Arduino libraries not call malloc and free is to allow the code to be built on the simplest possible embedded system with no C-runtime library. I used zlib to handle the inflate'ing of the compressed data. It's got an alloc/free callback option to provide your own memory management. The only problem is that it makes multiple calls to allocate different sized blocks of memory. This could be handled, but I thought it would be easier to just remove the calls entirely and have it work from a single block of memory. This way, the caller can decide how to provide that memory - by reserving a static block or dynamically allocating it in one shot.

Where Can I use it?

I decided to set a max image width of 640 x 32-bpp (which can also be 2560 x 8-bpp). The total memory required for decoding images of up to this width (and any length) is around 46K. This seemed like a reasonable balance, but unfortunately excludes a lot of smaller embedded devices. This couldn't be avoided because the zlib structures require 39K of RAM by themselves. The most likely systems that this will run on are: ESP32, Cortex-M and RISC-V (all requiring at least 64k of RAM). You can certainly run this code on a PC or Linux level machine, but why would you? A CPU with enough memory to run an Operating System has enough memory to run the full version of libpng. I included a Makefile and Linux test app for convenience, but I don't expect anyone to use it that way.

How Fast is it?

PNG decoding is all about inflating the compressed data and then de-filtering it using the various filter functions. These two steps should take nearly 100% of the decode time and are split about 50/50 on non-SIMD implementations. The overhead of reading the data from the file and managing the output pixels can also take time (and does with the pngle library). My library does its best to not add any additional latency to the zlib+filter time, so it ends up being relatively fast. I also disable the internal Adler32 CRC check by default which speeds things up by 10-20%. PNG has multiple levels of CRC checking - zlib checks the compressed data as it inflates it and the PNG file has its own CRC calculation that runs on that same data afterwards. The zlib CRC is disabled by default and I didn't add code to check the PNG CRC values. If you're not sure of the integrity of the file you're decoding, then enable the zlib level CRC checking.

Below is a simple benchmark test I wrote which decodes 3 versions of a 240x200 image and either delivers the native pixels or converts the output to RGB565. The conversion from palette colors to RGB565 could be sped up with a lookup table, but that will require more memory.

Wrap Up

With the "trifecta" of my JPEG, GIF and PNG decoders, embedded devices now have an efficient way to decode a wide range of standard image files. PNG specifically fills a need because of its high compression ratios, wide range of pixel types and support of an alpha channel. I hope it enables people to create brand new projects or allows existing ones to look prettier. As always, if you have any questions or comments, please leave them below.



If you appreciate my blog posts and open source work, please consider sponsoring me on Github:

My Github Sponsors Page

Comments

  1. How is the alpha channel converted into RGB565, what will be displayed at transparent pixels?

    ReplyDelete
    Replies
    1. The original pixel data is available; I just offer an easy function to convert it to RGB565. I also have a function to generate a bit mask with a threshold value for transparent pixels.

      Delete
  2. Hello, very nice library.
    The Github documentation states that the library needs at least 48Kb of available RAM to run. This is actually quite a lot. For example, on an esp32, after loading bluetooth, wifi, TLS, HTTP, we are often under 40Kb of available HEAP. Does the library really need so much RMA, and if so, why ?
    Thanks !

    ReplyDelete
    Replies
    1. 48K is actually "slim" for PNG. If you try to run libpng on ESP32, it will need 200K+ RAM to do the same job. The problem is that the FLATE compressed data can use the 32K window option. If you want to be able to decode all possible PNG files, then 48K is about as low as you can go.

      Delete
    2. Ok, thanks, I understand. So PNG decompression has to cope for all types of compression filters, including ones that need a large buffer. This is a bummer. I'm trying to find a viable solution to compress our images in flash and decompress them on the fly (like using your callback mechanism). Do I have a way to compress PNG files so that they do not use a filter that requires a large buffer ?
      Otherwise, it seems that the only solution for me in low RAM context is zip. Or a custom compression scheme. Too bad.
      Thanks for the discussion

      Delete
    3. If you use ZIP, you're using the same underlying compression type as PNG (FLATE) and it will require lots of RAM just like PNG. I just wrote a new imaging codec that uses almost no RAM called "SLIC". It doesn't compress as well as PNG, but it can come close. Give it a try --> https://github.com/bitbank2/SLIC

      Delete
    4. Thanks, I was about to try zip, but the uzlib library which I was going to use doesn't state the RAM needs upfront, I was just going to try it out. OK, I'll drop this lead.
      Your SLIC library looks fantastic, especially since we're using 24 bits images with large part of uniform colors. I'll look into it ASAP. I'll let you know.

      Delete
  3. Hi Larry, I have implemented your zlibs in my project and I compile it without problems. I use an atmel SAM4 and write the code in C language.
    I don't know arduino, much less cpp, then I ask: is it difficult to translate your code (png) into c?
    thank you in advance for you answer and sorry for my orrible english...

    ReplyDelete
    Replies
    1. My PNG decoder and other image codecs are written in C with a C++ wrapper class. You can find C examples in the linux folder of each repo.

      Delete
    2. Thanks Larry, you are right. I've seen cpp extension and therefore did not open linux esample, but it written in C.
      Have a nice day.

      Delete
    3. Nothing to do.. no way... now it's Christmas and I'll try it in a few days...I can't get rid of c++. Even if it's just a wrapper I don't know c++ and can't figure out how to delete it, how to change the .inl file to .c etc.... I'm discouraged :-(

      Merry Christmas Larry

      Delete
    4. I named it inl to be "inlined" or included in your C file and so that the Arduino IDE doesn't try to compile it on its own. Just include it like a header file in your C code and it will work.

      Delete

Post a Comment

Popular posts from this blog

Building the Pocket CO2 Project

How much current do OLED displays use?

Fast SSD1306 OLED drawing with I2C bit banging