My BLE Adventures
On my continuing quest to learn "The IoT", I've explored a long list of sensors, displays, microcontrollers, LEDs and some forms of wireless communication. Recently I've taken more of an interest in Bluetooth Low Energy because of its wide adoption in consumer devices and integrated support in some MCUs. This blog post documents some of the trivia/tricks I've learned along the way. The word 'Adventures' is really a euphemism; the BLE specification is completely documented, yet each companies' software (and API) implementation is unique, full of missing features, incompatibilities and outright bugs. It's been a somewhat frustrating journey to arrive where I have and hopefully this blog post will help you avoid some of the sticking points and dead ends I've hit along the way. Red text indicates a feature limitation, bug or other issue you'll probably find important.
First, some terminology:
BT 2.0/4.x/5.x - The Bluetooth spec is updated periodically with new features and a new revision number. These numbers don't mean a whole lot to the general public and even less when they're placed in advertisements for devices. For example, a wireless keyboard which advertises support for BT 4.0 tells you very little. 4.0 is when BLE was added, but that doesn't mean it's a BLE keyboard. If it claimed a number lower than 4.0, then you could be confident that it only supported classic BT. A few device manufacturers are smart enough to include "BLE" or "BT Smart" with their advertisements, but the majority leave you guessing.
GATT (Generic Attribute Profile) - a software protocol based on universally unique identifiers (UUIDs). The BLE spec defines many UUIDs for specific purposes, but you can also create your own proprietary ones.
Central - The BLE name for what is normally called the client. The Central initiates commands, connection requests and responses.
Peripheral - The BLE name for what is normally called the server. The Peripheral advertises information and optionally allows connections.
Characteristic - a data value transferred between Central and Peripheral. A characteristic can be set up to notify the software when its value has changed. You can read or write one byte or many at a time, this depends on the what the Characteristic is offering. A safe maximum for local buffering of this data is probably 512 bytes (on embedded devices)
Service - a collection of related Characteristics that work together to provide specific functionality.
Descriptor - provides additional information about a Characteristic. Sometimes used for things like units (temperature), or a switch to enable/disable a feature.
MTU (Maximum Transmission Unit) - the largest packet size that can be sent
BLE API
Each microcontroller provider has decided to implement Bluetooth in their own unique way, with their own API. Even though the BT specification is pretty specific, each vendor hasn't implemented the spec in its entirety nor have they been consistent about it. The major players for embedded BT APIs are Espressif Systems (ESP32), Arduino (Arduino Nano BLE 33), Nordic Semiconductor (nRF52 SDK), Adafruit (several) and Particle. The APIs are different enough that writing code which needs to run on multiple platforms consists of writing multiple versions of the code that are quite different. The Arduino ecosystem now supports a large number of embedded boards that were not created by Arduino, yet can share most of the same libraries. Unfortunately this isn't the case with Bluetooth support. For example, I've decided to focus on the ESP32 and Nano 33 BLE, both of which are fully supported in the Arduino IDE. Being part of the same development environment, however doesn't mean that they share the same software support. The BLE API for each of them is vastly different in terms of class/method names and capabilities. The only project that I've gotten to run identically on both is my Thermal Printer Library and only because it uses the simplest of features of BLE (scan and write to Characteristic). You'll probably have to choose sides and stay with that choice as you create more complicated projects because it will get even harder to switch and rewrite/relearn everything.
File Transfer
My first BLE experiment involved trying to send data (a file) as quickly as possible from one microcontroller to another. I used two ESP32 modules I had on hand. One of the downsides of BLE versus classic BT is the effective data rate is much lower because BLE saves power by adding pauses in between packets to turn off the transmitter. These pauses and the small packet size (default = 23 bytes) means that data doesn't go anywhere near as fast as the 2Mbs radio speed would lead you to believe. I discovered there are 2 main issues that you have some control over to determine how fast data can be transferred:
MTU Size - BLE can request a larger MTU as part of its connection negotiation. There isn't really a max MTU size, it's up to the software implementation, but there is a limit of 257 bytes per physical data block transmitted. An MTU larger than this will be broken up into multiple PDUs. Transmitting more info per block increases the speed of data transfer. The peripheral may reject/reduce your MTU size change request. Your ability to read/write large blocks of data at a time depends mostly on the BLE API implementation. A write request that's larger than the MTU payload size (whatever is currently agreed upon) will be broken up into multiple blocks.
WRITE / WRITE_NO_RESPONSE - Most devices I've seen allow writing data with either option. A "WRITE" will wait for an ACK (acknowledgement) packet from the receiver before sending more data. "WRITE_NO_RESPONSE" as the name implies, doesn't wait for a response (none is sent). As you can imagine, it's much faster to transmit data when you don't have to wait for an ACK. The ESP32 allows you to specify what type with a true/false flag in the write function. I found it was faster to use another Characteristic to act as an ACK mechanism and just write data as quickly as possible. It appears that the Arduino BLE API doesn't allow you to choose and always chooses to wait for the ACKs, so certain applications will be inherently slower.
The end result of my experiment on the ESP32 was that I was able to get a theoretical max data rate of about 57K bytes per second. On phones (iOS/Android) and other BLE devices (Nano 33 BLE), that number would probably be in the 18-23KBs range due to the inability to change the MTU size and/or the write type.
BLE Scanner Apps
There are a number of apps on cell phones and PCs which allow you to see nearby BLE devices and the GATT UUIDs that they advertise. These have proved to be especially useful for some of my experiments. These apps however, are not all created equal. I've got three different ones installed on my Android phone and not all of them are able to connect to every BLE device I've worked with. The one that seems to work best is the Nordic nRF Connect app (shown below). To figure out how to communicate with the BLE thermal printer, the Scanner app showed me all of the services and their characteristics (see image below). The ones with the WRITE flag looked promising and I was able to experiment directly in the app to see which characteristic would send data to the printer.
Thermal Printer
BLE Tiles / Beacons
These were one of the first products to appear after the BLE standard was created. Their original purpose was to be small sensors which could run for a year or more on a coin cell and periodically transmit some useful information. The first products which targeted consumers use BLE as a locator. A 'tile' is placed on a keychain or in a purse. With an app on your cell phone, you can make the tile beep, light up or detect when it goes out of range of your phone. The beacon pictured below is pretty typical of this type. It has a single push button, LED and piezo buzzer inside.
I was able to figure out how to communicate with it by using the BLE scanner and testing the advertised Services and Characteristics. Many of these BLE products don't really try to hide their functionality from prying eyes, but in the case of products which do - there are other ways of reverse engineering BLE communication. The simplest is probably to enable Bluetooth logging on your cell phone and analyze the log with a program like Wireshark. Another, more complex option is to use a smart radio receiver to listen to the raw radio packets.
Gaming Controls
For ten years, I earned a living writing emulators for coin-op and console games. I still have some nostalgia (and code) from that era, so gaming on embedded devices still holds some interest for me. One of the things I've wanted to do is use wireless controllers on the embedded boards which have built-in Bluetooth. One of my favorite old BT controllers is the SteelSeries:Free (photo below).
It's small, but comfortable and the BT (classic) implementation is good. It supports HID keyboard, gamepad and SPP (Serial Port) mode. It has two analog sticks (that produce analog values), a D-Pad and lots of buttons. I was able to figure out the serial port protocol (which mimics HID status reports) and get it working on Android, Linux, Windows and MacOS. I recently got it working on the ESP32 too because the ESP32 supports enough of the classic BT spec to function. Since it's classic BT (and no longer sold), it's not perfectly relevant for this post, but it provides some context. I've been looking for BLE gamepads (if such a thing exists) and haven't found a whole lot (yet). I did find two BLE controls which are considered "Wireless VR controllers".
On the left is the "MINI PLUS" and on the right is the "ACGAM R1". Both are available from multiple sellers under various names and cost $2-6 each from China. These both have analog sticks, yet only output D-Pad style values (centered, 100% up, 100% down, etc). They have various "modes" which mimic multimedia controls or gaming controls. They both implement BLE HID (human interface device) protocol. The embedded BLE API implementations that I've seen don't properly (or at all) implement an HID host to connect to these controls. It takes a decent amount of code to properly create an HID host which can recognize different device classes and properly parse the data "reports" that they generate. In my case I decided to take a shortcut and just write specific code to parse the specific reports generated by these controls. My first try to connect to these controls was on the ESP32. I was able to find the HID Service (0x1812) and the HID report characteristic, but I wasn't receiving any data when the control state changed. I probed them with the BLE Scanner app and saw that there were numerous Characteristics that all had the same UUID. I didn't know this before, but there's nothing stopping a BLE device from advertising multiple Services or multiple Characteristics that all use the same UUID. The only problem with this is that the current ESP32 BLE library for Arduino (1.0.4) doesn't support enumerating multiple Characteristics with the same UUID. This meant that I couldn't use these controls with any ESP32 Arduino device. I then tried to connect them to the Nano 33 BLE and surprisingly, it does support enumerating multiple Characteristics with the same UUID. I told it to notify my code on all of them and was able to get the HID report data from one of the many identical UUIDs. Here's my library running on the Nano 33 BLE
I used this library to control a little motorized car with the BLE controls too:
Future topics...
Adafruit vs Arduino vs ESP32 API
BLE Failures
First, some terminology:
BLE vs. Classic BT
The Bluetooth wireless standard was announced in 1998 and was meant to be a wireless personal area network (PAN) which had shorter range / lower power compared to WiFi. This is now referred to as "Classic" Bluetooth. Several revisions occurred over the years which added higher speed data, UART emulation and other features. BT 4.0 is when BLE (Bluetooth Low Energy) was added in 2010. This has also been called "Bluetooth smart". BLE is a completely different protocol which shares the same physical radio. BLE was designed to target a new set of devices such as information beacons/tiles, health/fitness and entertainment. You can read more about it here. Both Classic BT and BLE are now available as integrated features in some microcontrollers (e.g. ESP32, nRF5x).BT 2.0/4.x/5.x - The Bluetooth spec is updated periodically with new features and a new revision number. These numbers don't mean a whole lot to the general public and even less when they're placed in advertisements for devices. For example, a wireless keyboard which advertises support for BT 4.0 tells you very little. 4.0 is when BLE was added, but that doesn't mean it's a BLE keyboard. If it claimed a number lower than 4.0, then you could be confident that it only supported classic BT. A few device manufacturers are smart enough to include "BLE" or "BT Smart" with their advertisements, but the majority leave you guessing.
GATT (Generic Attribute Profile) - a software protocol based on universally unique identifiers (UUIDs). The BLE spec defines many UUIDs for specific purposes, but you can also create your own proprietary ones.
Central - The BLE name for what is normally called the client. The Central initiates commands, connection requests and responses.
Peripheral - The BLE name for what is normally called the server. The Peripheral advertises information and optionally allows connections.
Characteristic - a data value transferred between Central and Peripheral. A characteristic can be set up to notify the software when its value has changed. You can read or write one byte or many at a time, this depends on the what the Characteristic is offering. A safe maximum for local buffering of this data is probably 512 bytes (on embedded devices)
Service - a collection of related Characteristics that work together to provide specific functionality.
Descriptor - provides additional information about a Characteristic. Sometimes used for things like units (temperature), or a switch to enable/disable a feature.
MTU (Maximum Transmission Unit) - the largest packet size that can be sent
My Experiences (aka the trivia)
The following sections will document bits of info I learned by trying various experiments and encountering and then overcoming (most) issues. My work environment for all of this:- MacBook Pro 15 (mid 2018) running MacOS Catalina (10.15.2)
- Arduino IDE 1.8.10
- Samsung Galaxy S9+
- Various ESP32 dev boards (TTGO T-Display, Mike Rankin's color coin cell, M5Stick-C, etc)
- Arduino Nano 33 BLE Sense
BLE API
Each microcontroller provider has decided to implement Bluetooth in their own unique way, with their own API. Even though the BT specification is pretty specific, each vendor hasn't implemented the spec in its entirety nor have they been consistent about it. The major players for embedded BT APIs are Espressif Systems (ESP32), Arduino (Arduino Nano BLE 33), Nordic Semiconductor (nRF52 SDK), Adafruit (several) and Particle. The APIs are different enough that writing code which needs to run on multiple platforms consists of writing multiple versions of the code that are quite different. The Arduino ecosystem now supports a large number of embedded boards that were not created by Arduino, yet can share most of the same libraries. Unfortunately this isn't the case with Bluetooth support. For example, I've decided to focus on the ESP32 and Nano 33 BLE, both of which are fully supported in the Arduino IDE. Being part of the same development environment, however doesn't mean that they share the same software support. The BLE API for each of them is vastly different in terms of class/method names and capabilities. The only project that I've gotten to run identically on both is my Thermal Printer Library and only because it uses the simplest of features of BLE (scan and write to Characteristic). You'll probably have to choose sides and stay with that choice as you create more complicated projects because it will get even harder to switch and rewrite/relearn everything.
File Transfer
My first BLE experiment involved trying to send data (a file) as quickly as possible from one microcontroller to another. I used two ESP32 modules I had on hand. One of the downsides of BLE versus classic BT is the effective data rate is much lower because BLE saves power by adding pauses in between packets to turn off the transmitter. These pauses and the small packet size (default = 23 bytes) means that data doesn't go anywhere near as fast as the 2Mbs radio speed would lead you to believe. I discovered there are 2 main issues that you have some control over to determine how fast data can be transferred:
MTU Size - BLE can request a larger MTU as part of its connection negotiation. There isn't really a max MTU size, it's up to the software implementation, but there is a limit of 257 bytes per physical data block transmitted. An MTU larger than this will be broken up into multiple PDUs. Transmitting more info per block increases the speed of data transfer. The peripheral may reject/reduce your MTU size change request. Your ability to read/write large blocks of data at a time depends mostly on the BLE API implementation. A write request that's larger than the MTU payload size (whatever is currently agreed upon) will be broken up into multiple blocks.
WRITE / WRITE_NO_RESPONSE - Most devices I've seen allow writing data with either option. A "WRITE" will wait for an ACK (acknowledgement) packet from the receiver before sending more data. "WRITE_NO_RESPONSE" as the name implies, doesn't wait for a response (none is sent). As you can imagine, it's much faster to transmit data when you don't have to wait for an ACK. The ESP32 allows you to specify what type with a true/false flag in the write function. I found it was faster to use another Characteristic to act as an ACK mechanism and just write data as quickly as possible. It appears that the Arduino BLE API doesn't allow you to choose and always chooses to wait for the ACKs, so certain applications will be inherently slower.
The end result of my experiment on the ESP32 was that I was able to get a theoretical max data rate of about 57K bytes per second. On phones (iOS/Android) and other BLE devices (Nano 33 BLE), that number would probably be in the 18-23KBs range due to the inability to change the MTU size and/or the write type.
BLE Scanner Apps
There are a number of apps on cell phones and PCs which allow you to see nearby BLE devices and the GATT UUIDs that they advertise. These have proved to be especially useful for some of my experiments. These apps however, are not all created equal. I've got three different ones installed on my Android phone and not all of them are able to connect to every BLE device I've worked with. The one that seems to work best is the Nordic nRF Connect app (shown below). To figure out how to communicate with the BLE thermal printer, the Scanner app showed me all of the services and their characteristics (see image below). The ones with the WRITE flag looked promising and I was able to experiment directly in the app to see which characteristic would send data to the printer.
Thermal Printer
I've been on the lookout for interesting BLE devices and I was alerted to the existence of Bluetooth thermal printers. I searched the internet and couldn't find any existing Arduino projects to communicate with them, so I decided to figure it out myself and share the code if I came up with something useful (my usual MO). I found this one and, as usual, it advertises itself as supporting BT 4.0. I took a chance and hoped that it would be BLE and luckily it is. I can't vouch for its longevity, but I'm very pleased with both the hardware and firmware of this printer.
The ergonomics+aesthetics are also quite good. The power indicator LED (green in the photo above), turns blue when there is an active BLE connection. It comes with a removable 7.4V 1500mAh LiIon battery that's built into a snap-in pack on the bottom. The microUSB charging port also presents a virtual COM port which can be used to print. It came with very little documentation, but I confirmed that it follows this printer spec. As you can see in the BLE Scanner image above, it advertises 3 main services. I tried using the scanner app to write the bytes "Hello\r" to each service's WRITE characteristic and they all worked. I'm not sure why it advertises three identical services with different UUIDs, but I assume it's because some other printers have chosen one and this printer wanted to be compatible with lots of software. After getting some text and graphics to print, I noticed that some of my graphics were missing/corrupted. I remembered that I had been using the WRITE (without response) and figured I was sending data faster than the printer could handle it and it wasn't getting buffered. I switched to write (with response) and the printing came out perfect, but was 'stuttering' - the print head was stopping and starting instead of printing smoothly. I then tried using the non-response write and adding my own delay. This turned out to be the right (in some sense of the word) way to do it. These kinds of kludges seem to be pretty commonplace when it comes to Bluetooth. I'm almost finished with my Arduino library to control the printer. I used #ifdef to keep most of the code in common, yet allow it to run on both ESP32 and Arduino Nano 33 BLE. This involved more than just using different functions. The Arduino BLE code was 'unable' to see various BLE Services that the ESP32 was able to see. Originally I picked a specific UUID to communicate with the printer only to find out later that the Nano 33 doesn't even find that Service. Luckily all 3 Services on the printer behave the same way, so the Nano 33 code connects to a different Service UUID. Below is a short video of my demo sketch (from my library repo) running on the Nano 33 BLE:
The ergonomics+aesthetics are also quite good. The power indicator LED (green in the photo above), turns blue when there is an active BLE connection. It comes with a removable 7.4V 1500mAh LiIon battery that's built into a snap-in pack on the bottom. The microUSB charging port also presents a virtual COM port which can be used to print. It came with very little documentation, but I confirmed that it follows this printer spec. As you can see in the BLE Scanner image above, it advertises 3 main services. I tried using the scanner app to write the bytes "Hello\r" to each service's WRITE characteristic and they all worked. I'm not sure why it advertises three identical services with different UUIDs, but I assume it's because some other printers have chosen one and this printer wanted to be compatible with lots of software. After getting some text and graphics to print, I noticed that some of my graphics were missing/corrupted. I remembered that I had been using the WRITE (without response) and figured I was sending data faster than the printer could handle it and it wasn't getting buffered. I switched to write (with response) and the printing came out perfect, but was 'stuttering' - the print head was stopping and starting instead of printing smoothly. I then tried using the non-response write and adding my own delay. This turned out to be the right (in some sense of the word) way to do it. These kinds of kludges seem to be pretty commonplace when it comes to Bluetooth. I'm almost finished with my Arduino library to control the printer. I used #ifdef to keep most of the code in common, yet allow it to run on both ESP32 and Arduino Nano 33 BLE. This involved more than just using different functions. The Arduino BLE code was 'unable' to see various BLE Services that the ESP32 was able to see. Originally I picked a specific UUID to communicate with the printer only to find out later that the Nano 33 doesn't even find that Service. Luckily all 3 Services on the printer behave the same way, so the Nano 33 code connects to a different Service UUID. Below is a short video of my demo sketch (from my library repo) running on the Nano 33 BLE:
The nRF24 Hack
Nordic Semiconductor nRF24 modules are very popular due to their low cost and ease of use. They allow you to add wireless communication to your MCU projects with minimal effort.
The nRF24 physical radio/modulation method happens to be the same as Bluetooth and the channel range overlaps as well. BLE protocol uses channel hopping to minimize interference from other 2.4Ghz devices, while the nRF24 does not. However, a clever programmer figured out that it's possible to transmit BLE advertisements (like a beacon) from an nRF24 module. The channel range is incomplete and it cannot connect or receive data, but if you need to send out small packets of advertisement data, it works. I verified this on a project of mine as an interim solution. The downside is that your MCU must manually channel hop for an extended period of time to ensure that one or more of the packets is received. This takes much more power and time than a dedicated BLE device would normally need.
Nordic Semiconductor nRF24 modules are very popular due to their low cost and ease of use. They allow you to add wireless communication to your MCU projects with minimal effort.
The nRF24 physical radio/modulation method happens to be the same as Bluetooth and the channel range overlaps as well. BLE protocol uses channel hopping to minimize interference from other 2.4Ghz devices, while the nRF24 does not. However, a clever programmer figured out that it's possible to transmit BLE advertisements (like a beacon) from an nRF24 module. The channel range is incomplete and it cannot connect or receive data, but if you need to send out small packets of advertisement data, it works. I verified this on a project of mine as an interim solution. The downside is that your MCU must manually channel hop for an extended period of time to ensure that one or more of the packets is received. This takes much more power and time than a dedicated BLE device would normally need.
BLE Tiles / Beacons
These were one of the first products to appear after the BLE standard was created. Their original purpose was to be small sensors which could run for a year or more on a coin cell and periodically transmit some useful information. The first products which targeted consumers use BLE as a locator. A 'tile' is placed on a keychain or in a purse. With an app on your cell phone, you can make the tile beep, light up or detect when it goes out of range of your phone. The beacon pictured below is pretty typical of this type. It has a single push button, LED and piezo buzzer inside.
I was able to figure out how to communicate with it by using the BLE scanner and testing the advertised Services and Characteristics. Many of these BLE products don't really try to hide their functionality from prying eyes, but in the case of products which do - there are other ways of reverse engineering BLE communication. The simplest is probably to enable Bluetooth logging on your cell phone and analyze the log with a program like Wireshark. Another, more complex option is to use a smart radio receiver to listen to the raw radio packets.
Gaming Controls
For ten years, I earned a living writing emulators for coin-op and console games. I still have some nostalgia (and code) from that era, so gaming on embedded devices still holds some interest for me. One of the things I've wanted to do is use wireless controllers on the embedded boards which have built-in Bluetooth. One of my favorite old BT controllers is the SteelSeries:Free (photo below).
It's small, but comfortable and the BT (classic) implementation is good. It supports HID keyboard, gamepad and SPP (Serial Port) mode. It has two analog sticks (that produce analog values), a D-Pad and lots of buttons. I was able to figure out the serial port protocol (which mimics HID status reports) and get it working on Android, Linux, Windows and MacOS. I recently got it working on the ESP32 too because the ESP32 supports enough of the classic BT spec to function. Since it's classic BT (and no longer sold), it's not perfectly relevant for this post, but it provides some context. I've been looking for BLE gamepads (if such a thing exists) and haven't found a whole lot (yet). I did find two BLE controls which are considered "Wireless VR controllers".
I used this library to control a little motorized car with the BLE controls too:
Future topics...
BLE Failures
Thanks !! i had the same issue with the same controller and u saved my day
ReplyDelete