DornerWorks

How To Connect Mobile Apps & Voice Assistants to IoT Devices

Posted on August 4, 2020 by Brandon Butler

You want to interface your hardware with a mobile application. The problem sounds simple enough, but it quickly grows into a full-stack development project.

Let’s say you have a light that needs to be controlled via an app. What are the requirements for this app? Turning the light on and off is a good start, but we also need to consider other ways the app will be tied to the light.

  • Should the app report the state of the light?
  • Can the app control the light if they are at another location?
  • Should anyone be able to control the light?

For our demo application, the DornerWorks ESP32 Rapid Prototyping Board (RPB) allows control of the four LED lights on the board. The RPB is tied to a user account limiting LED control to the account holder. State commands to the RPB utilize Wi-Fi. Initial provisioning of Wi-Fi credentials is coordinated between the app and the RPB via a Bluetooth wireless connection.

How does a mobile application send commands to the RPB? While a direct connection via Bluetooth can get the job done, it will only work if the app is near the device. Using an internet connection via Wi-Fi allows the IoT device to receive commands from anywhere. At that point the mobile application is no longer directly communicating with the device. To facilitate communication from any location via Wi-Fi, a cloud backend is required.

At app launch, the user is prompted to login or create an account with an online service. This provides a way to uniquely identify user accounts and tie them to the app. When first connecting to the device, the device’s id is then also associated with the customer’s account. In this way the device is connected to the app.

Diagram on connecting IoT devices to Amazon Alexa-enabled devices, mobile apps, and Google Assistant-enabled devices.
Diagram on connecting IoT devices to Amazon Alexa-enabled devices, mobile apps, and Google Assistant-enabled devices.

We utilize many Amazon Web Services (AWS) for our cloud backend. First, Amazon Cognito isused to handle user account registration and authorization. A database hosted on Amazon allows us to tie customer accounts to device identifiers. Finally, an AWS Lambda function paired with an API Gateway acts as the centralized access point for the mobile application. The mobile application uses this cloud infrastructure to register/login to a user account, associate a device id to a customer account, and send commands to the RPB or read its current state.

To tie the RPB to the web service, we use an efficient IoT protocol known as MQTT. The RPB subscribes to an MQTT broker to listen for commands from the web service. When a user uses their phone to turn the light on, the phone sends a command to the API gateway. This is forwarded to the AWS Lambda function, which then sends a message to the MQTT broker. The result is the hardware device is notified of the directive and turns the light on.

DornerWorks ESP32 Rapid Development Board (RPB) can be controlled by a voice assistant or a mobile app.
DornerWorks ESP32 Rapid Development Board (RPB) can be controlled by a voice assistant or a mobile app.

This is quite a departure from a mobile app directly controlling an IoT device. In fact, the only time our mobile application directly communicates with the hardware is to use Bluetooth to send the initial Wi-Fi connection parameters. Once it has been confirmed to be working, the phone will no longer communicate directly with the hardware.

This setup provides some additional advantages. Once the centralized cloud infrastructure is setup, it becomes trivial to add additional services. For example, an Alexa skill can use the existing APIs to provide voice interaction. We achieved this quickly by creating another Lambda function that acts as an interpreter between the device API and an Alexa skill. Our Alexa skill links to the customer’s account. This lets the skill ask for a list of the user’s devices. The result is that we can ask Alexa to turn on our device, it forwards the command to the device API, which then forwards the command to the MQTT broker. The RPB board sees the command and turns on its light. Adding support for the Google assistant is a nearly identical process.

Connecting Amazon Alexa voice assistant to the app.
Connecting Amazon Alexa voice assistant to the app.

Alexa integration can be taken one step further. Once a user logs into to their account with the Alexa skill. The web service creates an association between the user account and the Amazon account. This allows the web service to send updates directly to the Amazon account. Now when a new device is set up, Alexa automatically knows about it and sends a push notification to the user’s phone. Additionally, we use this information to send general status updates to the Alexa skill.

Using your voice to control an IoT device is great, but what happens when you turn the device on while looking at the status on your phone? There needs to be a way to notify the mobile app that the device state has changed.

One solution is to have the user interface poll the web service every few seconds for device state. This means that the web service needs to cache the expected device state since we cannot easily request the current state from the IoT device. Polling for the device state every few seconds works but this seemed like a waste of resources. It is better to notify the app when the state of the device has changed to reduce server load.

We accomplish this using push notifications.

Turning LEDs on the RPB and off using the mobile app.
Turning LEDs on the Rapid Development Board on and off using the mobile app.

Every time the web service receives a command to change the device state, it sends a silent push notification to all mobile devices associated with the user with the changed state. The result is that the user interface updates automatically.

This project is a demonstration of the feature transformations that can be achieved for your connected product when IoT technology is integrated with a seamless companion mobile app experience. The methods and techniques can be applied to practically any connectable device. Whether that is controlling a light bulb, configuring a washing machine, or monitoring the temperature of a smart oven, customers benefit greatly from and even expect a mobile application to be a key component of your product’s ecosystem and experience.

Brandon Butler
by Brandon Butler
Mobile Application Development Engineer
Brandon Butler is a Mobile Application Development Engineer at DornerWorks.