The Dead End
A couple of months back, I was deeply invested in automating functions on my Mi Band. I really wanted to use smart replies to respond to messages right from my wrist. I quickly realized nothing of the sort had been properly implemented for this specific band.
Upon reading about how smart replies are generated natively in Android, I hit a massive roadblock. The suggested replies generated by messaging apps are heavily sandboxed. They are simply not accessible to third-party applications to intercept and forward to a smartwatch. It seemed like a dead end.
The On-Device Discovery
Then I had a realization. Smart replies are generated locally using on-device machine learning. If apps like WhatsApp can call the API to generate context-aware replies, I could potentially query that exact same API to generate my own set of suggested replies.
I dove into the documentation to understand how these replies are constructed.
"An app passes a conversation context to the on-device ML Kit to get suggested replies. The ML Kit uses the conversation to try to find relevant replies to the last message received from the remote user. If it is successful, it returns a set of 3 smart replies."
My new goal was clear. I needed to build a custom Android application that could take a single message string, feed it to Google's ML Kit, and return the three smart replies as a usable variable. Because I would not have access to the user's entire conversation history, I decided to simplify the process and feed it only the very last received message.
Building My First Android App
I wanted this tool to integrate perfectly with Tasker. This meant I had to build an official Tasker Plugin. Having never created an Android application from scratch before, this was a huge learning curve.
After countless hours reading through documentation, studying the ML Kit demo code provided by Google on their GitHub, and dissecting the sample code for creating Tasker plugins by João Dias, I finally got it working.
The result is "Smart Reply", a standalone Tasker Plugin that takes a string as input, processes it through ML Kit, and outputs the suggested replies. It saves the output as a string in the %reply variable with each suggestion on a new line. If the ML Kit is unable to generate a response, the plugin smartly falls back and sets the variable to "No smart replies found".
The Automation Flow
Once the plugin was built, I had to stitch the whole ecosystem together using a combination of apps. I used Tasker, AutoNotification, Notify for Mi Band, and my new Smart Reply plugin.
1. Interception
I use AutoNotification to intercept incoming notifications from any messaging app and save the actual message content into a Tasker variable.
2. Triggering
Once I receive a message, I use the Notify app to capture media control button clicks from the Mi Band. These clicks are broadcasted as intents into Tasker. For example, pressing the volume increase button on the band triggers the smart reply task.
3. Generation
Inside Tasker, once the intent is received, I pass the saved message content directly into my Smart Reply plugin. The plugin then queries ML Kit.
4. Delivery & Selection
The plugin returns the replies as an array with a newline character as a delimiter. These replies are sent back to my Mi Band as a custom notification. I then map the media buttons on the band to select the desired response. The Play button selects the first reply, the Previous button selects the second, and the Next button selects the third.
Fallback System (Quick Replies)
In the event that the plugin does not find any suitable smart replies, I built a fallback mechanism. We can reply to the message from a hardcoded default list of messages like "Yes", "No", and "Ok". This works similarly to the native Quick Reply feature of the Notify app.
The Final Outcome
It was an incredible feeling to bridge on-device machine learning with raw Tasker automation. While the replies might not be perfectly context-aware like native apps since I only analyze the very last message instead of the whole thread, it works brilliantly for quickly firing off responses directly from my wrist.
Action Video
You can watch a quick demo of the smart replies in action on YouTube:
▶️ Watch Demo on YouTubeGetting Started
- In your Tasker task, choose the plugin option and select "Reply".
- Input the message variable to get suggested smart replies.
- The output is saved in the
%replyvariable as a string with each suggestion on a new line. - If you want to view the full automation setup, you can import my Smart Reply Tasker Project.
Credits & Resources
A huge thanks to Google's open-source ML Kit examples and João Dias for providing the framework for Tasker plugins.
View Project on GitHub