Robot Engage Blog Buy Now
Blog

Misty Community Forum

Training Mission: Coding Misty to Recognize Emotions

Mission Objective: Learn how Misty can use third-party APIs and cloud services to respond to emotions she detects in the faces of people she photographs
Mission Type: Training
Launch Date: 12/20/2019
End Date: Ongoing

Mission Overview

Whether it’s navigating her environment or engaging with humans, the data you capture with Misty’s on-board sensors enables a huge variety of interactions. Plug that data into third-party APIs, and the possibilities abound. Misty’s last Training Mission of 2019 explores just one of those possibilities: coding Misty to analyze and respond to your emotional state.

In the first part of this Mission, we learn how to use Misty’s camera APIs to take pictures of the people she sees. We then explore some of the options you have for processing those images off the robot, so you can analyze the image and extract details about the faces they contain.

When you understand the basics of taking pictures and using third-party services, you can decide how you want Misty to respond to the emotions she detects. Bring your ideas to life and share the results in this week’s challenge!

Misty’s Camera APIs

Misty uses the RGB camera in her visor to detect, learn, and recognize the faces she sees. She can also uses this camera to capture photos and record videos of her environment.

The TakePicture command in Misty’s API lets you use this camera to take pictures in your skills . Misty saves the pictures she takes with this command to the SDcard on her Android device. You can retrieve these pictures with the GetImage command, or you can configure a TakePicture callback to do something with your picture data as soon as it’s ready.

Misty’s TakePicture command accepts the following arguments (be sure to see the docs for more details):

  • base64 - returns the image data as a Base64 string
  • fileName - name to assign to the image file for the captured photo.
  • width - desired image width (in pixels)
  • height - desired image height (in pixels)
  • displayOnScreen - displays the captured photo on Misty’s screen
  • overwriteExisting - whether Misty should overwrite an image with the same filename as the captured photo, if one already exists on her local storage.

In the following sample from a JavaScript skill, we use the TakePicture command to publish a picture to the web using the Imgur API. Note how we use BumpSensor events to trigger this sequence of actions, so that Misty takes a picture whenever one of her bump sensors is pressed.

misty.AddReturnProperty("BumpSensor", "IsContacted");
misty.RegisterEvent("BumpSensor", "BumpSensor", 10, true);

function _BumpSensor(data) {
    var isPressed = data.AdditionalResults[0];

    // If isPressed is true, takes picture
    if (isPressed) {
        misty.TakePicture("MyPic", 375, 812, false, true)
    }
}

function _TakePicture(data) {
    var base64String = data.Result.Base64;
    misty.Debug("Picture taken");
    uploadImage(base64String);
}

function uploadImage(imageData) {
    var jsonBody = {
        "image": imageData,
        "type" : "base64",
        "album": "<your-imgur-album-hash>" // You'll need to get your own hash from Imgur
    };

    // Update this call with your own bearer token
    misty.SendExternalRequest("POST", "https://api.imgur.com/3/image", "Bearer", "<your-access-token>", JSON.stringify(jsonBody), false, false, "", "application/json", "_imageUploadResponse");
}

// Returns the public URL for your image
function _imageUploadResponse(responseData) {
    misty.Debug(JSON.parse(responseData.Result.ResponseObject.Data).data.link);
}

Analyzing Portraits for Emotions

Once Misty’s taken your picture, we can pass it off to a face recognition API that offers the option to return details about the emotional state of the face (or faces) in the picture. There are dozens of services to choose from, including:

We don’t officially endorse any one service, and most of them should play nice with your skills or robot applications. Many offer a free trial period, so you can quickly start prototyping skills without signing up for a payment plan. Each one works in a slightly different way; be sure look for tutorials in the documentation for the service you want to use before you try to integrate a request into your skill or robot application.

Note: As a Misty developer, you are responsible for understanding how the skills and applications you build share data with the cloud. As always when sharing your data with a third party, be sure to familiarize yourself with how each service uses the images you share.

As an example of how this might work, this JavaScript skill code below uses the Face++ API to have Misty imitate the facial expression of the person she photographs. It builds on the Imgur sample shared above; first, Misty publishes the captured photo to Imgur, and then she sends the URL for that image to the Face++ API. The Face++ API returns an emotion object that provides a value rating for each of seven different emotions (anger, disgust, happiness, etc.) In the skill, we parse these values to determine the emotion with the highest value, and we code Misty to express an emotion that matches.

To try it out on your robot, sign up for accounts with Imgur and Face++. Then replace the tokens, album hash, and “secrets” in the code below with the credentials and information for your own accounts.

misty.AddReturnProperty("BumpSensor", "IsContacted");
misty.RegisterEvent("BumpSensor", "BumpSensor", 10, true);

function _BumpSensor(data) {
    var isPressed = data.AdditionalResults[0];

    // If isPressed is true, takes picture
    if (isPressed) {
        misty.TakePicture("MyPic", 375, 812, false, true)
    }
}

function _TakePicture(data) {
    var base64String = data.Result.Base64;
    misty.Debug("Picture taken");
    
    // Upload the image to Imgur!
    uploadImage(base64String);
}

function uploadImage(imageData) {
    var jsonBody = {
        "image": imageData,
        "type" : "base64",
        "album": "<your-album-hash>"
    };

    var imgurToken = "<your-token>"
    misty.SendExternalRequest("POST", "https://api.imgur.com/3/image", "Bearer", imgurToken, JSON.stringify(jsonBody), false, false, "", "application/json", "_analyzeImage");
}

function _analyzeImage(responseData) {
    
    var link = JSON.parse(responseData.Result.ResponseObject.Data).data.link;
    var facePlusPlusKey = "<your-face++-key>"
    var facePlusPlusSecret = "<your-face++-secret>"
    
    // Send URL of image to Face++ API
    misty.SendExternalRequest("POST", "https://api-us.faceplusplus.com/facepp/v3/detect?api_key=" + facePlusPlusKey + "&api_secret=" + facePlusPlusSecret + "&image_url=" + link + "&return_attributes=emotion", null, null, null, false, false, null, "application/json", "_analyzeImageResponse");
}

// Handle Face++ response data
function _analyzeImageResponse(responseData) {
    var emotions = JSON.parse(responseData.Result.ResponseObject.Data).faces[0].attributes.emotion;
    misty.Debug(JSON.stringify(emotions));
    var maxEmotion = Object.keys(emotions).reduce((a, b) => emotions[a] > emotions[b] ? a : b);

    // This is what Face++ thinks you're feeling the most
    misty.Debug(maxEmotion);

    // Misty imitates your emotions
    switch(maxEmotion) {
        case "anger":
            misty.DisplayImage("e_Anger.jpg");
            misty.PlayAudio("s_Anger.wav")
            break;
        case "disgust":
            misty.DisplayImage("e_Disgust.jpg");
            misty.PlayAudio("s_Disgust.wav")
            break;
        case "fear":
            misty.DisplayImage("e_Fear.jpg");
            misty.PlayAudio("s_Fear.wav")
            break;
        case "happiness":
            misty.DisplayImage("e_Joy.jpg");
            misty.PlayAudio("s_Joy.wav")
            break;
        case "neutral":
            misty.DisplayImage("e_DefaultContent.jpg");
            misty.PlayAudio("s_Acceptance.wav")
            break;
        case "sadness":
            misty.DisplayImage("e_Sadness.jpg");
            misty.PlayAudio("s_Sadness.wav")
            break;
        case "surprise":
            misty.DisplayImage("e_Surprise.jpg");
            misty.PlayAudio("s_DisorientedConfused.wav")
            break;
        default:
            misty.Debug("Sorry, I don't know what you're feeling. Take another picture?");
            break;
    }
}

Note: Running this code as-is updates a public Imgur album with the pictures you take. To keep your moody portraits a little more private, try coding Misty to send the base64 data straight to Face++, without using Imgur as the middle-man.

Coding Misty’s Response

Once your skill is synced with a third-party service, you can code Misty to react in whatever way you’d like. The example above shows how she might show a very simplistic empathetic response (she plays a sound and changes her eyes to mirror the emotion she detects). To take this a step further, Misty could try to comfort you when you’re feeling sad, or play a happy song when you smile at her. You could have her back away if she detects anger, and add in head and arm movement for more lifelike animations.

The code for this functionality doesn’t have to live inside a BumpSensor event callback. You could make all this a bit more lifelike by triggering Misty’s “detect emotion” functionality with a FaceRecognition event, instead. You might decide to incorporate speech capture, audio localization, and capacitive touch for sentiment analysis and robot interaction that’s even more nuanced.

Once you’ve got an idea for a skill of your own, head over to the challenge portion of this Mission and start building!

Mission Support

Hit a roadblock? Find a bug? Have feedback? Let us know! As Misty’s earliest developers, you lead the journey to making platform robots a reality. Your creativity makes all this possible, and we’re eager to hear your feedback. Reply to this post to start a conversation with a member of the Misty team.

Hi Johnathan & crew. Great mission. I decided to skip Imgur and send the image direct to Face++. Was able to get the Image_URL option to work with a tinyURL embedded in the POST, but am not able to pass images using the Image_file or Image_base64 options. It looks like the variable I pass into the POST is undefined and not carried forward after being set in the _TakePicture function. Can someone show me how you pass the base64 image successfully from _TakePicture function into the SendExternalRequest POST? Tried global variables but am doing something wrong. Hope someone else has completed this mission already. Thx.

Interesting; I would expect that callback to work in a similar way to the function used in the simple “upload-to-imgur” sample. I’ll see if I can dig into this myself sometime in the next few days, if a member of the community doesn’t beat me to it :slight_smile:

I am trying to upload the image to google cloud storage but it is not showing up there, nor I can see any errors in console. Am I missing any parameters or arguments?

I also tried imgur but I am confused with the token secret and album hash. Is there any way I can just upload the image publicly and use that link with other API to analyse the image

I haven’t had the bandwidth to scrape together a working sample of this yet. Any chance you’ve had luck here, @Cgkocks?

I’m not familiar with google cloud storage myself; afraid I can’t be much help there. One option for troubleshooting is to try your parameters for SendExternalRequest using the REST API endpoint in a client like Postman. This makes it easier to adjust parameters until you have things working just right, and then you can copy those parameters into your method call in the skill code.

There may be a way to upload the image publicly, but I’m not sure of the steps. I’d suggest checking out the Imgur API docs to see if you can find what you need there.

Is there any way to get the image directly as jpg or png instead of base64 string?

Just to confirm, is your question about sending the image file with the SendExternalRequest method?

Yes, It is. I think we need image object to upload it directly to GCS.

If you’re attempting to upload to GCS via this REST API, it doesn’t look like that’s possible from a JavaScript skill. That Google endpoint appears to require a reference to the path for the object you want to upload, and in the current implementation of SendExternalRequest, there’s no way to reference the local path of an asset.

If you’re comfortable trying out .NET SDK, however, you can reference files on Misty’s local storage, and then you could make use of an external library for making HTTP requests, which would allow you to upload directly to GCS.

No new updates yet. Will keep trying.

I spent an hour on this today, with no great results.

Passing a base64 string as a query parameter makes the URL too long for a valid request, and while the Face++ detect API does appear willing to accept a base64 string or binary file you send with a content-type of multipart/form-data, I’ve had no success sending the same base64 string in an application/json payload formatted as {"image_base64": "<your_base64_string>"}. The current implementation of the SendExternalRequest method doesn’t support multipart/form-data, and so it appears we are a bit stuck.

It’s not entirely a dead end, however; the .NET SDK allows you to make HTTP requests without relying on Misty’s SendExternalRequest command, and the requests you issue shouldn’t have the same content-type limitations. If you are comfortable trying out C#, this should be a fairly straightforward request to implement :slight_smile:

@johnathan I tried the sample code you have provided under mission section. But when I debug the code it does not return any URL to pass to face++ api. any idea what could be the reason?

Without seeing the code, I’d suggest double-checking that the album hash and API token you’re using are correct. If they are, then you may want to put an additional debug line in the callback function to make sure that function is getting triggered. If that function is not getting triggered, then our SendExternalRequest method isn’t getting a response, which means there could be a different issue with the request.