Home Community Blog Buy Now
Blog

Misty Community Forum

Tensorflow on Misty

Is it possible to upload tensorflow model on misty? The objective is to implement machine learning algorithms using tensorflow.

Hello and welcome to the community!
At the current time you cannot load your own model onto the robot.
Your best option for now is to write a skill that retrieves the data from the camera and sends it to an offline application that processes the data and sends it back to the skill to be handled appropriately.

1 Like

Can you tell us a little more about your application?

For example, if you are willing to write the skill in C#, you might be able to compile with one of the ports of TensorFlow to C#: https://github.com/migueldeicaza/TensorFlowSharp or https://github.com/SciSharp/TensorFlow.NET. In that case, it might be feasible to run everything onboard.

There is also a port of TensorFlow to JavaScript (JS) at https://www.tensorflow.org/js. It might be feasible to work this into a Misty JS skill and run everything onboard.

I am happy to hack on this with you if you are going to share your work as open source. No pressure :wink:

1 Like

I would recommend against doing that since the skill system runs on the 410 which is woefully underpowered for something like this.

You could also write your own Android app that uses tensorflow and a C# skill to communicate with it :slight_smile:

If I were to write an Android app, why would I use the Misty skill system at all?

This suggestion of writing low-level code (i.e., outside the existing Misty frameworks) comes up from time to time to get around limitations of the existing Misty frameworks; e.g.,

However, not only is it “not officially supported”, but I vaguely recall agreeing not to reverse engineer the Misty robot, which is required for understanding enough to actually program at that level.

In case you want to control the rest of the robot (like locomotion, display etc). Also, we mostly suggest writing a targeted app not that you replace all the functionality with your own software.

Don’t think any reverse engineering would be needed. Again, not saying you replace all the controls with your software (thought as far as I’m concerned you’re more than welcome to do that if you so desire). In this particular case, you’d write an app that would use tensorflow lite, load your own model and do whatever special processing you want, and then you’d write a skill to communicate with this app over the local network to control robot and do whatever.

Again, the easiest approach is to just write a skill that talks to an off-robot processor. But, if you have the skills you can do it on the robot as well. At the end of the day, the owner of the robot has pretty much full control over the two SOMs on the robot. With that comes some warnings that if you screw up the processors by, for example, flashing over the OS with your own version we’re not liable for that. That being said, I think we’ve been very accommodating of supporting people doing all kinds of things with their robots.

1 Like

As an example, here is some sample C# Code to have Misty take a picture (when clicking front left bumper), send to the Azure Computer Vision APIs, return the tags/objects in the picture as well as a caption summarizing the photo, and then speak the caption.

    private async void BumpCallback(IBumpSensorEvent bumpEvent)
    {
        //Only processing contacted events
        if (bumpEvent.IsContacted)
        {
            switch (bumpEvent.SensorPosition)
            {
                case BumpSensorPosition.FrontLeft:
                    await _misty.ChangeLEDAsync(0, 0, 255); // blue
                    // Play the camera shutter sound.
                    await _misty.PlayAudioAsync("s_SystemCameraShutter.wav", 20);

                    string imageFilename_no_ext = "image_to_analyze";
                    // Take the photo.
                    ITakePictureResponse takePictureResponse = await _misty.TakePictureAsync(imageFilename_no_ext, false, true, true, 1024, 768);
                    string imageFilename = imageFilename_no_ext + ".jpg";

                    HttpClient client = new HttpClient();
                    // the Analyze method endpoint
                    string UriBase = computerVisionEndpoint + "vision/v2.1/analyze";
                    // Request headers.
                    client.DefaultRequestHeaders.Add(
                        "Ocp-Apim-Subscription-Key", computerVisionKey);
                    // Request parameters. A third optional parameter is "details".
                    // The Analyze Image method returns information about the following
                    // visual features:
                    // Categories:  categorizes image content according to a
                    //              taxonomy defined in documentation.
                    // Description: describes the image content with a complete
                    //              sentence in supported languages.
                    // Color:       determines the accent color, dominant color, 
                    //              and whether an image is black & white.
                    string requestParameters =
                        "visualFeatures=Categories,Description,Color";

                    // Assemble the URI for the REST API method.
                    string uri = UriBase + "?" + requestParameters;
                    HttpResponseMessage response;
                    // Read the contents of the specified local image
                    // into a byte array.
                    byte[] byteData = (byte[])takePictureResponse.Data.Image;
                    // Add the byte array as an octet stream to the request body.
                    using (ByteArrayContent content = new ByteArrayContent(byteData))
                    {
                        // This example uses the "application/octet-stream" content type.
                        // The other content types you can use are "application/json"
                        // and "multipart/form-data".
                        content.Headers.ContentType =
                            new MediaTypeHeaderValue("application/octet-stream");

                        // Asynchronously call the REST API method.
                        response = await client.PostAsync(uri, content);
                    }
                    // Asynchronously get the JSON response.
                    string contentString = await response.Content.ReadAsStringAsync();

                    // Sample Response from Azure Computer Vision API
                    // { "categories":[
                    //       { "name":"abstract_","score":0.0078125},
                    //       { "name":"others_","score":0.0234375}]
                    //  ,"color":{"dominantColorForeground":"White","dominantColorBackground":"Black","dominantColors":["Black","White","Grey"],"accentColor":"987834","isBwImg":false,"isBWImg":false},
                    //  "description":
                    //       { "tags":["indoor","living","television","room","table","man","front","screen","flat","furniture","sitting","white","computer","black","large","woman","standing","cat"]
                    //        ,"captions":[{"text":"a flat screen television in a living room","confidence":0.76608372756628806}]},
                    //  "requestId":"83e042d3-f02d-48c4-a659-13e86c03c7b2"
                    // ,"metadata":{"width":768,"height":1024,"format":"Jpeg"}}
                    Newtonsoft.Json.Linq.JObject jsonResp = Newtonsoft.Json.Linq.JObject.Parse(contentString);
                    string imageDescriptionCaption = (string)jsonResp["description"]["captions"][0]["text"];
                    string speakText = $"I spy with my big eyes something that is {imageDescriptionCaption}";
                    await _misty.SpeakAsync(speakText, false, null);
                    //
                    await _misty.ChangeLEDAsync(0, 255, 0); // green
                    break;
            }
        }
    }
2 Likes