Robot Not a Dev? Pre-order Now
Not a Dev?

How to programmatically upload and display an image to Misty I

release-notes
misty-i

#1

We are able to upload images through the web API, as well as post an image from the samples, but are not sure how to generate the Byte Array String for a new image file we want to show. Does anybody have experience with this?


#2

Hi @Dalton.Crutchfield! The simplest way I can recommend is to use the API Explorer. If you get a chance, check out this topic and see if it helps. If not, please let us know! Thank you!

https://docs.mistyrobotics.com/onboarding/3-ways-to-interact-with-misty/api-explorer/#using-the-api-explorer-to-obtain-byte-array-strings-from-audio-or-image-file-data


#3

@Donna, thank you for the help.

We are able to use the API explorer to do this, but we were hoping to be able to do this programmatically in the hopes to develop a more complex program which combines many different elements of Misty. Are you able to push us in the right direction on this Byte Array?

Thank you again


#4

Hi @Dalton.Crutchfield! I’m checking on that… Promise I’ll get back to you when I can say something useful. :slight_smile:


#5

Hello again, @Dalton.Crutchfield! One of our team rightfully pointed out that we can look at the API Explorer code to see how this was accomplished. The file of interest is SampleUI.js, and I think what’s doing the conversion work are the FileReader method readAsArrayBuffer() along with byteArray.toString(). I’ve not actually tried it, so you should definitely poke around and see what works.

Here’s the URL for the Explorer download package:
https://s3.amazonaws.com/misty-releases/Misty-0.7/latest/api-explorer.zip

Please let me know if this isn’t helpful or you need more assistance. Thanks!


#6

@Donna Thank you for all of the continued help!


#7

@Donna,

My colleague (@vince) and I have been trying to work with this. We are able to get code (using python) that will compile and seemingly save a file, either audio or image. Unfortunately, It does not play or display the saved file. It seems to us that our encoding of the byte string does not match what it is looking for. We have tried multiple different ways to make a byte string, but it seems to continually have numbers not matching the examples.


#8

Can you share a code example? I have a Misty I, and I can try to reproduce the error.


#9

Thanks for letting us know, @Dalton.Crutchfield (and @vince)!

And thank you, @slivingston! That would be great.

On our end, I’ve checked in with Engineering, and they’ve affirmed that we’re not doing any hidden conversions to the generated data. It should simply be a comma-delimited array of values enclosed in quotes.

If you don’t mind posting a sample of one of your generated byte arrays here, that wouldn’t hurt.

Beyond that, we’ve got a few questions back to you, for when you have a chance:

  • Just as a sanity check, did you try the methods in SampleUI.js, and if so, did they work for programmatic upload and (dis)play with your robot?

  • Have you tried getting your Python-generated byte array to (dis)play anywhere, like on a web page?

Thanks!


#10

@slivingston,

Thank you for the help!

Here is the most recent attempt I was working on for audio. @vince, please add anything if you were doing it differently. The play audio works with the already saved sounds.

def uploadAudio(fileName):
    print("[INFO] Uploading Audio: {}".format(fileName))
    url = "http://{}/api/audio/".format(IPADDRESS)

    byteArray = open(fileName + '.mp3', 'rb').read()
    dataAsByteArrayString = byteArray.decode('cp855')

    payload = {
        "FileName": fileName,
        "DataAsByteArrayString": dataAsByteArrayString,
        "ImmediatelyApply": True,
        "OverwriteExisting": True
    }
    payload = json.dumps(payload)
    print(payload)
    headers = {
        'Content-Type': "application/json",
        'Cache-Control': "no-cache"
        }

    response = requests.request("POST", url, data=payload, headers=headers)
    print(response.text)

def playAudio(fileName):


    print("[INFO] Playing Audio: {}".format(fileName))
    url = "http://{}/api/audio/play".format(IPADDRESS)

    # payload = "{\n\t\"FileName\": \"002-Ahhh.wav\",\n\t\"Volume\": 100\n}"
    payload = "{" + "\n\t\"FileName\": \"{}\",\n\t\"Volume\": 100\n".format(fileName) + "}"
    # payload = "{" + payload + "}"
    # payload = "{\n\t\"FileName\": \"" + fileName + "\",\n\t\"Volume\": 100\n}"
    print("[INFO] Audio PayLoad: {}".format(payload))
    headers = {
        'Content-Type': "application/json",
        'Cache-Control': "no-cache"
        }

    response = requests.request("POST", url, data=payload, headers=headers)
    print(response.text)

#11

@Donna

Thank you again. I will have @vince give a couple examples of the byte strings we were getting. We were able to get the upload and play to work through the web client and the existing api, and have gotten the ability to programmatically display images already saved, and play already saved audio. It is a good idea trying to get the image/ audio to display elsewhere. So far we had it print the byte array and were noticing the differences between the array from the existing api and our byte array using the same photo.


#12

Thank you for your help.

Here is my code for getting the byte array by using Python:


with open(image_name, "rb") as image_file:
    encoded_file = base64.b64encode(image_file.read())
    encoded_string = np.frombuffer(encoded_file, dtype=np.uint8)
    print encoded_string

Here is the byte array I’ve got from above:

I used the same image to test by using SampleUI.js, and I got the byte array below (Uint8Array one):

What I got in Python is totally different with SampleUI.js. And the SampleUI.js one is correct. I just confused that how we can get the correct byte array both image and audio file in Python? Thank you!


#13

is base64 encoding necessary?

SampleUI.js does not do it, unless I missed something.


#14

should the trailing / be removed? instead,

http://{IPADDRESS}/api/audio

#15

@slivingston @vince @Dalton.Crutchfield Excellent note, Scott! We do NOT base64 encode the byte array data.


#16

@vince @Dalton.Crutchfield - Again, yes @slivingston you are correct. The format of that REST command should be:

http://{robot-ip-address}/api/audio

So, for example:

http://10.1.1.43/api/audio

No trailing slash and the IP address should be in front, no curly braces. :slight_smile:


#17

@vince I encourage you to try this without base 64 encoding. Do you want to take a look at this StackOverflow thread about getting a byte array in Python? It seems to suggest using the bytearray command.

If your output still doesn’t work when you don’t use the base64 encoding step, would it be possible for you to copy in the actual data to this thread? You could put the Python-generated data in one message and the SampleUI.js data into another message, if that is okay. Sorry! I’m having a hard time evaluating the data from the screenshots.


#18

@vince @Dalton.Crutchfield
also, if more debugging is needed, it would be useful to share the response from the HTTP server, e.g., add the following at the end of your functions:

if not response.ok:
    print(response.status_code, response.reason)

#19

@Donna Thank you for the solution! And thank you for your suggestion @slivingston !

I used the bytearray command in several days ago, and it didn’t work in that time. But I tried it again just now, it worked! That is so weird. I will do more tests to make sure this will be stable. If we get another problem, we will post on community.

Thank you so much!! @Donna @slivingston


#20

Thanks to everyone!

Here is our final working solution for anyone who has the same issue :

def save_audio_to_misty(audio_name, ip_address):
    with open(audio_name, "rb") as audio_file:
        byte_array = bytearray(audio_file.read())

  new_string = ""
    for i in range(len(byte_array)):
        if i == (len(byte_array) - 1):
            new_string += str(byte_array[i])
        else:
            new_string += (str(byte_array[i]) + ",")

    url = "http://"+ip_address+"/api/audio"
    payload = json.dumps({"FilenameWithoutPath": audio_name, "DataAsByteArrayString": new_string,
                          "ImmediatelyApply": True, "OverwriteExisting": True})
    headers = {
        'Content-Type': "application/json",
        'Cache-Control': "no-cache"
    }
    response = requests.request("POST", url, data=payload, headers=headers)
    print(response.text)