Robot Not a Dev? Buy Now
Not a Dev?

Misty Community Forum

Speaker access via Windows IoT Core

Is there the ability to utilize the speakers/sound system from the IoT Core side of Misty? Currently, I am working on a local voice implementation service to lower conversation latency. I have my code running/responding with a proper HTTP response, speech synth is creating a wav stream, and it is getting sent to the media player, but not playing.

Another option I thought of, is there a way to send files between the Windows IoT Core system and the Android system so that we can place the wav file directly on disk and play that via Javascript SDK?

Windows IoT Core speech service UWP app code can be found here:

1 Like

This is awesome @cbattlegear!

So right now, the only way we’ve been able to play audio from the IoT Core Media player is to connect a USB speaker of some sort to Misty. I haven’t tried this, but you could try pairing a bluetooth device to IoT Core and giving that a shot?

Placing files directly onto the Android system might be possible, but you would have to use adb to copy the files over, otherwise you could try using a REST call from your code to save the file on Misty.

I’m going to explore a few things and get back to you!

Are you using the C# SDK? You need to use the SDK calls to upload audio and play it. This will send the audio to the 820 and play it accordingly. You can also use REST calls but I imagine the SDK calls will be faster. Otherwise if you really want to play it locally on 410 then you would need to try hooking up a USB speaker as @Chris suggested, but it’s hard to know which speakers would be supported.

You can’t use adb as there is no adb for Windows ARM.

Thanks @Vlad I thought I saw an adb executable that worked on IoT Core, I must have been mistaken.

And as Vlad mentioned, we do have a C# SDK that’s in a closed Beta, let me know if you might be interested in it!

I didn’t realize there was a C# SDK! I would love to jump in and test it out if possible!

Was currently working on doing the REST API method but the less web calls the better!

We will get you the test instructions. Not sure how we haven’t done that yet!

1 Like

While waiting, is there a specific address I should be connecting to for the REST API from the IoT Core side? I am not able to hit the api currently using the external IP address.

@cbattlegear I’ve seen that most of the simple USB speakers tend to work without problem, though they’re not on any hardware compatibility list. The official list is here. Generally, this is working around the built in audio capabilities, but it would get you moving should you want to go that route. I’m a bit surprised that you can’t hit the REST API on the robot. I’ve not heard anybody say that they couldn’t hit the API from within a skill, using SendExternalRequest. Are you receiving an error?

I am trying to hit the rest api directly from a UWP app running directly on IoT core (referenced by @Vlad above as the 410). Essentially I would prefer to use the built in audio system if possible so I am trying to directly upload the file to the rest api to have it play that way instead of pushing it through USB.

I have tried the public IP (over USB to Ethernet), 10.10.10.10, and 10.10.10.100. All three of those are throwing an exception from HttpClient, inner message is “A connection with the server could not be established” so essentially the default, can’t find a route, message.

I know at this point I am kind of off the beaten path compared to normal development processes for Misty but I am trying to lower latency on text to speech.

Interesting. The internal IP addresses (the 10.10.10.10 and 10.10.10.100) are simply to support the ethernet bridge between the two core processors. I’ll test the HttpClient this morning and see if I’m getting the same routing issue.

@morgan I will push my code with the API call up to my github project (in the original post). It isn’t good code, but such is life.

@cbattlegear I created a new background task project and added two lines to the Run method, which seemed to work:

public void Run(IBackgroundTaskInstance taskInstance)
		{
			HttpClient client = new HttpClient();
			HttpResponseMessage result = client.GetAsync("http://10.0.1.163/api/device").GetAwaiter().GetResult();
		}

The IP address represents the normal IP address as described through the device info call. Do you perhaps have a network restriction that is preventing the robot from connecting to itself?

Interesting, to my knowledge we don’t have any restrictions for connectivity from that level. One item that may be important is my device info call has a different IP (my wifi connection) versus the IP I am using for development (my USB to Ethernet adapter).

Will try some different network configurations to see if it resolves the issue. It is a bummer it won’t work with the internal addresses so that you could have a static point to hit.

I can also use my USB adapter IP address without issue. Can you ping either IP from the IoT Core dashboard?

Can ping the USB IP Address from the dashboard (not the Wifi address though which is expected).

Couldn’t you just use 127.0.0.1 (i.e. localhost) as the IP address since you’re trying to connect from the device itself?

The loopback IP may work. Normally, that gets forbidden by the UWP runtime for reasons not entirely clear to me.

Traditionally the loopback will work when sideloading via Visual Studio (it automatically disables the loopback restriction for the UWP app when you deploy via debugging)

Interestingly no IP address is working for me at all for rest api calls via UWP even when I completely move to a different network that I know has no restrictions. Outbound to a different API works fine though. I will keep digging and see what I can do!

After some head scratching I found running

CheckNetIsolation.exe LoopbackExempt -is -n=MistyRobotics.Misty2HomeRobot_gand6cah5b678

in the Windows Device Portal allowed loopback (127.0.0.1) connections from other UWP apps.

My basic UWP app now can connect to the rest api!

2 Likes

That’s awesome! Thanks for sharing how you moved ahead @cbattlegear! We’re exploring a cleaner way but are you able to move ahead?