Community Blog Buy Now
Blog

Misty Community Forum

Connecting to Misty's audio/video stream with VLC

Misty supports two different modes for AV streaming:

  • Misty can transmit a live audio and video data stream to an external media server that you configure to run on the same network as the robot. Misty supports streaming over Real-Time Messaging Protocol (RTMP) or Real Time Streaming Protocol (RTSP). You must create and host the media server yourself and configure the server to publish a stream you can view with a streaming client (like VLC).
  • Misty can serve an RTSP stream herself, and you can view the stream with a client connected to the same network as the robot.

This post describes how to use Misty as her own media server and view the stream with VLC, a free media player that can serve as a video streaming client. (The steps to connect to a stream from an external media server are similar, but instead of using the rtspd:<port-number> URL, you will connect using the URL for the stream your media server provides.)

While the focus of this guide is on issuing remote commands to enable AV streaming, the same API commands we use here are also available in Misty’s JavaScript and .NET SDKs for use in your robot skills.

Note: Before you begin, be advised that Misty’s AV stream is not encrypted at this time. Devices on the same network as your robot (or in between your robot and the streaming server or client) can intercept the stream, play back the content, and re-publish the stream outside of your local network. Additionally, you are responsible for securing and encrypting any media content you choose to stream from Misty to services or devices outside of your local network.

You’ll want to start by downloading and installing the VLC media player. Then, follow the steps below to start Misty streaming and to view the stream with VLC.

1. Starting Misty’s AV Stream

  1. Boot up your robot. Make sure Misty is connected to the same network as the computer on which you installed VLC.
  2. Enable Misty’s AV streaming service by using one of the options below:
    • Option 1: Use the API Explorer to send a command to the EnableAVStreamingService endpoint (grouped under System).
    • Option 2: Send a POST request to <robot-ip-address>/api/services/avstreaming/enable using curl or a REST client like Postman.
  3. Start Misty’s AV stream by using one of the options below:
    • Option 1: Use the API Explorer to send a command to the StartAvStreamingService endpoint (grouped under Perception).
      1. Enter rtspd:1935 in the Url field (you can stream through a different port by changing the number after the colon).
      2. Enter 640 for the Width and 480 for Height. (You can use any of the supported resolutions listed in the documentation for this command; we are using a lower resolution here to decrease the likelihood of latency.)
      3. Make sure the rest of the fields are blank (you may need to delete the 0 characters that populate when the page loads).
      4. Click Send Request.
    • Option 2: Send a POST request to <robot-ip-address>/api/avstreaming/start using curl or a REST client like Postman. Use the following JSON payload (note that you can stream through a different port by changing 1935 to a different port number):
{
	"URL": "rtspd:1935",
	"Width": 640,
	"Height": 480
}

If everything worked, the blue tally light beneath Misty’s flashlight turns on and the robot returns a response with the following data:

{
 "result": true,
 "status": "Success"
}

You can now view Misty’s stream with VLC.

2. Viewing the stream with VLC

With Misty streaming, launch VLC. Then, follow the steps below that are appropriate for your operating system:

On a Mac:

  1. From the toolbar, select File → Open Network…
  2. In the URL field, enter rtsp://<robot-ip-address>:1935 (or, if you used a different port number, use that instead of 1935).
  3. Click Open. Misty’s video stream should appear in your VLC media player.

The orientation of Misty’s RGB camera in her visor causes the video stream to appear rotated 90 degrees counterclockwise. To rotate the video to the correct orientation:

  1. From the toolbar, select Window → Video Effects…
  2. Select the Geometry tab.
  3. Check the box next to Transform, and select Rotate by 90 degrees from the menu.

To decrease latency, try the following:

  1. From the toolbar, select VLC → Preferences.
  2. Select the Input / Codecs tab.
  3. Select Lowest Latency from the Default Caching Level menu.

On Windows:

  1. From the toolbar, select Media → Open Network Stream
  2. Enter the URL for your stream: rtsp://<robot-ip-address>:1935 (or, if you used a different port number, use that instead of 1935).
  3. Click Play. Misty’s video stream should appear in your VLC media player.

The orientation of Misty’s RGB camera in her visor causes the video stream to appear rotated 90 degrees counterclockwise. To rotate the video to the correct orientation:

  1. From the toolbar, select Tools → Effects and Filters
  2. Select the Video Effects tab.
  3. Select the Geometry tab.
  4. Check the box next to Transform, and select Rotate by 90 degrees from the menu. Click Save.

To decrease latency, try the following:

  1. From the toolbar, select Tools → Preferences.
  2. Select the Input / Codecs tab.
  3. Select Lowest Latency from the Default caching policy menu. Click Save
2 Likes

Awesome! I can confirm that the “On Windows” instructions also work on GNU/Linux.

Have you compared the latency between streaming from the onboard RTSP server vs. an external server, assuming that all hosts (including the external server) are on the same local area network?

Overall, RTSP from on robot to client on same network and RTSP to media server on same network to client on same network are roughly comparable ~1sec or slightly less of latency (with all buffers set to 0). RTMP to server to client is about 4-5sec of latency. Have not done any testing with server or client being on remote network than the robot.

5 posts were split to a new topic: Trouble AV streaming to VLC

Hi All,

it’s possible to change the source camera of the streaming to the wide-angle camera?

No; Misty’s streaming service can only use the RGB camera at this time.

Am I correct in assuming you would like to stream with the wide-angle camera in order to use other functions of the RGB camera while actively streaming?

Hi Johnathan,

thanks for your answer.

The idea is expand the field of view to the maximum possible.
Actually I’m starting to make things with the robot, and to teleoperate is more confortable to use a camera like as fish-eye.

If you consider reconfigure the cameras or add the posibility to change the source in C#, will be very apreciated.
Best regards!

The Wide FoV camera is part of the Occipital Structure Core sensor. As such it is not a normal camera that goes through the same hardware and software stack as a normal Android camera. So it is a bit harder that just switching to a different input.

That being said, it is possible to do, it just requires more work. It is currently not on our priority list but you should add this to the Wishlist as we do use that to frequently reassess community needs and priorities.

1 Like

Did you try to turn the robot head during teleoperation? Misty robot heads can move quickly and cover a large part of S^2 (a sphere). (e.g., try POST /api/head.)

Thanks @slivingston and @Vlad for our answers.

About the camera, when using the intel realsense, you can change between cameras without so much modifications. Is obvious that if the eye-fish camera is part of the core to obtain 3D depht is more tricky extract the data.

So about move the head, yes is an alternative, but if in some case are you moving the head to realize other operations, yo lost the track view totally. With the eye–fish you can control a little bit more area.