Robot Not a Dev? Pre-order Now
Not a Dev?

GraphQL as a data query language

graphql

#1

REST APIs are made for providing internet resources at predictable locations that are optimized for search engines and browsers. Robots aren’t search engines or browsers. GraphQL is query language that allows it’s users to describe the data that is available and how to get it. It supports schemas and introspection so each robot can have it’s own set of APIs available and you can see what is available for each robot by inspecting it’s schema or exploring it using a tool. It also supports subscriptions over websockets. Implementing GraphQL could take our existing REST API and our existing PubSub API over Websocket and combine them into a single, dynamic query language. It even supports getting multiple resources in a single HTTP call. There are a lot of existing developer tools for GraphQL that allow you to inspect the schema and test out different queries before trying to integrate them into skills. HTTP is just a transport for GraphQL, so GraphQL could be used to query data both locally through and SDK and remotely through HTTP and WebSocket.

GraphQL


#2

Justin: is GraphQL implementation one of the developers in the Misty community could undertake? Like a Cameron did with Python?


#3

You could theoretically create a client-side implementation of it that would wrap the HTTP endpoints but that would essentially sidestep many of the benefits of using GraphQL.

As an example from the Companion App, on the mapping screen there are a few pieces of information that are helpful. Orientation, Position and SlamStatus. In order to get the current SlamStatus we could write a GET HTTP request to http://192.168.1.1:80/api/GetSlamStatus. Easy enough, so now we have 1/3rd of the information. In order to get Orientation and Position we need to make another GET HTTP request. Which means we either start both requests and wait for both of them to return or we can do them serially on the client-side where we make 1 request and then make another. Each time you make an HTTP request, there is inherent overhead with making that request, opening the TCP socket, handshakes, exchange of data and closing the socket to name a few. So lets say that each request has ~1000 milliseconds of overhead. On the robot, getting that information may be fairly quick, let’s say 500 milliseconds to just get the information. So in order to get all 3 pieces of information from the client I need to write and make two requests (2000 miliseconds) for 1000 milliseconds of processing (2 requests of 500 milliseconds each) for a grand total of 3000 milliseconds. This means that in order for me to get one update of Orientation, Position and SlamStatus my code needs to wait for 3 seconds before I can process the data, 2 seconds of which is just the overhead of making a connection.

In graphQL, you describe all of the data you want in a single HTTP request similar to this:

query GetSlamData {
    slamStatus {
        sensorStatus
        status
    }
    orientation {
        yaw
        pitch
        roll
    }
    position {
        x
        y
        z
    }
}

So now I have described all of the data that I want in a single HTTP request, cutting the latency of this theoretical situation in half (more if you need more data from different endpoints). So we can describe multiple Api commands in a single request but that is just the start of the benefit of graphQL.

It also enforces a data schema which seems like it would be additional overhead from a maintenance standpoint but the benefits are yuuuuuge. For instance, a user can use a tool like graphiQL which reads the data schema and provides auto-completion as you’re typing for the data that is available. The best part is that the schema can by dynamic. Say that we wanted to allow someone through the SDK to extend our data schema so that the data from their misty backpack becomes available through our API. Now if they extended our schema, they’re data would show up in the same auto-complete tool that they can use to quickly test out requests and see what data is available (honestly similar in ease of use to blockly). This would be analogous to letting users write their own API commands as part of the SDK.

But there is more. Now lets say we want the robot to update our client in real-time of those changes. All we need to do from the client side is to take that same graphQL query above, open a websocket and send the request to the robot over that websocket. Then the robot can push updates to us when the data changes and the client no longer has to make a request every time it wants data. graphQL also has syntax for filtering which is more extensible than our current PubSub api. For the PubSub api you need to tell the robot in a separate request for each of the data properties you want updates for, with graphQL you can do all of that in a single request and you can do robot-side filtering of events using the same graphQL syntax.

I intend on making this a hackathon project for our next company hackathon. We used graphQL at my previous company and all of the developers who used the API absolutely loved it.


#4

This is a great description and useful examples, thanks for taking the time to put this together @justin! I’m going to dig in a bit more.


#5

Sounds pretty useful Justin; also sounds like a great conversation for Jason too.


#6

This also sounds very useful to me. To work on development of the ‘brain’
on a external machine. (more compute power and preferred OS type linux).
GraphQL or a API that could do this. But the websocket opened and notifying
when a change in data requested to send updates is important. For external
brain, for monitoring/data collection, surveillance, etc.

Specifically NLR/NLU talking to the robot but having my computer handle
the journal for the Speech-to-text and even a voice control option to change
the config to allow going to Google, Alexa, Wat.ai, etc.

So the brain could be developed/tested off Misty.


#7

in the current API or in the use of GraphQL proposed here, is there a notion of lossy communication being acceptable?

I ask because as far as I know, the answer is “no”. however, for robotics it is important to be able to express that I want to stream some sensor data as quickly as possible and that dropping some packets is OK.


#8

related to my previous question in terms of motivation, did you already document why it is necessary to use WebSockets for streaming and an HTTP API?

I ask because there are several well developed and battle-tested messaging systems for robots, so the decision to not use any of them is surprising. (I might have missed something.)


#9

I cannot answer for @justin

I personally do not care of the ‘method’ just the efficiency.

  • I would like to be able to get various sets of sensor data (audio, video,
    at times at least) and gather other data not by 10+ queries.
  • Sometimes you want to do a pull (query the robot)
  • Other data you want to push the data from the robot.
  • And be configurable on:
    • what data triggers a push/update
    • Trigger to monitor the stream (guard robot pointing out movement between certain hours or in certain areas).
    • what collection of data is pushed
      (since you do not want to push the audio/video just listen to the stream or setup a stream).

#11

What messaging systems are you referring to here? MQTT? AMQP? Aren’t those protocols on top of WebSockets? We used websockets and http because there is wide support in every programming language for either of those protocols. GraphQL is just a query description language that can be served over Http or really any kind of transport.

With GraphQL you can query multiple elements of data that may not be inherently related (ex: time of flights and pose).

Any GraphQL query can also be subscribed to. This means you can define the filter parameters that you want to control your event subscription (ex: interval, identifiers, fuzzy search, etc.). Granted we will have to implement the filters on the robot for you to use them but once in place it is pretty trivial. GraphQL isn’t meant to stream data like audio/video for that we’ll need to use a more specific protocol.

I’ll be honest, when I suggested this I was being very selfish. I work on the Companion App and often need various tidbits of information that I can’t get with a single HTTP request. Often times I want real-time updates of these things. For example, when displaying the map we want to show Misty’s location, orientation, the status of the SLAM sensor. If I could explain in a single query that I want this information and I want updates for it at an interval of 250 ms that significantly reduces the amount of complexity that I have to handle in the app. Also it doesn’t hurt that most mobile applications frameworks have pretty good support for GraphQL clients.

The default transport is HTTP because this technology was pioneered at Facebook for Mobile apps but in reality we could serve it over any transport or protocol. It has the additional benefit of being a self-describing API so anyone could query a given Misty and see what data points are available. Versioning is built-in so the API can evolve without breaking older clients. You only get the data that you query for which means the robot doesn’t have to serialize extra data and send it if the client doesn’t care about it which makes both the robot and the client more efficient.

Obviously I’m very biased as a mobile developer but I see this as being extremely powerful for robots. Thinking towards the future when Misty has more of a personality and is more aware of her surroundings, maybe she decides that she wants to add data to her own API. To me, that’s pretty cool!


#12

I’ve used it a few of times on projects, I’m down with testing to get it integrated with Misty for sure if things went that way. There is some learning but yes having a way to extract the data in the specific schema you need and only having one call to get it is far superior to a typical rest API that would need to have a ton of endpoints with documentation for schemas for each request, PLUS technically its more less just up to a consumer / dev to use the GraphQL client in whatever language and the Misty Team would handle the side on the robot that makes sure data is available. It’s a different paradigm than most, but worth exploring. I will try and think of some examples where GQL makes sense currently with making a skill.


#13

Notice that I wrote “for robots”. For example, LCM (https://lcm-proj.github.io/).

In fact, they are not. It is possible to encapsulate MQTT over WebSockets, but you probably should not do that.

Are third-party developers on Misty robots going to get access to raw sensor data and be able to send actuation commands? If so, then you should seriously consider prior work on messaging for robots.

One idea: for abstract queries like “status of the SLAM sensor” or “get current map”, use HTTP or whatever you prefer as a mobile developer. for raw access, use LCM or ROS communications.


#14

I have an existing app that exposes graphql endpoints with full subscription support. It was not “too difficult” but there are some things to consider.

  1. Shape of data
  • Must be known ahead of runtime
  • Must be typed to one of the known types that GraphQL supports
  • Must have a schema that describes the shape of data including types

If your data follows the conventions you will not see too many stumbling blocks. The places that I had issues were when the data type could be unknown ahead of run time. For example if you have a query that can return a blob of json and the contents are variable.