Home Community Blog Buy Now
Blog

Misty Community Forum

Using the RGB-D camera to protect Misty's head from hazards

So I ran into a problem with Misty where her time of flight sensors don’t detect certain objects that are lifted above the ground but lower than her head. For example a dresser with legs or a chair. This is dangerous when shes roaming around because she can bonk her head on objects that the ToF sensors have no way of detecting. My understanding is that the RDB-D camera uses an IR sensor, and I was wondering if it would be possible to somehow isolate this depth information for use in skills.

It’s definitely possible! You can get the depth of each pixel from the Occipital sensor. You wouldn’t want to use all the pixels because it would be too slow but you can experiment and select a few that are relevant. I know @Jackson has used pixel depth data in his skills before.

Just seeing this now, but that’s good to hear. Would I get the depth information by using the “TakeDepthPicture()” function? Or if I am already SLAM streaming could I get the depth information some other way? And I’ll go through Jackson’s posts and see if I can see what he did. Thanks

Yeah, it’s definitely possible! Keep in mind that values from Misty’s Occipital sensor are still somewhat unreliable and will likely contain many NaN values, so you’ll probably have to play with only taking sections of the output for best results.

To get started, you’ll probably want to use the takeDepthPicture() command. Once you call the command, you’ll want to process the data in a callback function. Here’s a little snippet to get you started:

function _TakeDepthPicture(data)
{
    const width = data.Result.Width;
    const height = data.Result.Height; 
    
    const status = data.Status;
    const depthImage = data.Result.Image;
    
    misty.Debug('The width, height are '+ width +", "+ height);
    misty.Debug('The status is '+ status);
    misty.Debug('The image array length is ' + depthImage.length);
    
    //If there is data in the depth image
    if (depthImage.length > 0){
        
        let testDepth;
        let depthCount = 0;
        let depthSum = 0.01;

        //Loop through each value
        for (let i = 1; i < depthImage.length; i++){
            
            if (depthImage[i] !== "NaN"){

                testDepth = depthImage[i]; 
                depthCount += 1; //Add one to the count
                depthSum = depthSum + testDepth; //Add the measured value to the sum
                aveDepth = depthSum/depthCount;  /Get the average
            
            }
      
        }

        misty.Debug("With a count " + depthCount + " the average depth is " + aveDepth);
    
    }
1 Like

Thanks! This is helpful

1 Like