Home Community Blog Buy Now
Blog

Misty Community Forum

2021.02.18 Release Notes

Greetings, Misty developers!

This system update adds a lot of functionality and improvements to Misty’s audio system(s). These were all built from our learnings with the new Temperature Screening Assistant application and have been brought into the core platform to dramatically improve Misty’s ability to work with audio in loud and/or difficult environments.

Thanks, as always, for being an early developer for the Misty II platform. We can’t wait to see what you build!

Installing the Update

Misty receives software upgrades as over-the-air (OTA) system updates. This update will be available within the next 24-48 hours (the precise timing varies by region).

Misty automatically checks for new updates each time she boots up. As long as her battery has enough charge, she automatically installs any updates that are available. If your robot doesn’t start to install this update the next time she boots up:

  • Check that Misty is connected to power or sitting on her wireless charging station.
  • Try connecting Misty to the Command Center to make sure she’s still on your Wi-Fi network.

If Misty is charging and connected to the internet, you can check whether an update is available in your region by connecting Misty to the Command Center and looking at the System Updates section.

Note: Misty reboots once during a system update. During an update, Misty ignores all commands except Halt and Stop. If Misty starts installing an update while charging, do not remove the power source until the update is finished and Misty’s eyes are fully open.

If you have issues with a system update or need technical assistance for other reasons, for the quickest response you can:

  • Post a message to the Support category here in the Community forums.
  • Contact the Misty support team through the chat embedded in this site, or by emailing support@mistyrobotics.com.

Release Contents

  • Misty II - Updates
    • Window IoT Core OS version: 10.0.17134 or higher - No updates
    • Android OS: (8.1) - No updates
    • Robot Version: 1.23.2.10882
    • MC Version: 1.23.2.130
    • RT Version: 1.23.2.130
    • Sensory Services App Version: 1.23.2.130
  • Web-based Tools - No updates.
  • Misty JavaScript Extension for VSC - No updates.
  • Misty App - No updates.
  • Documentation - No updates.

New Features

  • Added the ability to use Azure & Google speech services directly in a single command

    • What does this improve from previous methods? It provides an easier and lower latency process as you will no longer need to start speech capture, download the file, upload it to azure, wait to get a response back, and then parse the result. The single command(s) will send the files directly to either Azure or Google speech services and the event that comes back will have the transcribed text.
    • 4 new endpoints have been added which take all of the parameters for CaptureSpeech or StartKeyPhraseRecognition
      • CaptureSpeechAzure
      • StartKeyPhraseRecognitionAzure
      • Additional parameters:
        • SpeechRecognitionLanguage
        • AzureSpeechKey
        • AzureSpeechRegion
      • CaptureSpeechGoogle
      • StartKeyPhraseRecognitionGoogle
      • Additional parameters:
        • SpeechRecognitionLanguage
        • GoogleSpeechKey
    • These will show in API explorer.
  • Added ability to use Azure Custom Keyword in Cognitive Services Speech Studio.

    • Adds a new azure based keyphrase engine for wake word usage. This requires a custom built model from azure speech studio to be placed at “/sdcard/audio/keyphrase.table” and also requires the keyphrase_engine in the audio_config.json to be set to MS. This can be done with the new audio/config endpoint (more below) with the following parameters:
    • “keyphrase_engine” : “MS”,
      “keyphrase_engine_configuration”: {
      “keyphrase”: “/sdcard/audio/your_keyphrase_filename.table”
      }
  • This will enable you to name your robot anything you want (except Alexa, Siri, Cortona, Google, etc that are documented limitations in the Azure Speech Studio) and trigger speech utilizing the new wake word.

  • This is something you can change on the fly, Misty will reload and use the new wake word when you swap it out.

  • Added ability to toggle on/off Qualcomm® Fluence™ PRO

    • What is Fluence™ PRO? This is Qualcomm’s voice input technology that we utilise as part of the Qualcomm processors within Misty.
    • What does it impact if I turn it on/off? If you turn this off you lose the Echo Cancellation, Noise Suppression, and Audio Localization features that are currently available as these are provided by the Fluence™ PRO software.
    • Why would I want to do this? If you do not need the above features and audio processing is of utmost importance, turning off Fluence™ PRO will provide a better amplitude on the audio files recorded by Misty. This will, in some cases, provide better audio quality on recording for use within NLP and other audio ingestion applications.
    • How to turn it on and off? You simply hit the following REST endpoints:
      • /api/system/fluencepro/enable
      • /api/system/fluencepro/disable
    • These will not show up in the API explorer
    • Note: any time you toggle on/off Fluence™ PRO this cause the 820 processor to reboot and you will receive the 410<–>820 communication error on Misty’s visor until the 820 is fully back online
  • Added new commands to edit the audio config, mixer file, and microphone so that you have the ability to customize audio settings depending on the environment in which the robot is deployed.

    • Microphone - The current settings for the microphones in Misty’s head are set to a lower setting than the maximum possible which is impacting the amplitude of recordings. This enables you to set the volume to maxim by doing the following:
      • system/mixer/volume takes Volume parameter
      • Example call to set the volume to 90 POST /api/system/mixer/volume
      • {

“Volume”: 90

}

  • Volume takes an integer between 1-100 inclusive. Default volume = 84, we recommend setting to 90+

  • Mixer file - Allows you to change the mixer file that the system is using to a new file using the system/mixer/file command it will then take the Path parameter for the new file. This mixer file path refers to a file on the 820 processors. This is an advanced command, please make sure you know what you are doing with mixer files before changing!

    • The file needs to already exist on the 820 prior to running this command to switch the mixer file
    • POST /api/system/mixer/file
    • {

“Path”: “/sdcard/audio/mixer_paths_tasha_updated.xml”

}

  • New Audio config command audio/config takes a ConfigJson parameter - audio_config.json settings to set in JSON form

  • These will not show up in the API explorer

  • Note: any time you send commands to change mixer file or microphone volume it will cause the 820 processor to reboot and you will receive the 410<–>820 communication error on Misty’s visor until the 820 is fully back online

  • Raw Audio

    • New method for StartRecordingAudioRaw available in both JS and C#. This takes the same parameters as StartRecordingAudio it will just provide all 3 tracks of audio instead of the single track.
    • This will show in API explorer.

Bug Fixes & Improvements

Improved Logging

  • We’ve shifted the internal logging system. This will change the logging names if you have created automated systems for retrieving and parsing the logs. The file names will now be a daily rolling log instead of an hourly log.
  • Examples of previous and new:
    • Previous log name: QnASkill-20210215-09.log
    • New log name: QnASkill-20210215.log
  • The good news is that the logs will be more consistent with this change!

Known Issues

The following is a list of new known issues with this release. For a comprehensive list of the issues we’re tracking, see the Known Issues section of the Community forums.

  • No known issues.
1 Like

Hi Jeremy. Hope all is well. Have been working on this Custom Keyword and have created a new .table file with a new keyword.

  1. Having problems getting the .table file placed to the /sdcard/audio/ location.
    Can’t hit the WiFi IP address directly so assuming the SLAM patch was installed and have to go get a USB to Ethernet cable to connect, then find and load the new .table file.
  2. Problem understanding where and how to install the audio_config.json.
    The new endpoint is not in API explorer and I don’t see where the audio_config.json file should reside. Also not sure if we create a new json file or edit an existing one and how (REST POST?).
    Thanks for pointing me in the right direction.

Hi @Cgkocks,

Hope you are doing well. Thought I might jump in and share some info.

Issue 1:
Yes you would need to go through USB Ethernet as Patch 1 locked the doors on ADB. Here are the instructions, just in case you need them.

  1. Find the IP address of the USB to ethernet adapter by plugging it into your computer and looking at the network tab in your computer.
  2. Open a terminal or powershell
  3. In the terminal window type: ssh -L 127.0.0.1:5555:10.10.10.100:5555 administrator@ Note: if you get an error about ssh using obsolete cypher or something you have an old version of Windows IOT and you can get around the issue by doing: ssh -c aes256-cbc -L 127.0.0.1:5555:10.10.10.100:5555 administrator@
  4. log in using your Windows administrator pw found on the bottom of the robot
  5. you will be logged in to 410 via ssh
  6. now open another terminal or powershell (see step 2)
  7. Type adb connect 127.0.0.1:5555 into the new terminal window
  8. now you can adb as normal (as long as the other terminal doesn’t disconnect)

Command to push the .table file: adb push your_keyphrase_filename.table /sdcard/audio

Issue 2:
You can pull the audio_config.json from your robot to your local machine, edit it and then upload it to overwrite. But the easiest would be to just create a file named audio_config.json with the below contents and push it to Misty.
audio_config.json

{
    "misty_settings": {
        "vad_engine": "WebRTC",
        "vad_engine_configuration": {},
        "keyphrase_engine" : "MS",
        "keyphrase_engine_configuration": {
            "keyphrase": "/sdcard/audio/your_keyphrase_filename.table"
        }
    }

Command to push the audio_config.json: adb push audio_config.json /sdcard/audio

CP [• •]