Home Community Blog Buy Now
Blog

Misty Community Forum

Ways to implement code to let misty continue to detect speech after running first time

Hi all , currently i faced a problem making misty to call the speech to text after it first run from the skill , may i know how am i able to code it so that misty able to register the event again for it after it has done doing Speect to text and text to speech. Thanks , a snippet of my code has been uploaded as reference : Jun 12 6:20 PM - Codeshare

One more question…
Would like to know does anyone had experience building voice assistance with misty using Rasa ? So currently i had two endpoint one is for model(Intent) and another for response from the chatbot . What is the best way to implement the code on misty to keep continuously calling the same endpoint for conversation e.g Talking to chat bot , response from it continuous? Using a while loop ?

Hi @believeitcode,

There are two event callbacks you can use to know when an audio has finished playing. From the callback you can register to your events again.
Added a snippet here that will help:

Random info:
In the JS SDK try to avoid using while loops. It does not perform well. Instead use misty.RegisterTimerEvent("<event_name>", debounce/frequency_ms, keep_alive_boolean);
Example:

function _my_while_loop() {
    // Your Code


    misty.RegisterTimerEvent("my_while_loop", 10, false);
}

CP [• •]

Hi CP , Thanks for the snippet example , i having doubts on this part of the code below

// Audio play complete
function register_audio_play_complete() {

    // Find when an audio file ends playing
    misty.RegisterEvent("audio_play_complete_message", "AudioPlayComplete", 100, true);

    // Find when Local TTS ends playing
    misty.RegisterEvent("tts_play_complete_message", "TextToSpeechComplete", 100, true);
}

I’m not sure how this function can implemented in my current code as my TTS part using google TTS .

Currently my code i added had some bug which the event register in the callback questionResponse causing multiple call to STT/TTS at the same time . Hope you can correct me on my implementation

// Register to VoiceRecord Event to follow up with what happens after the recording is complete
misty.Debug("Testing of STT Q&A");

register_voice_record_complete();
misty.Pause(1000);

// Start Key Phrase Recognition "Hey Misty" & CaptureSpeech 
// parameters: overwriteExisting, silenceTimeout, maxSpeechLength, requireKeyPhrase, captureFile, speechRecognitionLanguage, key

// Handler response of CaptureSpeechGoogle (STT)
function _voice_record_complete_message(data) {
    misty.UnregisterEvent("voice_record_complete_message");
    misty.Debug(JSON.stringify(data));
    var filename = data.AdditionalResults[0];
    let success = data.AdditionalResults[1];
    let error_code = data.AdditionalResults[2];
    let error_message = data.AdditionalResults[3];
    var stt_result = data.AdditionalResults[4];

    if (success) {
        misty.Debug("Info: Audio recording successful.");
        misty.Debug(filename);
        misty.Debug("Misty Heard: " + stt_result);
        
        if(stt_result != "\n")
        {
            let arguments = {
                "question": stt_result
              }
              misty.Debug("Sending Request To FastAPI");
              misty.SendExternalRequest("POST","https://fast-api-misty.herokuapp.com/qna/",null,null,JSON.stringify(arguments), false, false, null, "application/json", "_questionResponse");
        }
    }
    else {
        misty.Debug("Error: Recording Audio");
        misty.Debug(error_message);
        misty.Debug(error_code);
    }
 }
 //Handler callback function from FastAPI Bert Model Q&A 
 function _questionResponse(data){
    if(data.Result.ResponseObject.Data != "{}\n"){
    let _data = JSON.parse(data.Result.ResponseObject.Data);
    misty.Debug("JSON Response: "+_data.answer);
    speakTheText(_data.answer);
    }
    register_voice_record_complete(); //Register VoiceRecord event again
}



function register_voice_record_complete() {
    misty.AddReturnProperty("voice_record_complete_message", "Filename");
    misty.AddReturnProperty("voice_record_complete_message", "Success");
    misty.AddReturnProperty("voice_record_complete_message", "ErrorCode");
    misty.AddReturnProperty("voice_record_complete_message", "ErrorMessage");
    misty.AddReturnProperty("voice_record_complete_message", "SpeechRecognitionResult");
    misty.RegisterEvent("voice_record_complete_message", "VoiceRecord", 100, true);
    misty.CaptureSpeechGoogle(true, 5000, 7500, true, true, "en-us", _params.APIKEY_Google);

}

Edit : I tested my code sometimes it happen and sometime it will not >.<
Edit2: If you would like to test the code you can change endpoint to https://fast-api-misty.herokuapp.com/qna/ in the code share link(orginal code w/no register event yet) it is a Q&A api using pre-train BERT model with

passage : On 14 September 2017, Halimah Yacob became Singapore first woman president, and first Malay president since Yusof Ishak. Halimah was elected unopposed under the first reserved presidential election, which was reserved for the Malay community.

You can try asking question such as : who is the first president of singapore (response Yusok Ishak)
Thanks

As you are using Google for TTS, it would be a typical audio file that is played at the end of TTS. Hence you should use AudioPlayComplete.

I didn’t run your code but this what needs to be added to the code I see in codeshare - to enable Misty to listen again after she has finished playing the audio.

register_audio_play_complete();

// Audio Play Complete
function register_audio_play_complete() {
    // Find when an audio file ends playing
    misty.RegisterEvent("audio_play_complete_message", "AudioPlayComplete", 100, true);
}

function _audio_play_complete_message(data) {
    if (data.PropertyTestResults[0].PropertyParent.MetaData.Name == "TTS.wav") {    
        misty.CaptureSpeechGoogle(true, 5000, 7500, true, true, "en-us", _params.APIKEY_Google);
    }
}

I have added it to the codeshare.

Thanks CP , But now i faced a new problem again as follows

  1. first attempt saying hey misty it will work (misty responsed to my intent) ,
    2.after the second event trying the invoke of wake word by saying hey misty with my intent there is no response(i assume misty didn’t hear what i spoke to him) .
    3.Now i tried again invoke the wake word by saying hey misty , it does not tried to record my voice again.

Edit: I tested out the problem came when i tried to invoke wake word and i stay silence. After that misty will not able to response to any wake word i said again

Hope you can help me resolve it , i been stuck on this for quite sometimes.
Regards
believeit