Robot Not a Dev? Buy Now
Not a Dev?

Misty Community Forum

Musical skills facilitated by Spleeter

while there is already support built in Misty robots for human voice localization, it might be interesting to supplement this with Spleeter to build skills that involve music. for example, one idea related to the “dance-off” mission is a skill where Misty robots listen to a song, extract the “drums”, and then move with the rhythm.

quoting from the PyPI entry:

Spleeter is the Deezer source separation library with pretrained models written in Python and uses Tensorflow. It makes it easy to train source separation model (assuming you have a dataset of isolated sources), and provides already trained state of the art model for performing various flavour of separation :

  • Vocals (singing voice) / accompaniment separation (2 stems)
  • Vocals / drums / bass / other separation (4 stems)
  • Vocals / drums / bass / piano / other separation (5 stems)

I like the idea of associating movement with audio input.

For a while I was working on a skill that turned Misty into a musical instrument. Audio files of sine waves were associated with certain arm actuator values, so that Misty would “sing” different tones based on her arm position. Turns out sine waves are pretty annoying to listen to at 100% of Misty’s volume. In any case it would be cool to see this notion flipped around in a skill where Misty discerns pitch, rhythm, BPM, instrument type, etc. and moves in accordance.