Robot Not a Dev? Pre-order Now
Not a Dev?

Misty Community Forum

Long Ping Times for Misty

I was getting no response on my MacOS, so I moved to my linux notebook also on
the same wireless and the speeds were much better. Okay that is odd, I must have
just got lucky for 3 times ping’ing. Now it is slow on my linux box also 165-700ms.

bash-4.4$ ping -c 10 192.168.1.151
PING 192.168.1.151 (192.168.1.151) 56(84) bytes of data.
64 bytes from 192.168.1.151: icmp_seq=2 ttl=64 time=730 ms
64 bytes from 192.168.1.151: icmp_seq=3 ttl=64 time=555 ms
64 bytes from 192.168.1.151: icmp_seq=5 ttl=64 time=481 ms
64 bytes from 192.168.1.151: icmp_seq=6 ttl=64 time=295 ms
64 bytes from 192.168.1.151: icmp_seq=7 ttl=64 time=334 ms
64 bytes from 192.168.1.151: icmp_seq=8 ttl=64 time=273 ms
64 bytes from 192.168.1.151: icmp_seq=9 ttl=64 time=682 ms
64 bytes from 192.168.1.151: icmp_seq=10 ttl=64 time=598 ms

--- 192.168.1.151 ping statistics ---
10 packets transmitted, 8 received, 20% packet loss, time 34ms
rtt min/avg/max/mdev = 272.955/493.608/730.149/165.726 ms

More curiosity made me wonder how it would behave from Android to my home linux box.
And the ping times were more more normal/regular/consistent, and again pinging misty was
more variable and still upto 300ms currently.

This is from Android (via adb) on Misty II to my linux box:

msm8996:/ # ping -c 5 192.168.1.182                                            
PING 192.168.1.182 (192.168.1.182) 56(84) bytes of data.
64 bytes from 192.168.1.182: icmp_seq=1 ttl=64 time=9.66 ms
64 bytes from 192.168.1.182: icmp_seq=2 ttl=64 time=15.9 ms
64 bytes from 192.168.1.182: icmp_seq=3 ttl=64 time=11.5 ms
64 bytes from 192.168.1.182: icmp_seq=4 ttl=64 time=11.0 ms
64 bytes from 192.168.1.182: icmp_seq=5 ttl=64 time=10.3 ms

--- 192.168.1.182 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 9.664/11.721/15.912/2.196 ms

How does the misty ‘networking’ work? Is pinging the Misty II address
go through Windows or Android and is it a passthrough?

On Android the main CPU usage is the ‘mm-qcamera-daemon’ about 80-95% of the CPU.

Tasks: 507 total,   2 running, 502 sleeping,   0 stopped,   0 zombie            
Mem:   2839788k total,  2269988k used,   569800k free,    15456k buffers        
Swap:   524284k total,        0k used,   524284k free,   952440k cached         
Tasks: 507 total,   1 running, 503 sleeping,   0 stopped,   0 zombie            
Tasks: 509 total,   2 running, 504 sleeping,   0 stopped,   0 zombie            
Mem:   2839788k total,  2273120k used,   566668k free,    15536k buffers        
Swap:   524284k total,        0k used,   524284k free,   955092k cached         
400%cpu  97%user   0%nice 106%sys 181%idle   0%iow   6%irq   9%sirq   0%host    
  PID USER         PR  NI VIRT  RES  SHR S[%CPU] %MEM     TIME+ ARGS            
  992 camera       20   0 198M  35M 7.9M S  115   1.2  15:31.87 mm-qcamera-daem+
 6118 root         20   0  11M 2.5M 1.6M R 53.1   0.0   0:00.10 top -n 1
  835 system       20   0  12M 2.7M 1.6M S  6.2   0.0   0:59.67 cnss_diag -q -f
  570 cameraserver 20   0 103M  22M 6.7M S  6.2   0.7   0:44.91 android.hardwar+
 3114 system       20   0 4.2G  65M  34M S  3.1   2.3   0:27.15 com.mistyroboti+

msm8996:/ # uptime
 00:28:27 up 17 min,  0 users,  load average: 4.45, 5.01, 3.60

msm8996:/ #  free -h
		total        used        free      shared     buffers
Mem:             2.7G        2.1G        548M        4.3M         15M
-/+ buffers/cache:           2.1G        564M
Swap:            512M           0        512M

And memory was not used that heavily

Thanks for doing this @markwdalton! Morgan will be unavailable this weekend, so we’ll pick this up on Monday. Thank you so much for taking the time to help give us all the extra information!

It is no problem, and it is not a rush. It is good if you ‘have a life’ :slight_smile:

This morning after on the charger (in the on state) all night. Misty II is still on, eyes are full, and the purple-ish light in the body. There is no response to ping 100% packet loss (so the web apps cannot connect either).

mbp-mwd:mistyii mwd$ ping -c 5 192.168.1.151
PING 192.168.1.151 (192.168.1.151): 56 data bytes
ping: sendto: No route to host
ping: sendto: Host is down
Request timeout for icmp_seq 0
ping: sendto: Host is down
Request timeout for icmp_seq 1
ping: sendto: Host is down
Request timeout for icmp_seq 2
ping: sendto: Host is down
Request timeout for icmp_seq 3

--- 192.168.1.151 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

From the Misty Companion App, it connected via Bluetooth. And it sees the IP address
(the battery reading in the app does not work). Attached is a screen shot in the app of ‘My Misty’ (network shows, but OS, etc does not)

And for more completeness. From the Misty Companion App it could engage the motors
and travel a few inches and it would stop (the front end off the charger) And I could backup. Then I lifted it off the charger, and tried moving forward again and it went a few inches, then backed up, then forward now it continued moving forward.

And interesting enough pings started working again now.

mbp-mwd:mistyii mwd$ ping -c 5 192.168.1.151
PING 192.168.1.151 (192.168.1.151): 56 data bytes
64 bytes from 192.168.1.151: icmp_seq=0 ttl=64 time=407.741 ms
64 bytes from 192.168.1.151: icmp_seq=1 ttl=64 time=323.454 ms
64 bytes from 192.168.1.151: icmp_seq=2 ttl=64 time=140.988 ms
64 bytes from 192.168.1.151: icmp_seq=3 ttl=64 time=464.944 ms
64 bytes from 192.168.1.151: icmp_seq=4 ttl=64 time=224.332 ms

--- 192.168.1.151 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 140.988/312.292/464.944/118.000 ms

So after it was off the charger, and after driving, the MyMisty in app did get the information and pings recovered.

But ping times are still long… (this may be related to the Misty Charger thread)

Just to be clear I am not sure if its recovery was because it was then off the charger, or because driving the motors. So next time, I will just pick it up off the charger (if it is still on at the time).

This power and no network is a separate issue.

Reading on MS Windoze 10 it has been a regular issue. From reading MS Windows support blame it on other things repeatedly. Even when in one ticket the user confirmed on Windoze 7 it had no packet loss or slowness on the same computer. Also his other machines: iPhone. Tablets and another windoze 7 machine all worked correctly.

Ping times from MS Windows to my local machine were better, but still not optimal.

PS C:\Windows\system32> ping -n 5 192.168.1.182                                                                                                                                                                                                                  `                                                                                                                                                            
Pinging 192.168.1.182 with 32 bytes of data:                                                                                                                                                                                                                     
Reply from 192.168.1.182: bytes=32 time=243ms TTL=63                                                                                                                                                                                                             
Reply from 192.168.1.182: bytes=32 time=8ms TTL=63                                                                                                                                                                                                               
Reply from 192.168.1.182: bytes=32 time=67ms TTL=63                                                                                                                                                                                                              
Reply from 192.168.1.182: bytes=32 time=48ms TTL=63                                                                                                                                                                                                              
Reply from 192.168.1.182: bytes=32 time=74ms TTL=63                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                 
Ping statistics for 192.168.1.182:                                                                                                                                                                                                                               
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),                                                                                                                                                                                                         
Approximate round trip times in milli-seconds:                                                                                                                                                                                                                   
    Minimum = 8ms, Maximum = 243ms, Average = 88ms 

NOTE: This is a local WiFi network so ping times should be under 10-20ms.
And ping times to Misty II during this time were 152 to 310ms.

On Misty II MS Windows this is from within “PowerShell” (limited version
of a shell) I hear bash is available on Windows.

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName                                                                                                                                                                                            
-------  ------    -----      ----- -----   ------     -- -----------                                                                                                                                                                                            
      0      32    46692      97108   1,533.66   2800   0 Misty.HomeRobot.App                                                                                                                                                                                    
      0      11    11044      23820     673.89    916   0 dwm                                                                                                                                                                                                    
      0       0       60         60     141.83      4   0 System                                                                                                                                                                                                 
      0      24    20988      42216      20.89   2332   0 powershell                                                                                                                                                                                             
      0      22    12460      40332      16.23   1504   0 svchost                                                                                                                                                                                                
      0       9     5092      23104       6.70   2956   0 RuntimeBroker                                                                                                                                                                                          
      0       3      904       3556       5.09   5396   0 conhost                                                                                                                                                                                                
      0      13     4652      16572       4.34   1364   0 svchost                                                                                                                                                                                                
      0      17     4532      20604       3.77   1288   0 svchost                                                                                                                                                                                                
      0       9     2492       7512       3.66   1032   0 svchost                                                                                                                                                                                                      0       5     2360       5900       3.56    760   0 services                                                                                                                                                                                                     0       9     3676       9816       3.50   1820   0 svchost                                                                                                                                                                                                      0       9     3332      11712       3.38    780   0 lsass                                                                                                                                                                                                  
      0      23     4792      12524       3.25   2284   0 svchost                                                                                                                                                                                                
      0       5     1740       5448       2.84   5376   0 sshd                                                                                                                                                                                                   
      0       4     4592      11640       2.58   1632   0 svchost                                                                                                                                                                                                
      0      11     3240       8592       2.56   1300   0 svchost                                                                                                                                                                                                
      0      10     6372      18980       2.52   2548   0 svchost                                                                                                                                                                                                
      0      10     3224      14580       1.98    924   0 svchost                                                                                                                                                                                                
      0      24     5668      15304       1.97   2484   0 SearchIndexer                                                                                                                                                                                          
      0      11     8204      16516       1.83   1604   0 svchost                                                                                                                                                                                                
      0       3      916       2988       1.83   5420   0 cmd                                                                                                                                                                                                    
      0      11     7688         12       1.61   3620   0 WindowsInternal.ComposableShell.Experiences.TextInput.InputApp                                                                                                                                         
      0       7     2044       8240       1.45   1384   0 svchost                                                                                                                                                                                                
      0       9     3208      11556       1.44   2132   0 svchost                                                                                                                                                                                                
      0       6     2388       9208       1.30   1100   0 svchost                                                                                                                                                                                                
      0       8     3088      17072       1.27    572   0 sihost                                                                                                                                                                                                 
      0       7     2664      10924       1.23   2320   0 svchost                                                                                                                                                                                                
      0       8     3532      12752       1.08   2592   0 svchost                                                                                                                                                                                                
      0       8     3212      17008       0.94    576   0 svchost  

Also it was not pingable again (while sitting on the charge) it was still on.
So I simply picked it up and set it on the ground and now it is at least pingable.

mbp-mwd:mistyii mwd$ ping -c 5 192.168.1.151                                                                                                                                                                                                             
PING 192.168.1.151 (192.168.1.151): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
Request timeout for icmp_seq 0
ping: sendto: Host is down
Request timeout for icmp_seq 1
ping: sendto: Host is down
Request timeout for icmp_seq 2

--- 192.168.1.151 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

Then I picked it up moved it off the charger and pings mostly worked.
(NOTE: 20% of the packets were lost, and still upto 452ms when it should
              be under 10ms or max of 20ms on my local network).
mbp-mwd:mistyii mwd$  ping -c 5 192.168.1.151
PING 192.168.1.151 (192.168.1.151): 56 data bytes
64 bytes from 192.168.1.151: icmp_seq=0 ttl=64 time=239.492 ms
64 bytes from 192.168.1.151: icmp_seq=1 ttl=64 time=452.180 ms
64 bytes from 192.168.1.151: icmp_seq=2 ttl=64 time=8.055 ms
64 bytes from 192.168.1.151: icmp_seq=3 ttl=64 time=349.232 ms

--- 192.168.1.151 ping statistics ---
5 packets transmitted, 4 packets received, 20.0% packet loss
round-trip min/avg/max/stddev = 8.055/262.240/452.180/164.903 ms

Thanks @markwdalton! We’re going to dig in this AM :slight_smile:

Feel free to call me or if you want live access, you can do that via ‘tmate’ for
terminal windows or vncserver or similar if you want desktop.

I am in no hurry, I can still work on it. I just am ‘verbose’ gathering data. :smile:

Mark

Mark,

I’ve done some testing today with regards to this and I thought I’d share my findings. These aren’t wholly conclusive, but I’m working towards understanding the source of the problem.

I’ll start with the design of the robot. Currently, there are two computing modules within the robot’s head, a 410 module, and an 820 module, both from Intrinsyc. The 410 module is running Windows 10 Iot Core and the 820 is running Android 8.1. Today, in a normal, factory-fresh robot, all wifi communication goes through the 820, with specific ports routed to he 410. Today, ICMP requests are served by the 820.

My experiment centered around understanding average ping times to the robot, max ping times, and the number of timeouts that occur. To that end, I created script that pinged the robot’s IP address 1000 times, with a 200 ms delay between requests. I set the timeout to 2000 ms, even though that’s ridiculous.

My first experiment was centered around testing a robot that has just booted and either charging or not charging, with the thought that the wireless charger was somehow interfering. The resulting metrics for charging and not charging under these conditions were similar:

Cold boot, not charging: Avg: 64.66, Max: 1672, Errors: 1
Cold boot, charging: Avg: 69.94, Max: 1742, Errors: 21

WOW, the ping times were high, and the errors during charging were unexpected. I’m still testing the not charging case to prove that the recorded error count is correct (I suspect it was a fluke), but that’s an exercise for tomorrow.

I tried subsequent runs after 30 minutes, 60 minutes, and 90 minutes, both charging and not charging, all of which looked similar:

30 minutes: Avg: 79.37, Max: 1647, Errors: 28
60 minutes: Avg: 84.13, Max: 1423, Errors: 25
90 minutes: Avg: 82.67, Max: 414, Errors: 20
120 minutes: Avg: 90.37, Max: 1026, Errors: 35

That all looks pretty poor but doesn’t show appreciable degradation over time. For funsies, I connected an ethernet to USB dongle (subtracts the wifi component and the communication bridge from the 820 to 410) to the backpack and pinged the Windows instance directly, just to see if it was also responding poorly:

Avg: 5.97, Max 29, Errors: 0

That looks better, but isn’t useful in getting Android to respond correctly. Next, I took the 820 development kit and ran the same ping tests:

Avg: 61.94, Max: 631, Errors: 0

HMMMM… that’s interesting. The max response time is still awful, but the average is plausible and the errors are gone. Next, I started commenting out sections of our init scripts on the 820 to see if we’re doing something with the networking on the device to cause the errors. There were four methods that looked suspect, so I commented each out in combination to see if I could get the adapter to respond in a reasonable way. After a few tries, I got to a place where I’m at ~0 errors. The ping times however are still quite bad with the average being ~100 ms.

I’ll be doing more testing this week to see if I can get better information, but it looks like I’m narrowing in on the source of the errors. I’m still a bit puzzled by the swings in ping times. I’ll be enhancing my testing methodology this week to track standard deviation, and probably playing with the timeouts to see what shakes out as nominal.

More to come.

2 Likes

That is excellent news.
I booted to MS Windows 10 Home on my Llinux notebook and saw
similar behavior until I removed the Windows Defender Firewall.

I did try using my Android phone (Galaxy 9S+)
and used its HotSpot, then a Android tablet and my notebook to test to
MistyII (All connected to the phone Hotspot)
Linux -> Android tablet - slow 200-500 ms
Linux -> Misty II - 68-800ms

To Misty II I did 100 pings: 6% packet loss, min: 11ms max: 1531ms average: 463ms

So perhaps the slow pings here are not as abnormal (wireless network… sigh…)
But the packet loss is a issue and could explain some of the slow and failed responses.
I switched back to 2.5Ghz wifi router for 100 pings:
Linux -> Android Tablet - 7% packet loss, Min: 3ms, Max: 724ms, Ave: 333ms
Linux -> Misty II - 14% packet loss, min: 5ms, max: 819ms Ave: 352ms
Linux -> Android phone - 0% packet loss, Min: 9ms, Max: 1038, Ave: 416ms
… repeat … - 0% packet loss, Min: 7ms, Max: 1730, Ave: 420ms
(Android 9: kernel 4.9.112)
Linux-> Roku - 9% packet loss, Min: 2ms, Max 121ms, Ave: 14.5ms
Linux->MacOS - 0% packet loss, Min: 2ms, Max 95ms, Ave: 9.4ms
Linux->GoogleHome - 0% packet loss, Min: 1ms, Max 84ms, Ave: 8.4ms
MacOS->Linux - 0% packet loss, Min: 3ms, Max 503ms, Ave: 124.9ms

So more issues came up with the 100 packets Android did worse than I was
seeing in the 5-10 packet bursts.

We must have been working on the same thing at the same time!
Using the hotspot again I ran about 25 tests. I didn’t screen shot the numbers, but no packet loss, nearly all of them were in the 100-300 ms. The highest was 700 ms and lowest was 6 ms. The robot still won’t drive forward. Other than that she did all the commands I ran through and still was much quicker than the home wifi. So I’ll take it.

Note: This time I didn’t turn off the home WiFi.

@markwdalton @ScottC
Can you both check the kernel running on the 820? I have a hunch it’s a specific version causing issues.

adb connect <ip>
adb shell
uname -a
getprop | grep version

My original robot with the long ping Times was sent back to CO for diagnostic testing. It should have arrived there today. I haven’t tested the ping times for the replacement yet. I’ll try today and get back to you.

I just tested the new robot with similar results. On the hotspot all the ping times were between 100-300 ms. On the home WiFi it did get down to one ping of 7 ms, but the other 8 out of 10 pings were over 1000 ms with the highest being 4049 ms.
I’m sorry, I couldn’t figure out where to put in that code to find out the kernel version.

1 Like

@ScottC
To install adb, go to SDK Platform Tools release notes  |  Android Developers
and get the “SDK Platform Tools” for your OS. Once that’s installed, open up a terminal and type the above commands.

If you’re seeing this on a new robot, then my guess about the version number might not be correct.

After you get the version information, in the same terminal, try running “ping <your computer’s IP>” to ping your PC from the 820.

At least in my case, the ping times were bad going from PC -> 820 but not 820 -> PC.

I tried to do this, I couldn’t get any of it to work.

No worries, I ruled out the possibility it’s version related.

Good to know, thanks @travis!

So for you @ScottC : I reached out in a direct message to run through any other diagnostics we might need!

A quick and positive update. After talking with Chris and realizing my WiFi was causing more problems than just with Misty. I did some research and found the AT&T supplied (and required) gateway has notorious problems with point to point contacts. I purchased a 3rd party router today and configured the AT&T gateway into just a modem. Some quick tests with Misty proved she was drivable and responsive. I did 20 ping tests, 0 packet loss, the highest pings were in the low 100s ms, the lowest were single digit. Much better than the 60% loss and thousands of ms ping times before.
Unfortunately, I’m not all the way out of the woods yet. As I type this I see my internet connection dropping in and out. At lest I’m narrowing down issues though.

2 Likes

Hey @ScottC! If you have the command center loaded, you should be fine without an internet connecting when interacting with misty (the commands can be sent without requiring an internet connection)

Let us know if we can help further!

1 Like