Remote Robot Debugging

Being a remote employee of my company, I have a lot of experience working...well...remotely. Still, there are some things I just don't do from a remote location. Fixing a homemade robot that is being controlled by a computer I've never even seen before is one of them. This is where persistence and quick thinking pay off.

Anyone who has been reading this blog will know that I have been working on a labyrinth game that is controlled by a Nintendo Wii Balance Board. We made a YouTube video that got a few hits when we released it, and it gained some recognition in the hacker/diy community. Fellow member of my local hackerspace, hevnsnt, had inquired about taking the device to the Shmoocon convention in Washington D.C., but ended up deciding to leave it here in Kansas City. Well, the Tuesday before the conference, hevnsnt had a change of heart:

from hevnsnt
to Jestin Stoffel, SomeoneKnows
date Tue, Feb 3, 2009 at 2:35 PM

I am starting to regret our discussion at last thursday's meeting where I said I didnt want to take the labyrinth to shmoocon (D.C.), and I would like to talk it out with you.

So, we arranged to meet that night to do the hand off. I had the code he needed, as well as a Balance Board he could borrow, and @SomeoneKnows had the actual robot. We each met with hevnsnt separately because he was on a tight schedule, and never actually got a chance to test things on his computer. I had a small chain of emails with him about getting my code to compile, but got that part working without too much trouble. He hooked up the robot, and had issues:

from hevnsnt
to Jestin Stoffel, SomeoneKnows
date Wed, Feb 4, 2009 at 9:37 PM

So I got it all hooked up tonight -- and the motors go crazy (small movements) and for some reason the laby's movements do not correlate to the wii fit. What do I need to change?

Long story short, we could not figure out this problem before hevnsnt's flight, so he left for the conference with a malfunctioning robot.

It bothered me that despite my efforts, the robot still did not "just work" as I always attempt to achieve with my projects. Still, this was not the conditions under which the project was designed. Instead of using Linux directly, hevnsnt was running the LabyWiinth on a virtual machine running on OS X. We didn't know what issues this could cause, but we first decided to check if the bluetooth adapter that connected to the Balance Board was working correctly.

from Jestin Stoffel
to hevnsnt, SomeoneKnows
date Thu, Feb 5, 2009 at 12:06 AM

You can always check to see that your standard output from the
labywiinth program produces reasonable values. If that's the case, the
Arduino is where the problem is. Try not redirecting the output of
labywiinth so you can just look at it.

After hevnsnt responded that our values looked fine, we thought there might be trouble on the Arduino microcontroller that controls the robot's servos.

from SomeoneKnows
to hevnsnt, Jestin Stoffel
date Thu, Feb 5, 2009 at 12:26 AM

Do you have the Arduino programming environment on your computer if it needs reprogrammed? The Arduino is pretty much exposed without protection. Wonder if it could have gotten zapped with static electricity or something?

We sent hevnsnt a copy of the Arduino sketch for him to reload onto the robot, but to no avail. The labyrinth was still jittering uncontrollably. This made me curious about the debugging output that I had coded into the Arduino sketch. Even though nothing is reading it, I have the Arudino constantly writing it's current status to the serial port. This helped me solve a power issue when first building the circuit for the robot. Although hevnsnt's symptoms were different, I couldn't help but think that he had the same problem I once had.

from Jestin Stoffel
to hevnsnt
date Fri, Feb 6, 2009 at 2:08 PM
In a different terminal than the one you are running it from, type

cat /dev/ttyUSB0

I knew this would tell me what was happening on the robot, but when I got a response, it was not at all the problem I was having early on. I was not supplying adequate power to my Arduino, and as a result it would reset whenever attempting to move the servos. My debugging output showed my reset message occurring more than once, and thus I knew right away what was happening. This is what I expected to see, but unfortunately, hevnsnt sent me this:

from hevnsnt
to Jestin Stoffel, SomeoneKnows
date Fri, Feb 6, 2009 at 3:19 PM

So I have found the problem.. Just dont know the solution

The command I am typing

labywiinth > /dev/ttyUSB0

Using your command below, i watched what was sent to ttyUSB0
(excerpt)
Wrote 45 to horzServo
Wrote -20201 to vertServo
Wrote 145 to horzServo
Wrote 145 to horzServo
Wrote 0 to horzServo
Wrote 6655 to vertServo
Wrote 7199 to vertServo
Wrote 1335 to vertServo
Wrote 145 to horzServo
Wrote 3335 to vertServo

With no reset message at all, I knew something else was up. The output shows that even though the output coming from the computer alternates between sending a command to horzServo and vertServo, the Arduino was sending the commands to the servos in no discernible pattern. Furthermore, most of the values being sent to the servos were *way* out of range. The only allowable values should be between 0 and 180, so this explains the erratic movement of the game board.

It was at this point when I was almost ready to give up. I did not have the robot in front of me, so trial and error would take a long time. It seemed that there were people at Shmoocon waiting to see this, so I felt that I had to keep trying, regardless of how long it took. I started writing up an email that contained the following:

It looks to me like input is being skipped. If output of labywiinth
writes 'w' than 's' and repeats that, we should always be alternating
between horzServo and vertServo as well. Since according to the
debugging output, we are NOT alternating correctly, it tells me that
sometimes a 'w' or 's' gets skipped. This would cause the arduino
program to accumulate digits to make values like '6655' instead of 66
for vertServo and 55 for horzServo. In addition, we can probably assume
that 'w' and 's' are not the only characters being skipped somehow, and
that individual digits are also being skipped. Truthfully, this blows.

I was just in the middle of writing up a few more ideas to test the code, when it hit me. If there was data being skipped, that means that the computer is sending data faster than the Arduino can read it. I know that the Arduino is reading the serial port as if data is being sent at 9600 baud, because that is what my USB to serial driver has always used as default. If hevnsnt's port was writing data at something like 19200 baud, this is *exactly* what I would expect to see. I prepended a smaller note to the top of the email I was writing:

from Jestin Stoffel
to hevnsnt, SomeoneKnows
date Fri, Feb 6, 2009 at 3:51 PM

Perhaps the /dev/ttyUSB0 baudrate is different than what I have used.
Try changing the baudrate in the arduino sketch file from 9600 to 4800
or 19200, and then recompile and load to the arduino. If those don't
work, try some other rate (2400, 4800, 9600, 19200, 38400, 57600,
115200, 230400, 460800, 921600). You can read my rationale below.

-- The rest of this is what I wrote before realizing it's probably the
baudrate --

At this point, I was confident that we had a solution, and sure enough, I got this back:

from hevnsnt
to Jestin Stoffel, SomeoneKnows
date Fri, Feb 6, 2009 at 4:47 PM

I got it working..

Did a stty 19200 > /dev/ttyUSB0
compiled code for 19200

working great!!!!!

So, by the time the after party at HACDC came around, the LabyWiinth was ready for action!

Now the only problem is that this game is *ridiculously* hard.