github/sowbug

Debugging a G35 Installation

Editor’s Note: This draft post was discovered tonight. It was probably written in 2012.

“Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.” – Sherlock Holmes

My home Christmas light installation is operational after debugging for several hours over the past week. They’d work for a while, but reset every few programs, and then freeze with odd colors. This was unexpected, as this physical deployment usually runs only my production-quality code, installed only after a bunch of testing in my office. Moreover, these lights had run flawlessly for every applicable holiday (Christmas, Halloween, Valentine’s Day, and even a brief run to congratulate the SF Giants) since 2011.

The lights seemed cursed. I first suspected a bad power supply, which typically explains all unreliability with my electronics projects. Nope, swapping in a new one didn’t change anything. OK, so my Like A G35 shield was flaky? Again, swapping in a second exhibited the same behavior. One strand has a short? Either strand flaked out the same way when run alone. And because my workflow normally involves swapping one of two Arduinos into the installation, I’d ruled out a bad board, too.

I was about to revert to an earlier Git commit (though there weren’t any relevant changes since last known good), when I thought that maybe this was that once-in-a-blue-moon situation where the toolchain was to blame. This happened a lot when I was a junior engineer, and has happened much less now that experience has replaced my own super powers. I remembered that a few weeks ago I’d installed a beta build of the Arduino IDE for the Teensy 3 that I got via Kickstarter. That beta added an ARM compiler… but could it have introduced a regression?

I downloaded stock Arduino 1.0.2 and rebuilt. The lights have been running about an hour now without issue. They’d have reset many times by now.

I’m not sure what lesson to learn here, or how I could have figured this out sooner. It’s a very healthy attitude to suspect one’s own work as the source of errors. That’s almost always where the bug is. But after you’ve eliminated yourself as a possible cause, it’s time to suspect invariants, such as your toolchain.