Saturday, April 25, 2015

Unit Test of Embedded System Code

Manifesto: Unit Test of Embedded System Code

I'm not sure eactly what this means, but I intend to apply it, starting now.
And why not give my efforts a grandiose name?

Having just spent an interesting Saturday in a Code Retreat graciously hosted by OC Tanner, including fabuluous food, I hereby resolve to start applying some of what I learned, even though I am not at all certain what that means. Why let ignorance stand in the way of application? Here are my intended actions. To anyone who knows what they are doing these ideas may sound infantile, but so be it. Has anyone else already figured this all out? A minute of Googling the title of this post does produce some hits such as this paper from Parasoft.

What does it mean to test embedded code?

I'm envious of programmers who can live in the perfect world where their universe is just software: PC and web apps, databases, transaction processing, etc. Sure there is hardware there executing code and storing data, but you never have to touch it or debug it. Your PC either works or it doesn't. You have powerful debuggers and analyzers and never need to touch a multimeter or oscilloscope.

Embedded systems are a lot messier. For example I am working on a driver for the AD7794, a 24-bit delta-sigma analog to digital converter, with PGA and a bunch of other programmable features. It works up to 125 C, too!  Good news: it does a lot and the options are useful. Bad news: it is really complex and so the driver has to manage that complexity. It uses a rather unusual (in my experience anyway) SPI interface which does not synchronize with assertion of the slave select. My first problem was that I could not find any good examples of general purpose C code, so I started writing my own from scratch. The simplest reading of a device status register seemed to return nonsensical (but not random) data. How do I debug this? The failure might be in the part or in some hardware-access routine.

Then, another system uses a confusing series of shift registers to drive AC and DC loads in a system under control. How do I really test my code without also testing the outputs of these shift registers, for which I need special hardware? So in fact we are spinning a quick-turn circuit board to display the state of all system outputs and provide a means to test system inputs. It's not fully closed-loop and therefore capable of fully automated functional test, but it's better than the current state of affairs where we can't even tell if we are driving most of the loads (you can't see a low-wattage heat pad turn on).  This board will let us visually watch ones and zeros being walked across outputs both low-voltage DC and 120 VAC.

Testing once is not enough

Maybe in the perfect world of software it is, but not in the embedded space. When you go home at night why not leave test code running with some way of logging errors? You might be shocked to find that there is some failure 0.00001% of the time. If you can run a million or more tests overnight you might be lucky and see 10 such failures. This is indeed lucky: better to find and fix it now early on than have customers later report mysterious failures in the field.

Consider functional test hardware as part of the original system design

What good is a system which can't be functionally tested? Usually this means special custom hardware (it can be simple, but it is still custom) just for the purpose of functional test. So consider that as part of the original system design and include it in the budget and schedule. If it's a design for someone else, xplain to your customer the benefits of your doing so.

Add a C++ test class to every device driver

I'm working on several C/C++ device drivers for prototypes of commercial products. They are really C with just the thinnest layer of C++. I normally write tests as I go but don't save all of them. Many times they end up as chunks of commented-out code (a bad pattern/habit I'm working on correcting), or old versions of the driver which get discarded or lost in an archive. Instead, why not write a deliberate test class which can then be invoked from a simple driver test program and run throughout the product's life, as needed? This way the tests stay as part of the project but don't have to get built into the shipping binary. It seems like this should work. I'll try it straight off and let you know. Maybe everyone else in the world already does this and I'm the last to adopt it.

Mock up data to test algorithms

What I mean here, is that if there is some data processing needed to convert from a sensor's native format (e.g, TMP102 stores only negative temperatures in a 13-bit two's complement requiring two separate byte reads) to a useful one (such as degrees C or F), it is a good idea to pass simulated data of all possible values to that routine so that you know it works and the converted temperature values will be correct. I've seen open source drivers that just didn't bother with this and ignore the special conversion needed for negative temperatures. This is especially true if the algorithm has other dependencies such as using calibration coefficients pulled from device registers or calibration memory. What if those calibration values are wrong, or span all possible values for the given data type: will a calculation overflow or wrap around? This overlaps into boundary testing.

Test all boundaries

If a 12-bit ADC returns a value into a 16-bit data type, and it is right-justified, the top nibble should always be zero filled. But what if it isn't: either the converter fails, or it is not initialized properly, or noise infects the data lines? Will downstream calculations or conversions fail? Test all such routines with the maximum possible data values. Clip values to the maximum permissible in your code if that makes sense. For example in a PID algorithm, there is the danger of "integral windup" but this can be handled by limiting the range of your integral variable.

Catch all exceptions and at least report a unique code

Sometimes if there is a possible, but inconceivable, error state I put a message such as "this should never happen: error XXX" where XXX is a unique integer so every error is distinct. Imagine my suprise when I see this very error message later in system test. Clearly something big has not met my expectations, but at least a) I know it happened and b) I have a clue where to look in the code.

Use a C/C++ documentation tool

Documentation was never mentioned in the code retreat. Apparently it is not even taught in most CS and CE curricula. What's wrong with this picture? That's a topic for another day. One thing I like and miss about Java is javadoc: it is baked right in to the tools so there's no valid reason to not use it. With C/C++ you have to take extra steps to even find and install a tool. I'll try doxygen first.

Wednesday, February 18, 2015

Crawling up the Linux and Python Learning Curves

Why I'm writing about this:

  1. It helps me think it through on a deeper level and understand it better if I can explain it in complete sentences.
  2. My experience might be useful to others. The blogs I have read about other setup experiences have been very valuable.
  3. It's my first attempt to give something back to the open Linux community, which has made possible an amazing collection of software and resources.
  4. So I don't forget what I have done, and why.
  5. To share with attempts to start a local Python For Kids programming group, starting with my almost-13 year old son.

Over a year ago I set the goal to learn Linux and Python. Why? Because Linux servers pretty much run the world. Even little off the shelf NAS systems such as those by Synology (I use them) are running Linux. The more-or-less preferred Linux admin language is Python, so it ships with all standard Linux distros, even small ones like on Raspberry Pi and Beaglebone Black.

Plus I am in the midst of a commercial project using embedded Linux on a Beaglebone Black, and at the core of it, the Linux on it is pretty much the same as other Linux.

For over a year now, I've been running Ubuntu (now 14.10) on a Lenovo x131e netbook, with a Core i3, 8 GB RAM, 320 GB HDD, 1366x768 TFT screen. It is actually dual-booted with Windows 7 Pro but it has been months since I've booted into Windows. I need to do that and catch up on Windows updates. But I digress. This is the system I take with me every day when I teach at Salt Lake Community College, so I use it to log into the wireless network there. I use Libre Office to edit class materials. I use Chrome and FF to acces the Canvas instructional framework. So far, I have found it as easy to use as Windows 7 Pro. Some things are actually better: battery life seems a little longer, connection to Bluetooth mice (Logitech and MS Sculpt) is more reliable. Boot times are shorter. Ubuntu 14.10 seems a little less stable than 14.04 LTS: occaisonal lock-ups, and sometimes I start an application (e.g. Chrome) but it can't create a window in the display even though it's running (htop or System Load Indicator show it active). But overall, at least as stable as Win 7 Pro.

I travel around locally a lot and find it easier to connect to numerous WiFi points in Ubuntu. Also I tether to Samsung Relay 4G phone, and this is painless: the phone appears as a wired Ethernet connection to Ubuntu, and data speeds are at least 2X higher on average than using the phone as a WiFi hotspot.

So, spurred on by this success, I have built up an AMD quad-core A8-5600K desktop with a 120 GB SSD, 1 TB HDD, 8 GB RAM, etc. It also dual boots Windows 7 Pro and Ubuntu (now 14.10). It's now my second desktop system.

Then a few weeks ago I purchased a Lenovo Yoga 11e netbook with quad-core Intel 2930 CPU, 4 GB RAM, 128 GB SSD, and 1366x768 IPS multi-touch screen. It came with Windows 8.1 Pro but I have pulled that SSD and replaced it with a Sandisjk 128 GB and loaded Linux only on it. I'm typing this on that system, running Ubuntu Mate 14.10. An MS Sculpt BT mouse is connecting very reliably. But the touchscreen doesn't work at all, and the trackpad is so-so. But the screen is great: bright, wide viewing angles, good color, but glossy and shows all fingerprints. Also suspend doesn't seem to work correctly at the moment. I'd like to get all these things sorted out.

I'm making some Google docs as I go and will post view-only links to those here.

About me:

  1. Not a windows-hater. I like Windows 7 Pro for the most part and use it daily for my work. Many programs I need to use (Altium Designer for one) are only Windows. USB on Windows is a nightmare: I use many development boards with USB interface and they all seem to assume they will be the only USB device installed on my computer. Wrong. Too often, clashing ensues. It can be a real headache, requiring uninstalling and rebooting, ad nauseum. Still, USB in Windows 7 is a lot better than under XP.
  2. Not in love with Windows 8.1 since I don't want my PC to look like a tablet or phone, and I don't want to connect with the MS cloud. I don't spend all day on social media or gaming. I don't want a dumbed-down "friendly" screen of tiles: I was fine with the Windows 7 GUI. Is USB a lot better under Windows 8.1? I don't know.
  3. I'm an electrical engineer with background in hardware design of mixed-signal (that means digital and analog) systems, including embedded controllers.
  4. Embedded Java enthusiast, though sadly in the US that market has pretty much dissipated due to neglect and stupid licensing by Sun ($100K to make Sunspots commercially? Really? No wonder there were zero takers). My company, Systronix, produced TStik, JStamp, JStik, SaJe, and other (at the time) ground-breaking embedded Java systems.
More to come.

Monday, February 16, 2015

Wired Ethernet at Home: Still a Great Idea

Wired gigabit Ethernet is going into our home remodel and I am sooo glad we did that. I will detail how I did it, including setting up dual digital TV tuners, a media server, DVR, HTPC, large-screen projection system, dual-band WiFi, VOIP phone, better-than-average WiFi security, and most of a 1000-foot roll of Cat5e. I've tried to do it all relatively cheaply too.

Why is (wired) Ethernet better than WiFi?

  1. More secure, if you care about that. Much harder for someone to sniff your wired Ethernet.
  2. Much better throughput. 1000BaseT is over 4X faster than 2.4 GHz WiFi and 2X faster than 5 GHz.
  3. No channel congestion. Use a sniffer (e.g. Fing or WiFi Analyzer) and it's astonishing how clogged 2.4 GHz is. No one else is on your wired Ethernet but you. 
  4. Easier management of guests: they just plug in. No need to enter their MAC addresses into a table of allowed WiFi clients.
  5. Wire is cheap: 1000 feet of Belden UTP Cat 5e is $136 online. Jacks and other hardware are reasonable too. You do have to wire up the jacks and connect to switches. But this is easier than you might think. If I can do it, so can you.

    Why is WiFi better than wired?
    1. Those damn wires. It is a bit of a hassle plugging in. WiFi is so easy. But like too often eating fast food, easy does not mean best or even good.
    2. Those damn wires: retrofitting into an existing building can be a hassle. In my case, we are tearing out most of the walls anyway so we have a golden opportunity.
    3. Phones, tablets, and even some netbooks don't have an RJ45 jack, so they have not choice but WiFi.
    4. No switches, jack, routers to deal with. Well, at least fewer than with WiFi.
    More to come!

    Tuesday, January 27, 2015

    Fuel Surcharge???

    A quotation received today for shipping some heavy countertop material by truck has a 17% "fuel surcharge". Huh? Diesel is at its lowest price since 2010. This is a 26% decrease from a year ago. If the trucking industry wants our sympathy when fuel prices are high it needs to reciprocate when prices are low.


    Thursday, October 16, 2014

    Connecting an un-updated Kindle Keyboard to Windows 7

    My latest time sink involved the gift of a Kindle Keyboard device, also called the Kindle 3rd generation or K3. It looks like this:
    with round keys. I've never used a Kindle but the idea of really long battery life sounded great, and I don't mind reading from my Nexus 7 tablet (some people can't stand reading from a screen vs a paper book; I am not one of them).

    But I could not register the KK to my Amazon account. This KK has 3G and WiFi. I could get the WiFi to work and connect to my home access point. But attempts to register I finally called Amazon and was pleasantly surprised to get through to a real person in only a couple of minutes. It turned out my KK had a really old software version (3.0.2)  and that was apparently to blame for inability to register via WiFi. The 3.1 update has to be installed manually.

    But the KK would not connect to a Windows 7 PC. I tried two different systems, and multiple USB ports on each. After over an hour of googling, restarting, lather/rinse/repeat, I finally was able to get Windows 7 to recognize the Kindle and load a device driver. Here's what finally worked:
    1. Turn off WiFi on the KK. I don't know if this step was necessary, but it was recommended by others.
    2. Unplug USB between the KK and the PC
    3. Power off the KK by holding the power slider to the right until the screen blanks
    4. Restart the PC
    5. Connect USB between the PC and the powered-off Kindle.
    6. Woohoo! The Kindle came up in a different screen advising about using USB to charge while still being able to read or use the Kindle store. And the PC recognized the USB device as a Kindle. 
    7. I dragged and dropped the Kindle 3.1 update onto the Kindle device in Windows Explorer, the file copied, then I ejected the Kindle and uplugged the USB cable. Amazon has instructions for this.
    8. Start the software update on the KK: 
      1. From Home, press the Menu button, and then select Settings.
      2. Press the Menu button, and then select Update Your Kindle. This option will be grayed out if the most recent update has already been installed or if the file transfer was not successful.
      3. Select OK to perform the update. Your Kindle will restart twice during the update. After the first update, you will see Your Kindle is Updating.

    Updating to Kindle 3.1 let me register wirelessly and then it also became able to complete additional updates wirelessly. I could load update 3.3 and then 3.4 via WiFi. After the updates I still can't connect if the KK is powered on, so there is something not quite right about its USB interface.

    OK, now I have a "vintage" (they came out in 2010) Kindle Keyboard reader! They are still selling for about $90 used on Amazon, so that is a good sign. And it turns out I also have the Kindle Keyboard cover with light, which works really well, thanks to the high contrast e-ink display:

    Monday, July 28, 2014

    Will that be paper - or not?

    I want to be more efficient, I really do. I want to follow the advice of experts, since they are, well, experts. So fortunately this email arrived today.
    Paperless Whitepapers then?
    My mouse pointer was poised over the "Register Now" button. I was imagining getting rid of the office laser printer. Never buying reams of paper at Costco again. No more paper cuts!

    But then, in the nick of time I was saved from making a horrible mistake by the next email.
    Why is it not surprising that the Government version of InformationWeek tells me I can't get rid of paper? After all, it works sooo well for Congress.

    What if I sign up for both? Would it be like registering for a "matter" and an "anti-matter" webinar, ending in my annihilation?

    Tuesday, September 24, 2013

    JavaOne 2013: (Keep On) Making the Future Java

    Mark Twain apparently never said "the coldest winter I ever spent was summer in San Francisco". But after Saturday, the weather has been great. So here we are in sunny, warm San Francisco for JavaOne 2013. We have been commuting on the Ferry from Alameda to Fisherman's Wharf. This has to be the Best Commute in the World.

    It has always seemed a waste of resources to make the bodywork of cars different every year for no reason other than appearance. So it doesn't bother me that Java One was not re-themed this year but was again tagged with "Make the Future Java". Apparently the future is still not here, so we all need to keep working on it, at least for another year.

    Again this year I was fortunate to speak, in collaboration with my long time collaborator in such things, Jim Wright, in BOF 7452 on the topic of SmartStat Micro, an open-source HVAC thermostat. This is an ongoing project and a topic for future posts.

    This year things started with a community leads meeting and excellent dinner (Hotel Nikko food seems to me to be a cut above their neighbor hotels), MC-ed by Sonya Barry, who is currently battling cancer (read about that in her blog). This year events proper kicked off on Sunday. Today is Tuesday and here are some random impressions.

    This is the first year that wireless has actually worked well! Kudos to the subcontractors who set it up. This has got to be hard: providing wireless to accomodate 60,000 people. Actually the JavaOne portion of that is a lot less: maybe 10,000 (just a guess), but still that is a lot of traffic.

    Web Fundamentals by James Ward (at 8:30 this morning, groan) was fabulous.  In a dense hour he laid out what hours of googling and reading books (online and print) has failed to give me. He works for TypeSafe and they have some nice looking web tools which I am now inspired to try. It's a 200 MB download... good thing the wireless works.

    Sun SPOTs are dead. When current stock is gone no more will be made. This is the final chapter in a sad story. SPOTs could have been the Arduino for the Internet of Things. But at a commercial license fee of $100K, what can you expect? Suppose Arduino had a $10K license fee for commercial use? It would have flopped too. Now SPOTs are open source, but no one seems to care. There are better options out there: you guessed it, mostly based on Arduino. $400 gets me two wireless SPOTs and a base station, or for $75 I can get a pair of Arduino WiFi Spark Core devices.

    You never have to go more than half a block in San Francisco to get a decent latte. And at JavaOne you can get as many as you want for free in the "Buzz Shack". Someone got this right!

    There is emphasis this year about "embedded Java"  - and here we mean some attempted rejuvenation of Java Micro Edition- and I attended several sessions related to this. Judging by attendance it may be too little, too late. Sun let Java Micro Edition languish for years, and seemed to care only about Java ME on phones. Now, what is the compelling use case? On the other hand, "embedded" these days can mean a $45 BeagleBone Black with orders of magnitude more power than an embedded device of yore. So Java SE Embedded is more apropos with current hardware. "SE Embedded is the new ME".

    In past years Oracle was pushing SE Embedded on the Plug Computer which is pretty cool for around $100, but it is not hackable or embeddable. The only I/O interface is: USB. So forget about adding your own low-cost sensors, or controlling devices. No I2C or SPI, so you can't add your own OLED displays or color touchscreens

    Oracle this year seems almost obsessed with the Raspberry Pi. Pi is cool - don't get me wrong - but it is for educational use only, and is not intended or appropriate for commercial use (a whole other topic, Google it if you are curious, or just try to get a data sheet for the Pi's Broadcom processor). Instead there are newer embedded Linux systems such as BeagleBone Black which are intended for OEM use, so I would expect Oracle to be all over the BBB. Maybe next year. In any case, if it will run on the Pi it should also run on the BBB. So if it can be made to run acceptably on Pi, it should run that much better on other more capable targets such as BBB. That's a wonderful side effect of both Pi and BBB running embedded Linux.

    So *why* should we run Java on a small embedded Linux system? What is the compelling use case? At JavaOne there is sometimes the "we should do it in Java because Java is cool" mentality, sometimes with little consideration for whether it is practical or benefical. Ten years ago this held some weight. But now Java is the status quo and newer languages (e.g. Python) are claiming better ease of use and programmer efficiency. 

    For one thing, SE Embedded and Netbeans have some impressive debugging, monitoring, and tuning tools such as VisualVM and VisualGC which were featured in the presentation Deploy: Getting Up to Speed on Oracle Java SE Embedded, TUT7730. These are pretty impressive tools, if you care about what your application is really doing: how much memory is it consuming, where performance bottlenecks are occuring, and so on. Remote debugging is possible on the stable Java 7 SE Embedded, too, but only with the latest NetBeans 7.4 nightly builds. 

    Which brings me to the point that Oracle sometimes takes a long time to make a decision and actually do something, but if they ever reach that point, they tend to do it well, and they tend to stick with it. By "do it well" I mean including a pretty robust JVM and usable tools for debug, monitoring, and tuning. (Such tools are grossly lacking in the Arduino world). Sun would often change focus every year,  driven by the current winds of marketing hype. This never played well with industrial developers who typically have to maintain and support systems for at least five years, and often at least 10. Embedded Java developers have been burned in the past so we are now more than a little shy about considering Java again. What if Oracle changes their mind in a year or two? Witness the dead end which Sun SPOTs have now become (in fairness that is not Oracle's fault, but you get my point).

    This is one aspect of Arduino which helps make it popular: the whole Arduino IDE and its libraries are open source, and since the code is all standard C/C++ and the underlying compiler is GCC, how badly can you be stuck in the unlikely event the developers abandon further work? If we only have access to the binary releases of the JVM for ARM, what do we do if Oracle abandons the effort?

    More thoughts on this to come...