Voting System (De)Certification, Reloaded (Part 3 of 2)

Thanks to some excellent recent presentations by EAC folks, we have today a pleasant surprise of an update to our recent blogs Voting System Decertification: A Way Forward (in Part 1 and Part 2). As you might imagine with a government-run test and certification program, there is an enormous amount of detail (much of it publicly available on the EAC web site!) but Greg and I have boiled it down to a handful of point/counterpoints. Very short version: EAC seems to be doing a fine job, both in the test/certification/monitoring roles, and in public communication about it. At the risk of oversimplifying down to 3 points, here goes: 1. Perverse Incentive

Concern: ES&S's Unity 3.2.0.0 would be de-certified as a result of EAC's investigation into functional irregularities documented in Cuyahoga County, Ohio, by erstwhile elections direction Jane Platten (Kudos to Cuyahoga). With the more recent product 3.2.1.0 just certified, the "fix" might be for customers of 3.2.0.0 to upgrade to the latest version, with unexpected time and unbudgeted upgrade expense to customers, including license fees. If so, then the product defect, combined with de-certification, would actually benefit the vendor by acting to spur customers toward paid upgrades. Update: Diligent work at EAC and ES&S has resulted in in ES&S providing an in-place fix to its 3.2.0.0 product, so that EAC doesn't have to de-certify the product, and customers don't have to upgrade. In fact, one recent result of EAC's work with Cuyahoga County, the county was able to get money back from the vendor because of the issues identified.

Next Steps: We'll be waiting to hear whether the fix is provided at ES&S's expense (or at least no cost to customers), as it appears may be the case. We'll also be watching with interest the process in which version 3.2.0.0+ fix goes through the test and certification process to get legal for real use in elections. As longtime readers know, we've stressed the importance of the emergence of a timely re-certification process for products that have been certified, need a field update, and need the previously used test lab to test the updated system with testing that is as rigorous as the first time, but less costly and more timely.

2. Broken Market

Concern: This situation may be an illustration of the untenable nature of of a market that would require vendors to pay for expensive testing and certification efforts, and to also have to forego revenue when otherwise for-pay upgrades are required because of defects in software.

Update: By working with the vendor and their test lab on both the earlier and current versions of the product, all customers will be able to obtain a no-fee update of their existing product version, rather than being required to do a for-fee upgrade to a later product version. Therefore, the "who pays for the upgrade?" question applies only to those customers who actually want to to pay for the latest version.

Next Steps: Thanks to the EAC's new process of publishing timelines for all product evaluation versions, it should be possible to compare the timeframe for the original 3.2.0.0 testing, the more recent 3.2.1.0 testing, and the testing of the bug fixed version of 3.2.0.0. We can hope that this case demonstrates that a re-certification process can indeed be equally rigorous, less costly, and more timely.

3. Lengthy Testing and Certfication

Concern: The whole certification testing process costs millions and takes years for these huge voting system products of several components and dozens of modules of software. How could a re-test really work at a tiny fraction of a fraction of that time and cost?

Update: Again, thanks to publishing those timelines, and with experience of recent certification tests, we can see the progress that EAC is making towards their goal that an end-to-end testing campaign of a system to be less than 9 months and a million dollars, perhaps even quarter or a third less. The key, of course, is that a system be ready for testing. As we've seen with some of the older systems that simply weren't designed to meet current standards, and weren't engineered with a rigorous and documented Q&A process that could be disclosed to a test lab to build on, well, it can be a lengthy process -- or even one that a vendor withdraws from in order to go back and do some re-engineering before trying again.

Next Steps: A key part of this time/cost reduction is EAC's guidance to vendors on readiness for testing. That guidance is another relatively recent improvement by EAC. We can hope for some public information in future about how the readiness assessment has worked, and how it helped a test process get started right. But even better, EAC folks have again repeated a further goal for time/cost reduction, but moving to voting system component certification, rather than certifying the whole enchilada - or perhaps I should say, a whole enchilada, rather than the whole plato gordo of enchilada, quesadillas, and chile relleno, together with the the EMS for it all with its many parts - rice, frijoles, pico de gallo, fresh guacamole ... (I detect an over-indlugence in metaphor here!)

One More Thing: As we've said before, we think that component level testing and re-testing is the Big Step to get whole certification scheme into a shape that really serves its real customers - election officials and the voting public. And we're proud to jump in and try it out ourselves -- work with EAC, use the readiness assessment ourselves, do a pilot test cycle, and see what can be learned about how that Big Step might actually work in the future.

-- EJS