How to Test Voting Systems?
I got a great and deceptively simple question recently: what guidelines should be used for testing of voting machines?
OK, so testing is not the most fascinating topic, even for geeks, but this is testing of an important kind. Without the state and Federal governments' independent test and certification program, we could well see voting system products with completely unsubstantiated claims of reliability, security, etc. Instead, today's certification programs ensure that before a voting system product is legal for use (in most states), it has first been subjected to some testing of whether it is fit for use. Now, of course, opinions differ on just how fit for use existing system actually are, but let's set that aside for today, and focus on the question.
For starters, we need to broaden the question a bit: how to test voting systems. Today, it doesn't make much sense to test a voting machine, such as a precinct ballot scanner, without also testing the one and only other machine that consumes the scanners' output to create election results; even if the scanner is perfect, the tabulator's defects could create incorrect election results. The whole system needs to be tested. You might ask why the phrase "one and only" -- that's because every voting system is proprietary, with proprietary interfaces between components, and hence no interoperability between components.
For testing a voting system, there are several pieces that I think are required, and only one exists today. The existing piece is the parts of the not-yet-adopted guidelines from the EAC (with help from NIST) called the VVSG-NG, which tells test labs how to test for reliability, physical characteristics, accessibility etc.
The missing parts are those that define what a voting system actually is! A good test lab project - I am speaking aspirationally here - would include comparing the actual system to a specification of a voting system; and comparing the documentation of each component to a functional specification that describes what the individual component is supposed to do; and developing test plans to determine whether those functions are implemented correctly; and executing the test plans in a repeatable way.
And that's just black box testing against currently non-existent specs. Then there is the task of reading all the source code to determine whether the system implements all and only the documented functions; and to augment the test plans with "inside information" so that the test plans reflect the implementation-level details that are not present in specs or docs; and ditto from reading design docs as well as source. This type of design and code reading is essential to making an assessment of whether the test plans and procedures are effective for the particular system being tested. If the test lab people don't read all of this, or don't understand all of it (e.g., limited design docs, obscure source code), then they don't know how to test some chunks of the system that for all they know might undercut the system's ability to perform its required functions reliably. So there needs to exist some guidance for this part of the work, and accountability requirements that enable the test lab to show that it followed the guidance.
Lastly, though not strictly necessary, but in the Good Idea department, would be testing against data representation standards that define the format for a voting system's components' input and output data, logging data, etc. That would help a lot with interoperability and transparency.
That's a lot of non-existent ingredients to what I think would be a good recipe for meaningful testing and certification of voting systems. But please don't think I'm complaining that we can't have a good test and certification scheme because there are so many gaps. Quite the reverse. I think that a much improved certification scheme is possible, because we're working on filling those gaps.
-- EJS