Reply #10: Update from a Transparency Project volunteer

[GuvWurld]

Another volunteer has told me he'll be posting video of the Registrar discussing this. Meanwhile, the most important thing I see in the report below has to do with audit logs that cover the system's tracks when reporting inaccurate results, thus destroying assurances of built-in redundancies and making a mockery of logic and accuracy testing.

---

Edit to add: comments below now posted online
http://hum.dreamhosters.com/etp/news/20081204.html

Commentary on Premier Election Systems' GEMS central tabulator not
counting 197 ballots in Humboldt County.

by Parke Bostrom <[email protected]>
20081204

Disclaimer

My name is Parke Bostrom. I am a volunteer working on the Humboldt
County Election Transparency Project (ETP). (See
http://hum.dreamhosters.com/etp/ for more information on the ETP.) I
do not work for the County of Humboldt and I am not directly involved
in the primary counting of the vote in Humboldt County. Much of the
information below is second or third-hand and may need to be
corrected, updated or clarified as more reliable information emerges.

Commentary

On Sunday November 30th the ETP volunteers (with assistance from
Humboldt County elections office employees) finished the ETP scanning
of the ballots from November's election. We finished several days
prior to the deadline for the Elections Office to certify the results.

According to the ETP's initial (and largely unanalyzed) results, we
scanned 216 more ballots than the county's primary count system
(Premier's GEMS). As the ETP is still a new, developing "beta-ish"
project, it seemed most likely to me that the extra ballots were due
to some error in the execution of the ETP. I expected us to be able
to identify the cause of the error, correct our results, and improve
our procedures in future elections.

On (I believe) Tuesday December 2nd, County Registrar of Voters
Carolyn Crnich certified the election results provided by GEMS. On
Wednesday December 3rd, ETP volunteers working together with Crnich
and the elections office identified the cause of the largest part of
the 216 ballot discrepancy. The GEMS totals (which had just been
certified) failed to include 197 ballots from "Deck 0". The ETP had
also scanned these ballots, so they showed up in our totals. The ETP
therefore identified a serious error in the vote count generated by
GEMS.

Some background information on the counting process helps explain the
nature of "Deck 0". Deck 0 contains some (but not all) of the
absentee ballots from a single precinct (in this case, precinct
1E-45). Absentee ballots are sent out about a month before the
election, and are returned to the election office continuously over
the entire month leading up to the election. At multiple points in
time, whenever the number of returned ballots is sufficiently large,
all the queued incoming ballots that have been received thus far are
sorted by precinct into "decks" of ballots. A deck contains ballots
from only one precinct, but as this sorting happens at multiple points
in time, the ballots from a single precinct are almost always spread
across three or four decks.

Several days before the election (I believe the Saturday before), the
Elections Office began feeding the decks of ballots into the GEMS
central tabulator. The Elections Office is not allowed to print out a
report of the vote totals until the polls close on election night, but
the computerized counting process can begin several days before
election night. The reason for starting in advance is that machine
counting of thousands of ballots, while faster than hand counting, is
nonetheless a time consuming process.

Deck 0 was the first deck of ballots counted by GEMS. It is called 0
(instead of 1), because it is common practice for computer programmers
to start counting at 0.

Upon discovering that deck 0 was not included in the certified
election results, Crnich contacted Premier. I did not observe that
conversation, but from what I understand, Premier claims the deck 0
results are sometimes automatically deleted by the system when a
subsequent deck is intentionally deleted and rescanned. It is
expected that at least several later decks will be deleted and
rescanned as part of normal procedures. The reason for this is that
there is no way to amend a deck's results, so if you need to amend a
decks results, you have to rescan the entire deck. However, when you
intentionally delete a given deck's results (prior to rescanning), you
expect that the other decks results will remain unchanged.

Premier claims (or so I hear tell) that Deck 0 is sometimes "randomly"
deleted because different programmers worked on GEMS at different
points in time, and some programmers start counting at 0, while other
programmers start counting at 1. (Given my own experience as a
programmer, it may be the case that deck 0's results are not really
deleted, but were instead "skipped" or not included in the updated
report. This is not a terribly happy distinction, however, because
the updated report is still wrong, indeed just as wrong as if deck 0's
results had in fact been deleted.)

The ETP had 216 extra ballots. 197 of those were from deck 0. That
means that the ETP now has 19 extra ballots to account for. 19 extra
ballots could be due to a variety of causes, including: GEMS double
feeding ballots (counting 2 ballots as 1), and various operator
errors/anomalies on the part of the ETP volunteers scanning the
ballots. In the June election the ETP counted 2 more ballots than
GEMS. 216 extra ballots are truly alarming. 19 extra ballots (out of
60,000+) are worth investigating and accounting for.

After additional investigation, we learned the following information:

The precinct voted ballots were fed into GEMS on election night, as is
standard procedure. From November 5th until November 23rd, no
additional results were fed into GEMS. During this period, the
elections office was busy auditing the rosters containing voters'
signatures, preparing the remaining vote-by-mail ballots for scanning,
and reviewing and preparing provisional ballots for scanning.

Scanning resumed on November 23rd. Prior to scanning, the elections
office printed out a GEMS report. The report contained the same
totals from the final report on election night, as it should. The
remaining ballots were all fed into GEMS during November 23rd through
25th.

When the dropped ballots were discovered, Premier requested copies of
the GEMS database in order to try to determine what went wrong.
Premier claims that deck 0 was deleted sometime during the processing
of decks 131 through 135.

Crnich remembers that during the scanning of deck 132, the GEMS
program was restarted (indeed, the whole computer hosting GEMS was
rebooted) *after* deck 132 had been "opened" but *before* any ballots
had been scanned into deck 132. The reason for the reboot was that
the ballots for deck 132 were of a different type (vote-by-mail,
provisional, mail-ballot-precinct, and precinct voted ballots all
being distinct ballot types). Standard procedure is to reboot the
computer when switching ballot types. (Requiring reboots seems to me
like a kludge/workaround designed to cover up bugs and design flaws in
GEMS.) Our best guess is that restarting GEMS while deck 132 was open
causes GEMS to silently delete deck 0. At no point did GEMS give any
indication that anything was going wrong.

There are other anomalies in the GEMS audit log. There are actually
three "deck 0"'s: deck 0 of vote by mail, deck 0 of provisional
ballots, and deck 0 of mail-ballot-precincts. Deck 0 of 1E-45
(containing the 197 dropped ballots) does not show up in the audit log
and those ballots are not included in the final report. The other 2
deck 0's also do not appear in the audit log, but apparently their
votes are included in the final report. This means the audit log is
not truly a "log" in the classical computer program sense, but is
rather a "re-imagining" of what GEMS would like the audit log to be,
based on whatever information GEMS happens to remember at the end of
the vote counting process.

The version of GEMS the county is using is 1.18.19. This version is
also used by Santa Barbara and San Luis Obisbo counties in California.
It is also used by the entire state of Maryland. Premier has an
internal memo dating from 2004 describing the deck 0 problem. The
workaround to the problem is to manually create and delete all the
deck 0's prior to scanning any ballots. Santa Barbara and San Luis
Obisbo counties are aware of the workaround and have done it in all
recent elections. However, the Humboldt elections office has been
unable to find any official Premier documentation or addenda
describing the workaround. Apparently, the workaround was
communicated from Premier to counties via word of mouth, and the
knowledge of the problem and the workaround may have departed from
Humboldt County due to employee turnover. Additionally, in 4 years
Premier has not fixed the underlying problem.

The California Secretary of State knew neither of the problem, nor of
the workaround. The problem passed certification prior to the 2006
top-to-bottom review, made it through the 2006 top-to-bottom review
undetected and existed in the version of GEMS that was re-certified
after the 2006 top-to-bottom review.

Noteworthy points:

1. The ETP can effectively discover and identify errors in the
primary count, errors that otherwise would not have been detected.

2. The deck-0 problem may be affecting other jurisdiction using GEMS
version 1.18.19, or, for all we know, any version of GEMS whatsoever.

3. Even though this is the second election for the ETP, in many ways
it is a first. In the June election, we used Microsoft Windows
software to control the scanner. Between June and November, with the
help of a third party open source developer, we gained the capability
to control the scanner using the open source Linux operating system.
This allows us to publish all the source code used to control the
scanner, thereby greatly increasing the transparency of the technology
we are using. Additionally, in the future I hope to automate the
process of discovering and identifying discrepancies between ETP
results and primary count results. In a few more elections, the ETP
should be able to identify errors in the primary count *before*
certification of the election results, rather than *after*. So the
ETP will only get better with time.

4. To the best of my knowledge, GEMS does not have the capability to
export machine readable per precinct/per deck results in an easily
analyzable format. If GEMS had the capability to export machine
readable information, it would be a lot easier to audit GEMS (even
without the ETP). AS it stands, it is very difficult to notice that
decks or precincts have been dropped from the final report due to this
serious deficiency in the design of GEMS. (If GEMS does have such a
data export capability, no one has been able to tell me how to use it
- not even Premier's election support specialists.)