Democratic Underground Latest Greatest Lobby Journals Search Options Help Login

BREAKING: Humboldt Transparency Project Reveals Bug In Diebold GEMS Tabulator

Printer-friendly format Printer-friendly format
Printer-friendly format Email this thread to a friend
Printer-friendly format Bookmark this thread
Home » Discuss » Topic Forums » Election Reform Donate to DU
GuvWurld Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 09:01 PM
Original message
BREAKING: Humboldt Transparency Project Reveals Bug In Diebold GEMS Tabulator
I had nothing to do with any of this but I did just speak to one of the Transparency Project volunteers who said the issue was related to "batch zero," a bug known by Diebold to sometimes drop votes from the combined total of all precincts. This bug apparently eluded California's Top To Bottom Review.

Dec. 4, 2008
Election Results: Wrong
Posted by Hank Sims under Government, Politics

The results from the Nov. 4 election that were certified by the Humboldt County Board of Supervisors on Monday were incorrect, according to County Clerk-Recorder Carolyn Crnich in a phone call this morning. Over 200 absentee ballots were not included in the officially certified totals, most of them from one particular precinct in Eureka.

The error was apparently the result of a flaw in the Diebold-manufactured GEMS software used by the county to tabulate votes. Crnich said that she has verified this flaw with Diebold, and has reported the error to the California Secretary of State. Crnich said that she has proof that the ballots were, in fact, run through the machine. The software simply dropped the data.

Most of the uncounted ballots have now been recounted, Crnich said. The final outcome of relevant races remains unchanged.

However, around 20 ballots dropped from Diebolds tabulation remain unidentified.

The error, which appears to be a longstanding flaw in the Diebold software, was discovered by the all-volunteer, open-source Humboldt Transparency Project. This election was the Transparency Projects first full run-through.

Crnich and Transparency Project volunteers will have more to say about the discovery later this afternoon. Updates to follow.


County's certified election results wrong
The Times-Standard
Posted: 12/04/2008 01:16:27 PM PST

The results from the Nov. 4 election that the Humboldt County Board of Supervisors certified this week are incomplete, Humboldt County Registrar of Voters Carolyn Crnich said today.

Crnich said almost 200 ballots were inexplicably dropped from the final election results. The error did not change the outcome of any of the races, Crnich said.

The discrepancy, Crnich said, was discovered through the Humboldt Transparency Project, a program where images of each ballot cast in the election are posted online with sorting software to enable anyone with a home computer to conduct recounts as they see fit.

Crnich said it dropped ballots were traced back to a problem with the Diebold GEMS software the county uses to count votes. She said the problem was reported to the California Secretary of State's Office and to Diebold.
Refresh | +19 Recommendations Printer Friendly | Permalink | Reply | Top
Patsy Stone Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 09:19 PM
Response to Original message
1. It's not a bug; it's a feature.
Edited on Thu Dec-04-08 09:20 PM by Patsy Stone
Printer Friendly | Permalink | Reply | Top
Melissa G Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 09:28 PM
Response to Reply #1
2. Sad, but true.
:hi: Pats!
Printer Friendly | Permalink | Reply | Top
Patsy Stone Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 10:55 PM
Response to Reply #2
7. Hey, girl!
Happy almost Friday! :hi:
Printer Friendly | Permalink | Reply | Top
bleever Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 09:46 PM
Response to Reply #1
5. You pay extra for that.
No really: you do!
Printer Friendly | Permalink | Reply | Top
Patsy Stone Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 10:55 PM
Response to Reply #5
8. Still paying
Printer Friendly | Permalink | Reply | Top
bleever Donating Member (1000+ posts) Send PM | Profile | Ignore Fri Dec-05-08 12:45 AM
Response to Reply #8
9. Ileana is a tiny dancer,
a dancer for money: she'll do what you want her to do.

If the cash is flowing back into Cuba because of relaxed policies re: money and travel, she'll become a believer.
Printer Friendly | Permalink | Reply | Top
stillcool Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 09:29 PM
Response to Original message
3. I hope it catches on!

Supervisors accept election results
Jessie Faulkner/The Times-Standard
Posted: 12/02/2008 01:15:35 AM PST

EUREKA -- The Humboldt County Board of Supervisors praised Registrar of Voters Carolyn Crnich and her staff during a special meeting Monday just prior to voting unanimously to accept the election results.
Actually, Humboldt County's Election Division goes beyond the mandated 1 percent, she said, picking and tallying one precinct from each individual race. With the Eureka City Council races, for example, one precinct will be chosen to count the City Council candidate votes, she said.
The transparency project -- scanning and posting each ballot on the Internet -- may answer questions that previously would have been dependent upon a recount.

On Monday, the open source software, available to anyone free of charge from the Internet, was tallying the votes on the scanned ballots. Crnich said she expects the ballots to be available for public review by Wednesday. Visit to find out where the ballot files will be located.

This allows everybody to look at the ballots, she said.

The computer software also allows users to specify which group of ballots they wish to review rather than having to browse all the ballots online.

County rolls out first-of-its-kind election audit tool
Thadeus Greenson/The Times-Standard
Posted: 06/29/2008 01:29:35 AM PDT
Ever want to call for an election recount? Well, the Humboldt County Elections Office might soon give you the chance to do just that, and more.

The Humboldt Transparency Project, a pilot program which officially kicked off last week, aims to allow anyone in the world the opportunity to access images of every ballot cast in an election, which can then be cross-sorted and tabulated by anyone who wants to conduct a recount from the comfort of their own home.

According to Humboldt County Registrar of Voters Carolyn Crnich, the vision is simple: After the ballots from an election are counted and the results certified, an image of every ballot cast in the county would be put on line. Sorting software would let people view the ballots by precinct, district, race or ballot measure to recount as they see fit.
Though still in its infancy, Crnich said it's possible the project will put the Humboldt County Elections Office on the map.

There are exciting, exciting possibilities here, and it's something that can be duplicated anywhere, Crnich said.
Printer Friendly | Permalink | Reply | Top
dixiegrrrrl Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 09:33 PM
Response to Original message
4. I am sending to Obama team.
After all, he does have a web page that says he wants feedback.

Lots and lots of feedback...
anyone care to send also???????

not ust me, myself and I. ya know what I mean?
Nudge nudge wink wink...
Printer Friendly | Permalink | Reply | Top
Wilms Donating Member (1000+ posts) Send PM | Profile | Ignore Thu Dec-04-08 10:42 PM
Response to Original message
6. And Charlie Brown conceded his election just days ago.

Maybe he should ask for a recount. It was a very narrow race.

Printer Friendly | Permalink | Reply | Top
GuvWurld Donating Member (1000+ posts) Send PM | Profile | Ignore Fri Dec-05-08 02:58 AM
Response to Original message
10. Update from a Transparency Project volunteer
Edited on Fri Dec-05-08 03:21 AM by GuvWurld
Another volunteer has told me he'll be posting video of the Registrar discussing this. Meanwhile, the most important thing I see in the report below has to do with audit logs that cover the system's tracks when reporting inaccurate results, thus destroying assurances of built-in redundancies and making a mockery of logic and accuracy testing.


Edit to add: comments below now posted online

Commentary on Premier Election Systems' GEMS central tabulator not
counting 197 ballots in Humboldt County.

by Parke Bostrom < >


My name is Parke Bostrom. I am a volunteer working on the Humboldt
County Election Transparency Project (ETP). (See / for more information on the ETP.) I
do not work for the County of Humboldt and I am not directly involved
in the primary counting of the vote in Humboldt County. Much of the
information below is second or third-hand and may need to be
corrected, updated or clarified as more reliable information emerges.


On Sunday November 30th the ETP volunteers (with assistance from
Humboldt County elections office employees) finished the ETP scanning
of the ballots from November's election. We finished several days
prior to the deadline for the Elections Office to certify the results.

According to the ETP's initial (and largely unanalyzed) results, we
scanned 216 more ballots than the county's primary count system
(Premier's GEMS). As the ETP is still a new, developing "beta-ish"
project, it seemed most likely to me that the extra ballots were due
to some error in the execution of the ETP. I expected us to be able
to identify the cause of the error, correct our results, and improve
our procedures in future elections.

On (I believe) Tuesday December 2nd, County Registrar of Voters
Carolyn Crnich certified the election results provided by GEMS. On
Wednesday December 3rd, ETP volunteers working together with Crnich
and the elections office identified the cause of the largest part of
the 216 ballot discrepancy. The GEMS totals (which had just been
certified) failed to include 197 ballots from "Deck 0". The ETP had
also scanned these ballots, so they showed up in our totals. The ETP
therefore identified a serious error in the vote count generated by

Some background information on the counting process helps explain the
nature of "Deck 0". Deck 0 contains some (but not all) of the
absentee ballots from a single precinct (in this case, precinct
1E-45). Absentee ballots are sent out about a month before the
election, and are returned to the election office continuously over
the entire month leading up to the election. At multiple points in
time, whenever the number of returned ballots is sufficiently large,
all the queued incoming ballots that have been received thus far are
sorted by precinct into "decks" of ballots. A deck contains ballots
from only one precinct, but as this sorting happens at multiple points
in time, the ballots from a single precinct are almost always spread
across three or four decks.

Several days before the election (I believe the Saturday before), the
Elections Office began feeding the decks of ballots into the GEMS
central tabulator. The Elections Office is not allowed to print out a
report of the vote totals until the polls close on election night, but
the computerized counting process can begin several days before
election night. The reason for starting in advance is that machine
counting of thousands of ballots, while faster than hand counting, is
nonetheless a time consuming process.

Deck 0 was the first deck of ballots counted by GEMS. It is called 0
(instead of 1), because it is common practice for computer programmers
to start counting at 0.

Upon discovering that deck 0 was not included in the certified
election results, Crnich contacted Premier. I did not observe that
conversation, but from what I understand, Premier claims the deck 0
results are sometimes automatically deleted by the system when a
subsequent deck is intentionally deleted and rescanned. It is
expected that at least several later decks will be deleted and
rescanned as part of normal procedures. The reason for this is that
there is no way to amend a deck's results, so if you need to amend a
decks results, you have to rescan the entire deck. However, when you
intentionally delete a given deck's results (prior to rescanning), you
expect that the other decks results will remain unchanged.

Premier claims (or so I hear tell) that Deck 0 is sometimes "randomly"
deleted because different programmers worked on GEMS at different
points in time, and some programmers start counting at 0, while other
programmers start counting at 1. (Given my own experience as a
programmer, it may be the case that deck 0's results are not really
deleted, but were instead "skipped" or not included in the updated
report. This is not a terribly happy distinction, however, because
the updated report is still wrong, indeed just as wrong as if deck 0's
results had in fact been deleted.)

The ETP had 216 extra ballots. 197 of those were from deck 0. That
means that the ETP now has 19 extra ballots to account for. 19 extra
ballots could be due to a variety of causes, including: GEMS double
feeding ballots (counting 2 ballots as 1), and various operator
errors/anomalies on the part of the ETP volunteers scanning the
ballots. In the June election the ETP counted 2 more ballots than
GEMS. 216 extra ballots are truly alarming. 19 extra ballots (out of
60,000+) are worth investigating and accounting for.

After additional investigation, we learned the following information:

The precinct voted ballots were fed into GEMS on election night, as is
standard procedure. From November 5th until November 23rd, no
additional results were fed into GEMS. During this period, the
elections office was busy auditing the rosters containing voters'
signatures, preparing the remaining vote-by-mail ballots for scanning,
and reviewing and preparing provisional ballots for scanning.

Scanning resumed on November 23rd. Prior to scanning, the elections
office printed out a GEMS report. The report contained the same
totals from the final report on election night, as it should. The
remaining ballots were all fed into GEMS during November 23rd through

When the dropped ballots were discovered, Premier requested copies of
the GEMS database in order to try to determine what went wrong.
Premier claims that deck 0 was deleted sometime during the processing
of decks 131 through 135.

Crnich remembers that during the scanning of deck 132, the GEMS
program was restarted (indeed, the whole computer hosting GEMS was
rebooted) *after* deck 132 had been "opened" but *before* any ballots
had been scanned into deck 132. The reason for the reboot was that
the ballots for deck 132 were of a different type (vote-by-mail,
provisional, mail-ballot-precinct, and precinct voted ballots all
being distinct ballot types). Standard procedure is to reboot the
computer when switching ballot types. (Requiring reboots seems to me
like a kludge/workaround designed to cover up bugs and design flaws in
GEMS.) Our best guess is that restarting GEMS while deck 132 was open
causes GEMS to silently delete deck 0. At no point did GEMS give any
indication that anything was going wrong.

There are other anomalies in the GEMS audit log. There are actually
three "deck 0"'s: deck 0 of vote by mail, deck 0 of provisional
ballots, and deck 0 of mail-ballot-precincts. Deck 0 of 1E-45
(containing the 197 dropped ballots) does not show up in the audit log
and those ballots are not included in the final report. The other 2
deck 0's also do not appear in the audit log, but apparently their
votes are included in the final report. This means the audit log is
not truly a "log" in the classical computer program sense, but is
rather a "re-imagining" of what GEMS would like the audit log to be,
based on whatever information GEMS happens to remember at the end of
the vote counting process.

The version of GEMS the county is using is 1.18.19. This version is
also used by Santa Barbara and San Luis Obisbo counties in California.
It is also used by the entire state of Maryland. Premier has an
internal memo dating from 2004 describing the deck 0 problem. The
workaround to the problem is to manually create and delete all the
deck 0's prior to scanning any ballots. Santa Barbara and San Luis
Obisbo counties are aware of the workaround and have done it in all
recent elections. However, the Humboldt elections office has been
unable to find any official Premier documentation or addenda
describing the workaround. Apparently, the workaround was
communicated from Premier to counties via word of mouth, and the
knowledge of the problem and the workaround may have departed from
Humboldt County due to employee turnover. Additionally, in 4 years
Premier has not fixed the underlying problem.

The California Secretary of State knew neither of the problem, nor of
the workaround. The problem passed certification prior to the 2006
top-to-bottom review, made it through the 2006 top-to-bottom review
undetected and existed in the version of GEMS that was re-certified
after the 2006 top-to-bottom review.

Noteworthy points:

1. The ETP can effectively discover and identify errors in the
primary count, errors that otherwise would not have been detected.

2. The deck-0 problem may be affecting other jurisdiction using GEMS
version 1.18.19, or, for all we know, any version of GEMS whatsoever.

3. Even though this is the second election for the ETP, in many ways
it is a first. In the June election, we used Microsoft Windows
software to control the scanner. Between June and November, with the
help of a third party open source developer, we gained the capability
to control the scanner using the open source Linux operating system.
This allows us to publish all the source code used to control the
scanner, thereby greatly increasing the transparency of the technology
we are using. Additionally, in the future I hope to automate the
process of discovering and identifying discrepancies between ETP
results and primary count results. In a few more elections, the ETP
should be able to identify errors in the primary count *before*
certification of the election results, rather than *after*. So the
ETP will only get better with time.

4. To the best of my knowledge, GEMS does not have the capability to
export machine readable per precinct/per deck results in an easily
analyzable format. If GEMS had the capability to export machine
readable information, it would be a lot easier to audit GEMS (even
without the ETP). AS it stands, it is very difficult to notice that
decks or precincts have been dropped from the final report due to this
serious deficiency in the design of GEMS. (If GEMS does have such a
data export capability, no one has been able to tell me how to use it
- not even Premier's election support specialists.)

Printer Friendly | Permalink | Reply | Top
GuvWurld Donating Member (1000+ posts) Send PM | Profile | Ignore Fri Dec-05-08 12:52 PM
Response to Original message
11. full article in today's Times-Standard

Software glitch yields inaccurate election results
Thadeus Greenson/The Times-Standard
Posted: 12/05/2008 01:24:16 AM PST

The first of its kind Humboldt Election Transparency Project has uncovered a glitch in the county election's software that resulted in almost 200 ballots not being counted and the county certifying inaccurate election results.

The 197 uncounted ballots would not have changed the outcome of any of the election's races, according to Humboldt County Registrar of Voters Carolyn Crnich.

Crnich said the company that provides the county's election software, Premier Elections Solutions (formerly known as Diebold Election Systems, Inc.), seems to have known about the glitch at least since 2004.

Crnich said a discrepancy in vote counts came to her attention after the election was officially certified by the Humboldt County Board of Supervisors, while she and volunteers were preparing ballot images for the transparency project.

Printer Friendly | Permalink | Reply | Top
GuvWurld Donating Member (1000+ posts) Send PM | Profile | Ignore Fri Dec-05-08 05:06 PM
Response to Original message
12. I just submitted this LTTE
sent it to the Times-Standard and North Coast Journal

Dear Editor:

First I'd like to congratulate Kevin Collins, Tom Pinto, Mitch Trachtenberg, Parke Bostrom and all the volunteers of the Election Transparency Project. Their work revealed a discrepancy caused by Humboldt's electronic voting equipment last month.

Over the last few years I've made many different arguments for getting rid of the Diebold (now Premier) equipment used to count votes in Humboldt County. Somehow it wasn't enough that they "count" in secret, can be easily manipulated without detection, and report results impossible in a legitimate election.

Somehow local decision makers weren't deterred from doing business with a company that admitted to illegally installing uncertified software here and elsewhere; that was sued in class action suits filed by company shareholders; and whose then-CEO said he was "committed to helping Ohio deliver its electoral votes" to Bush in 2004.

Now we learn that Humboldt has finally experienced what is euphemistically called a "glitch." In reality it was a bug in Diebold's central tabulation program, GEMS. This caused the results of November's election, already certified as accurate by Registrar of Voters Carolyn Crnich, to be proven inaccurate.

Worse still, Diebold knew about the bug at least four years ago and never fixed it. Other counties were made aware of the problem and told how to work around it. Crnich says she never knew, and I believe her.

This raises many questions, most important among them: who dares defend the continued use of these machines and the county's relationship with Diebold/Premier?

Dave Berman
Eureka, CA
Printer Friendly | Permalink | Reply | Top
emlev Donating Member (1000+ posts) Send PM | Profile | Ignore Sat Dec-06-08 01:38 AM
Response to Reply #12
14. Very good letter
and thanks for helping publicize this really important situation, even if you did have "nothing to do with it." You certainly had some to do with it indirectly, I might add, as you have done such great work to raise awareness in Humboldt of the problems with electronic voting.
Printer Friendly | Permalink | Reply | Top
GuvWurld Donating Member (1000+ posts) Send PM | Profile | Ignore Sat Dec-06-08 07:30 PM
Response to Reply #14
15. aww, shucks
Printer Friendly | Permalink | Reply | Top
tbyg52 Donating Member (1000+ posts) Send PM | Profile | Ignore Sat Dec-06-08 12:38 AM
Response to Original message
13. Aw, too late to get it to 20! Still, thanks, GuvWurld! nt
Printer Friendly | Permalink | Reply | Top
GuvWurld Donating Member (1000+ posts) Send PM | Profile | Ignore Sat Dec-06-08 08:39 PM
Response to Original message
16. Update from Parke Bostrom

UPDATE 20081206

Yesterday (Friday) I spoke with Kim Zetter, a reporter from Wired,
regarding my above commentary. Kim had already reviewed my commentary
with Crnich and Trachtenberg. Based on my conversation with Kim, the
following clarifications are in order:

1) GEMS is not rebooted when switching ballot types. It is the ballot
feeder/central count optical scan machine that must be power-cycled.
(I do not know what the proper name for this piece of equipment is.)
This machine is much less sophisticated than GEMS, so it is tolerable
that it needs to be restarted when switching ballot types. At the
same time, it is even worse that possible mis-use of this separate
piece of equipment could cause GEMS to silently delete deck 0.

2) According to Kim, Crnich now believes that deck 132 may not be the
trigger that caused the deletion of deck 0. (This is a major change
from Crnich's position at our 3pm meeting on Thursday.) Deck 132 was
scanned on November 3rd (prior to the election). Crnich believes that
the deck 0 results were intact at least until scanning was resumed on
November 23rd. Therefore, it is doubtful that deck 132 (scanned weeks
earlier) caused the problem. This shows that GEMS does not provide
information that is useful for identifying when and what went wrong.

Printer Friendly | Permalink | Reply | Top
DU AdBot (1000+ posts) Click to send private message to this author Click to view 
this author's profile Click to add 
this author to your buddy list Click to add 
this author to your Ignore list Wed Sep 20th 2017, 08:47 PM
Response to Original message
Advertisements [?]

Home » Discuss » Topic Forums » Election Reform Donate to DU

Powered by DCForum+ Version 1.1 Copyright 1997-2002
Software has been extensively modified by the DU administrators

Important Notices: By participating on this discussion board, visitors agree to abide by the rules outlined on our Rules page. Messages posted on the Democratic Underground Discussion Forums are the opinions of the individuals who post them, and do not necessarily represent the opinions of Democratic Underground, LLC.

Home  |  Discussion Forums  |  Journals |  Store  |  Donate

About DU  |  Contact Us  |  Privacy Policy

Got a message for Democratic Underground? Click here to send us a message.

© 2001 - 2011 Democratic Underground, LLC