votingworks / arlo Goto Github PK

It would be really nice to highlight who the tool has computed as the winner to help people sanity check their data input.

Currently Arlo just uses the initial 90% likely-to-finish sample size computed in the first round for all subsequent rounds. It should instead compute the 90% likely-to-finish sample size for each successive round based on already audited ballots. get_sample_sizes in sampler.py already does this, setup_next_round in app.py just needs to call it.

User Story: As a user I would like a simplified way to log each audit board in to their respective data entry interface, so that if I have a lot of boards (or limited time) I can do so efficiently while still getting the boards to the correct place.

Background: Once we have a data entry interface for each audit board, we will need a way to authenticate users and direct them to the correct interface for their particular audit board. Persistent login schemes are less than ideal since users are one-time volunteers and co-located with the election official, and we'd like to improve on CORLAs model of making the election official responsible for logging each audit board on to their device (cumbersome, makes less sense if we start seeing boards that want to enter data on their personal phones.) Providing QR codes that the election official can print and distribute still gives us the necessary security while minimizing overhead/upkeep that traditional login schemes require.

Requirements:

• Add an additional button below the "Download retrieval list" button in the UI, labeled "Download Audit Board Credentials for Data Entry"
• When clicked, the button should generate a PDF containing 1 page per audit board, labeled with:
  -- Audit board name
  -- Basic instructions
  -- QR code that contains both the website URL of that board's data entry interface and whatever hash/code (if any) they need to authenticate
  -- The human-friendly version of that URL
  (see screenshot)
  **NOTE: Audit board credentials should be audit-specific, not round-specific, so they'll only need to be shown as we create Round 1.

Open questions:

• The human-friendly URL in the mockup is an idea based on what we've discussed, but I'm not tied to it. If there's a good reason to do something other than "random series of words" for that URL, let's discuss!
• Sizing of the QR code (and really all the QR code details are up to the implementer - the screenshots are meant to give you an idea, not be prescriptive.

We should use the election name/race name in the naming of the retrieval list file. Currently it just gives us ballot-retrieval-jurisdiction-1-1.csv, would be great if it were ballot-retrieval-Washtenaw-1-1.csv.

Entering large (but realistic) elections still cause weird behavior and simulation timeout. E.g. putting in the 2016 presidential primary causes a negative ASN as well as a timeout.

We should tell the user how likely the audit is to complete in one round if they pick the ASN for their sample size.

e.g. "Batch Name" not found

User Story:
As an audit admin, I would like to be able to use a sample size of my choosing for Round 1, so that I can go outside the suggested options if I need to.

More detail:
Note that this issue only implements this in Round 1, not in subsequent rounds. That's by design, as the use case for alternate rounds is less clear. Right now we're giving audit admins a ton of options for the sample size, but there are still good reasons an admin might want to use a number other than what we're giving them:

• For a pilot, if the 90% sample size is small (like 35 ballots, as just happened in MI!) you might want to be able to bump that up just to give people enough practice, even though that's far more ballots than you'd actually need.

• The math is constantly evolving for these audits, so to test new math you might want to run alternate calculations outside the tool while still taking advantage of most of Arlo's features, which you can do fairly easily if we let you control the size of each round.

Requirements:

• Add a final option in the list of sample size options for Round 1 labeled "Enter your own sample size (not recommended)"
• Once that radio button is selected, a text box should appear that enables the user to enter a number (text box rather than number selector, please!)
• If the radio button is deselected, the data entered should be cleared and the textbox should disappear
• If the user does select this option, we should validate that they have a) entered a number and b) entered a number that is less than or equal to the total number of ballots
• If the user selects this option, this is the sample size we should send to the back end as the desired sample size for the round.

Ballots only come in whole numbers, so any fractional sample size should be rounded up to the next integer.

Environments: Tested on current master (and on the branch for PR support multiple contests and candidates)

ERROR in app: Exception on /jurisdiction/jurisdiction-1/manifest [POST]
Traceback (most recent call last):
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
    raise value
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/morgan/Desktop/votingworks/arlo/app.py", line 327, in jurisdiction_manifest
    setup_next_round(election)
  File "/Users/morgan/Desktop/votingworks/arlo/app.py", line 81, in setup_next_round
    sample_sizes = sampler.get_sample_sizes(sample_results(election))
  File "/Users/morgan/Desktop/votingworks/arlo/sampler.py", line 198, in get_sample_sizes
    asns = self.get_asns()
  File "/Users/morgan/Desktop/votingworks/arlo/sampler.py", line 139, in get_asns
    asns[contest] = math.ceil((math.log(1/self.risk_limit) + (z_w / 2)) / ((p_w * z_w) + (p_r * z_l)))
ZeroDivisionError: float division by zero
127.0.0.1 - - [24/Jul/2019 11:07:35] "POST /jurisdiction/jurisdiction-1/manifest HTTP/1.1" 500 -
127.0.0.1 - - [24/Jul/2019 11:13:51] "GET /audit/status HTTP/1.1" 200 -

Passing Scenarios

1. 1 Contest 2 Choices
   a. Type 1 vote for a
   b. Type 2 votes for B
   c. Type 100 votes for total ballots
   d. Submit Audit settings and upload ballot manifest
   Result: Able to submit form

2. 1 Contest 3 Choices
   a. Type 1 vote for a
   b. Type 2 votes for B
   c. Type 3 votes for c
   d. Type 100 votes for total ballots
   e. Submit Audit settings and upload ballot manifest
   Result: Able to submit form

3. 2 Contests 2 Choices
   a. Type 1 vote for a for both contests
   b. Type 2 votes for B for both contests
   d. Type 100 votes for total ballots
   e. Submit Audit settings and upload ballot manifest
   Result: Able to submit form

4. 2 Contest 3 Choices
   a. Type 1 vote for a for both
   b. Type 2 votes for B for both
   c. Type 3 votes for c for both
   d. Type 100 votes for total ballots
   e. Submit Audit settings and upload ballot manifest
   Result: Able to submit form

Failing Scenarios

1. 1 Contest 2 Choices
   a. Type 19 vote for a
   b. Type 20 votes for B
   c. Type 40 votes for total ballots
   d. Submit Audit settings and upload ballot manifest
   Result: Error

2. 1 Contest 3 Choices
   a. Type 19 vote for a
   b. Type 20 votes for B
   c. Type 10 votes for c
   d. Type 40 votes for total ballots
   e. Submit Audit settings and upload ballot manifest
   Result: Error

3. 2 Contests 2 Choices
   a. Type 19 vote for a for both contests
   b. Type 20 votes for B for both contests
   d. Type 40 votes for total ballots
   e. Submit Audit settings and upload ballot manifest
   Result: Error

4. 2 Contest 3 Choices
   a. Type 19 vote for a for both
   b. Type 20 votes for B for both
   c. Type 10 votes for c for both
   d. Type 40 votes for total ballots
   e. Submit Audit settings and upload ballot manifest
   Result: Error

Ballot Manifest: Ballot Manifest May 2019 Election - WYANDOTTE.csv

Currently retrieval list is just sorted seemingly at random with respect to batch name, ballot number, and audit board. It would be great if it was sorted in order of audit board (numerically, not lexicographically), batch, and then ballot number.

• landing page with a big button that says "start new audit"
• POST /new-audit --> returns UUID
• redirect to /election/{uuid} and that's the new front page.

spinner appears in two places when uploading manifest.

Steps to Reproduce

1. Load new form
2. Create two contests. Each contest has the same values that will complete in one round (Example: 792, 1325, 2123)
3. Fill in Audit Setting and submit form
4. Select Audit Boards and upload Ballot Manifest
5. Enter the same values with intent to complete the audit (Example: 100, 167)
   Expected Result: Both Contests's audits show as complete with the button for downloading the audit report
   Actual Result: One contest shows as incomplete. The other one shows as complete. There is no second round returned from the API.

GET/POST Edit/Create an audit board member endpoint
GET Audit board status endpoint
POST/GET Ballot endpoint
POST Audit board sign off endpoint
POST Get correlated url from human readable token

Also need to update the uuid generation for elections to be human-readable, and update the POST to /election/{electionId}/audit/jurisdictions to take only an array of strings (for the audit board names) and return the full object with human-readable uuids, the names, and empty member arrays.

Or we need to update the jurisdictions POST endpoint to either accept human readable tokens and/or uuids, or have either or both of those generated on the back end and returned in the jurisdictions array.

When handling multiple audits we get a 500 error when the backend tries to create a new jurisdiction.

 ERROR in app: Exception on /election/1e7a89e2-9f8c-4639-b378-0e3817d91156/audit/jurisdictions [POST]
Traceback (most recent call last):
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: jurisdiction.name

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
    raise value
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/morgan/Desktop/votingworks/arlo/app.py", line 350, in jurisdictions_set
    db.session.commit()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1027, in commit
    self.transaction.commit()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 494, in commit
    self._prepare_impl()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 473, in _prepare_impl
    self.session.flush()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2459, in flush
    self._flush(objects)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2597, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 154, in reraise
    raise value
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2557, in _flush
    flush_context.execute()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    insert,
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1084, in _emit_insert_statements
    c = cached_connections[connection].execute(statement, multiparams)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 399, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value.with_traceback(tb)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: jurisdiction.name
[SQL: INSERT INTO jurisdiction (id, election_id, name, manifest, manifest_filename, manifest_uploaded_at, manifest_num_ballots, manifest_num_batches, manifest_errors, manifest_fields) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: ('c4d0d917-ee6d-4a75-8d3d-74748d21f499', '1e7a89e2-9f8c-4639-b378-0e3817d91156', 'Jurisdiction 1', None, None, None, None, None, None, None)]
(Background on this error at: http://sqlalche.me/e/gkpj)
127.0.0.1 - - [30/Aug/2019 13:49:40] "POST /election/1e7a89e2-9f8c-4639-b378-0e3817d91156/audit/jurisdictions HTTP/1.1" 500

At https://github.com/votingworks/arlo/blob/master/arlo-client/src/components/AuditForms/SelectBallotsToAudit.tsx#L99 we are hard coding the jurisdiction name. So either we need to remove the unique constraint or create unique names for each jurisdiction. If we allow users to set the jurisdiction name we'll need a way to validate that the name they submit is unique before posting the form. What's our best solution here?

User story:
As an engineer I'd like to replace on-demand simulation of appropriate sample sizes with lookups of precomputed values wherever possible, so that I can streamline the computation needed for any particular audit.

Basic task: Simulate & store a variety of scenarios (maybe for every possible diluted margin 1%-99%, what's the % of ballots you'd need to look at to have x% chance of completing in 1 round, for values of x from 50-95%?) and store that data, so that you can look up the % of ballots needed and multiply it by the # of total ballots in a given audit scenario, instead of simulating on-demand.

Current sample size calculations use p_w, etc., when they should be using s_w, etc., per BRAVO. See #143 for similar.

Description:
When you download the audit report, the batch identifiers in the list of ballots are ID strings rather than the human-readable batch names: "(Batch 25064fad-1e05-40eb-8be0-84fbf278eb40, #92)." See attached screenshot.

Issue is due to psycopg2 in the Pipfile (see psycopg/psycopg2#674).

Proposed fix: switch from psycopg2 to psycopg2-binary in Pipfile.

To reproduce: enter any contest data, any 20 digit seed (12345123451234512345 works well) and click "estimate sample size" ... when the next section of the page loads, the seed will have changed to be "12345123451234513000"

Refactor to blueprintjs

Rather than force users to pick one of our radio buttons, we should allow text entry of arbitrary sample sizes. E.g. if 323 ballots is the 80% sample, and 375 ballots is the 90%, users should be able to pick 350 ballots (or anything else).

Currently on master after POSTing to /election/:id/audit/reset further calls to endpoints using that id result in 500 errors.

ERROR in app: Exception on /election/9710b544-8fb1-474f-ac50-97d8b8505be5/audit/status [GET]
Traceback (most recent call last):
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
    raise value
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/morgan/Desktop/votingworks/arlo/app.py", line 207, in audit_status
    election = get_election(election_id)
  File "/Users/morgan/Desktop/votingworks/arlo/app.py", line 44, in get_election
    return Election.query.filter_by(id = (election_id or '1')).one()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3289, in one
    raise orm_exc.NoResultFound("No row was found for one()")
sqlalchemy.orm.exc.NoResultFound: No row was found for one()

When providing a contest with choice A: 10 votes, choice B: 20 votes, and total ballots as 30 votes, we receive the following sample size options:

Having multiple radio select options with the same value confuses the browser about which radio button to show as selected when any of them is clicked on (though it doesn't affect the functionality of the form submission). The workaround is very hacky and not recommended practice. However, it seems to be not be an ideal user experience to have multiple options with the same value and different probabilities anyway.

I recommend reducing down options that have the same size value (using the highest prob value of them) to prevent this being confusing for anyone, if this should arise in the real world. This could also be an extreme edge case that we don't need to worry about. Opinions?

@PlauditsForAudits found our first math bug - woohoo! Here's the fix (which we need to implement) and the explanation:

"In sampler.py (as of this morning?), the code at lines 100-102 reads

            'p_w': win_votes/v_wl,   
            'p_r': rup_votes/v_wl,
            's_w': win_votes/ballots,

That looks like a misreading of the BRAVO paper, which is understandable because this part of the BRAVO paper is almost unreadable. p_w and p_r should be fractions of all ballots, and s_w should be a fraction of the two-way. So it's

            'p_w': win_votes/ballots,
            'p_r': rup_votes/ballots,
            's_w': win_votes/v_wl,

The easiest way in the BRAVO paper to see that this is correct is to stare at footnote 16 on page 7. Note that p_w and p_l are smaller than s_w and s_l, respectively: again, p_w and p_l are fractions of all ballots, while s_w and s_l are fractions among the top two candidates."

Arlo should warn/disallow users from putting in fewer total ballots than the sum of the voters for the candidates (e.g. Cand A gets 600 votes, Cand B gets 400 votes, but Total Ballots is set to 100 by mistake). Doing this now causes the tool to freeze up.

@benadida It appears as though the API accepts a unique id for the candidates in the choices array on contests. But the /audit/status endpoint is only returning name and numVotes values. This means that we don't have the unique IDs to pass back to the 'POST /jurisdiction/<jurisdiction_id>/<round_id>/resultsendpoint as part of the results object. Before, with the hardcodedcandidate-n` IDs, it didn't matter, but with generated unique IDs we need those returned from the API.

POST /jurisdiction/<jurisdiction_id>/<round_id>/results -- the results for one round

{
	"contests": [
		{
			id: "contest-1",
   			results: {
				"candidate-1": 55, // uses candidate id
				"candidate-2": 35
			}
		}
	]
}

POST /audit/basic -- the overall audit configuration
{
	name: "Primary 2019",
	riskLimit: 10,
	randomSeed: "sdfkjsdflskjfd",

	contests: [
	    {
			id: "contest-1",
			name: "Contest 1",
			
			choices: [
				{
					id: "candidate-1", // id is passed to api
					name: "Candidate 1",
					numVotes: 42
				}
			],
			
			totalBallotsCast: 4200
		}
	]
}

GET /audit/status -- get the whole data structure for the whole audit
{
	name: "Primary 2019",
	riskLimit: 10,
	randomSeed: "sdfkjsdflskjfd",

	contests: [
	    {
			id: "contest-1",
			name: "Contest 1",
			
			choices: [ 
				{ // missing id
					name: "Candidate 1",
					numVotes: 42
				}
			],
			
			totalBallotsCast: 4200
		}
	],
...

Originally posted by @MorganLove in #45 (comment)

@mcchilders On mobile the table of ballots extends past the viewport, and so for usability, it would make sense to have the most important two columns be on the left side for ease of access with less scrolling. Which two columns should be there? I'm assuming the status one with the re-audit buttons would be one of the two, right?

User Story: As an audit admin I'd like Arlo to pregenerate my ballot label and placeholders sheets with my ballot manifest, so that I don't have to waste time writing them out individually.

Background:
As part of the chain-of-custody piece of the audit, every ballot that is removed from a stack of ballots is labeled with a removable label that lists the batch name and ballot number for that particular ballot. Similarly, a piece of brightly colored paper bearing the same info is inserted in the stack where the ballot was removed, to serve as a placeholder. This ensures that we always know which selected ballot is which, and also exactly where it came from. Since we are generating the ballot retrieval list, we already have all the information that needs to be on these labels/placeholders, so we can save auditors some time if we generate PDFs that they can just print along with the retrieval list.

Requirements:

• Adjust the "download ballot retrieval list" button in the UI to say "Download the ballot retrieval list, labels, and placeholder pages for Round "
• When the user clicks that button, three files should be downloaded: the retrieval list we have now, a PDF of labels, and a PDF of placeholder pages. Each of these should be separate files in case one of them isn't needed, and also because the user will need to pause in between printing each file to switch between regular paper, labels, and brightly colored paper for the placeholders.
• Label PDF should fit on the Avery 16460 template (https://www.avery.com/templates/16460) and look like the attached example, 1 label per sampled ballot (if a ballot is sampled twice, it should only get one label)
• Labels should be ordered as you would read across the page, left to right, top to bottom
• Placeholder pages should look like the attached example, 1 placeholder page per sampled ballot (if a ballot is sampled twice, it should only get one placeholder page)

Math is fairly straightforward, will require changes on both backend and frontend.

User story:
As an audit admin I want some flexibility in the number of ballots I need to audit in a particular audit round, so that I can optimize my audit to finish in one round (or not) as I wish.

Basic task:
Replace the current sample size UI, which takes a value from the API and displays it onscreen with no interaction, with a set of radio buttons representing several sample size options. For each potential sample size other than the ASN and choose-your-own, the API will furnish both the number of ballots (e.g. 141) and the % likelihood of completing the audit in one round using that sample size (e.g. 80%). When the user selects one of these options, that chosen sample size needs to post back when the user clicks "Select ballots for audit."

• Option 1: ASN sample size #, without an expected % of completion (we can calc that outside the tool if we absolutely need it) (E.g. “125 samples (BRAVO Average Sample Number)“)
• Option 2: 70% expected completion rate # (e.g. “129 samples (70% chance of reaching risk limit in one round)”
• Option 3: 80% expected completion rate # (e.g. “175 samples (80% chance of reaching risk limit in one round)”
• Option 4: 90% expected completion rate # (e.g. “295 samples (90% chance of reaching risk limit in one round)”
• Option 5: choose your own sample size (text box where the user can input whatever number of samples they wish) (E.g. “Other sample size:[ ]“)

Wireframe:

(full wireframes available at: https://balsamiq.cloud/sacw0cz/pn32zzn)

Open questions:

• Should we let the user select from multiple sample size options on each audit round, or just do this on Round 1 and assume that the % chance of completion selected for Round 1 is an acceptable setting for all rounds?

RLA background:
The sample size assigned for any given audit round is an estimate of the number of ballots the auditors will need to review in order to reach the desired risk limit for the audit. Because the samples/ballots themselves are chosen at random, however, we can't know the exact number of samples that are needed. To get around this uncertainty, the original BRAVO implementation uses the Average Sample Number (ASN) across a variety of scenarios as the estimated sample size for the round, and then increases the sample size in subsequent audit rounds if necessary. In some scenarios, however, particularly in audits where the ASN is small or the audit process is tightly timeboxed, it may be preferable to pad the ASN a bit and draw a larger initial sample of ballots if that larger initial sample will lower the likelihood of having to complete additional audit rounds. This issue adds that option to the UI, by including the ASN as the minimum sample size option and adding alternate options with a higher likelihood of completing in a single round.

The ballot retrieval list generated for the last VA pilot audit contained two ballots at position zero (e.g. Batch 101, ballot 0). Seems like we may be indexing starting at zero instead of 1? Need to bump that for ballot indexes.

• /election/{uuid} URLs
• leave the old URLs working for now
• ensure each election lives in its own universe.

specifically node_modules

Deadline: July 30th (VA audit pilots begin July 31)

User story: As an audit admin I want to be able to enter data (names/vote totals) for 3+ candidates in a particular contest, so that I can use ARLO for audits of contests with more than 2 candidates.

Basic task: Adjust the UI to allow users to enter data for as many candidates as they need for a given contest. Note that the back end math to handle more than 2 candidates already works as long as the contest can only have 1 winner, and the data model/API are ready for multiple candidates, as well.

Implementation notes: Could just add a button to the UI to "add another choice" that would trigger the addition of another set of text fields when clicked (see mockup). If filled, that's another candidate, if left blank, ignore. Increment the # in the labels logically (e.g. "Name of Candidate/Choice 3, Name of Candidate/Choice 4" etc.) Open to other ideas, too!

Mockup:

User story: As an audit admin I want to be able to split my ballot retrieval list relatively evenly between a specific number of audit boards, but without splitting samples in the same batch of ballots between multiple boards if possible, so that my audit boards can retrieve their ballots fairly independently.

Basic task: Our ballot retrieval list splitting among audit boards is simple/easy right now - just divide the list as evenly as possible between however many boards we have. That works for now, in that it minimizes the edits needed to the ballot retrieval list, but we'd like to replace that with something more sophisticated to do this well without any human intervention ... which isn't simple. Probably worth taking a look at the CORLA logic DW implemented, since it does this robustly and might be a good pattern/inspiration?

ASN computations don't account for ties, so we get weirdness like this:

The software should just kick out of doing simulations altogether and tell the user to do a full hand recount. It's highly unlikely this will ever come up, but chaos monkeys are everywhere.

User story: As an audit admin I want to be able to enter data (names/vote totals) for multiple contests, so that I can use ARLO to audit overlapping contests in my jurisdiction.

Basic task: Adjust the UI to allow users to enter data for as many contests as they need for a given audit.

Implementation notes: Could add a button to the UI to "add another targeted contest" that would trigger the addition of another set of contest data text fields when clicked (see mockup). If filled in, that's another contest, if left blank, ignore. Increment the # in the labels logically (e.g. "Name of Contest 2, Name of Contest 3" etc.) Open to other ideas, too! Contest data fields include: Contest name, candidate/choice names/vote totals, total ballots cast.

Mockup:

After submitting the first form with a randomSeed value of 11111111111111111111 or 12345123451234512345 (or any 20 digit number) there is a server 500 error:

"GET /audit/status HTTP/1.1" 200 -
[2019-08-09 15:02:42,064] ERROR in app: Exception on /audit/basic [POST]
Traceback (most recent call last):
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
    raise value
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/morgan/Desktop/votingworks/arlo/app.py", line 260, in audit_basic_update
    db.session.query(TargetedContest).filter_by(election_id = election.id).delete()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3690, in delete
    delete_op.exec_()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1689, in exec_
    self._do_pre()
  File "<string>", line 1, in <lambda>
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1735, in _do_pre
    session._autoflush()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1577, in _autoflush
    self.flush()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2459, in flush
    self._flush(objects)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2597, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 154, in reraise
    raise value
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2557, in _flush
    flush_context.execute()
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj
    update,
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 996, in _emit_update_statements
    statement, multiparams
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1468, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 154, in reraise
    raise value
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/Users/morgan/.local/share/virtualenvs/arlo-cgZpGxiy/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
    cursor.execute(statement, parameters)
OverflowError: Python int too large to convert to SQLite INTEGER
127.0.0.1 - - [09/Aug/2019 15:02:42] "POST /audit/basic HTTP/1.1" 500 -

Reducing the digits to 19 makes this error not occur.