🐞 When serializing decrypted spoiled ballots, placeholder selections should be removed

[JohnLCaron]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

in election_record/spoiled_ballots/spoiled_ballot_<ballot_id>.json, decrypted spoiled ballots still contain placeholder selections.

1. this is likely confusing for people examining spoiled ballots to verify they are correctly decrypted.

2. this causes validation 12.B ("An election verifier should also confirm that for each decrypted spoiled ballot, the selections
   listed in text match the corresponding text in the ballot coding file.") to fail, eg:

***Ballot Selection id (justice-supreme-court-5-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (justice-supreme-court-4-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (referendum-pineapple-2-placeholder) not contained in contest (referendum-pineapple).
 ***Ballot Selection id (justice-supreme-court-5-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (justice-supreme-court-4-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (justice-supreme-court-5-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (justice-supreme-court-4-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (referendum-pineapple-2-placeholder) not contained in contest (referendum-pineapple).
 ***Ballot Selection id (justice-supreme-court-5-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (justice-supreme-court-4-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (referendum-pineapple-2-placeholder) not contained in contest (referendum-pineapple).
 ***Ballot Selection id (justice-supreme-court-5-placeholder) not contained in contest (justice-supreme-court).
 ***Ballot Selection id (justice-supreme-court-4-placeholder) not contained in contest (justice-supreme-court).
 ***12.B Spoiled PlaintextTally Names Validation failed.

Expected Behavior

placeholder selections should not appear in decrypted spoiled ballots

Steps To Reproduce

run TestEndToEndElection
examine election_record/spoiled_ballots/spoiled_ballot_<ballot_id>.json

Environment

OS: Ubuntu 20.04

Anything else?

In decrypt_with_shares.py, skip any placeholder selection:

def decrypt_contest_with_decryption_shares(
    contest: CiphertextContest,
    shares: Dict[GuardianId, DecryptionShare],
    crypto_extended_base_hash: ElementModQ,
) -> Optional[PlaintextTallyContest]:
    plaintext_selections: Dict[SelectionId, PlaintextTallySelection] = {}

    for selection in contest.selections
        if selection.is_placeholder is True:
          continue
...

[rc-ms]

Thank you again good sir. I will see if we can slot this work in with the same work on handling overvotes, which will obviate the need to display placeholder values. i assume dr @benaloh is supportive? (if he doesn't reply here I'll check separately)

[JohnLCaron]

I noticed that my python fix doesnt work because CiphertextBallotSelection.is_placeholder doesnt exist. Maybe add that field, or look at the selection name which ends with "-placeholder".

[keithrfung]

I think this issue should be delved into further. I think this checks out for verifiers but I think this is a question of how the collection is at the end. We really shouldn't be adding the placeholders at all at the decryption stage. Perhaps it's better a question of how the decryption works as opposed to serializing.

[benaloh]

I don't think that we need the placeholders to be decrypted, but what's most important is the presentation. I agree completely that showing placeholders to voters creates confusion.

Ideally, we want there to be a clear voter-readable indication of the contents of each spoiled ballot. It should then be possible for a verifier to confirm that this voter-readable information is consistent with an accurate decryption of the ballot. I don't think that the placeholder decryptions are necessary for this (although there's no harm in providing them for verification). The things that need to be verified are essentially that the encrypted ballot matches the confirmation code and the decrypted selections match what is shown to the voter.