Checklist: challenges.csv data file format

[MattTriano]

Requirements for the challenges.csv data file format

Per page 27 of the policy notice and Notices of Changes to the process.

Column Names, Order, and Data Types (dtypes):

1. challenge (dtype: string)
2. challenge_type (dtype: single-character string)
3. challenger (dtype: string)
4. challenge_date (dtype: string)
5. rebuttal_date (dtype: string)
6. resolution_date (dtype: string)
7. disposition (dtype: single-character string)
8. provider_id (dtype: integer here, string in challenger)
9. technology (dtype: integer)
   • dtype test: test_challenges_column_dtypes__technology
10. location_id (dtype: integer)
    • dtype test: test_challenges_column_dtypes__location_id
11. unit (dtype: string)
12. reason_code (dtype: single-character string)
13. evidence_file_id (dtype: string)
14. response_file_id (dtype: string)
15. resolution (dtype: string)
16. advertised_download_speed (dtype: integer)
    • dtype test: test_challenges_column_dtypes__advertised_download_speed
17. download_speed (dtype: integer)
    • dtype test: test_challenges_column_dtypes__download_speed
18. advertised_upload_speed (dtype: integer)
    • dtype test: test_challenges_column_dtypes__advertised_upload_speed
19. upload_speed (dtype: integer)
    • dtype test: test_challenges_column_dtypes__upload_speed
20. latency (dtype: float)
    • dtype test: test_challenges_column_dtypes__latency

• Correct column names
  • test: test_challenges__column_names
• Correct column order
  • test: test_challenges__column_order

Column Content Requirements

• challenge: String with a length between (inclusive) 1 and 50 characters.

  • test: test_challenges_col_content__challenge

• challenge_type: Enum in [A, S, L, D, T, B, P, E, N, V, F, M, X, Y, Z]

  • test: test_challenges_col_content__challenge_type

• challenge_date:

  • Must be in ISO 8601 extended date format.
    • test: test_challenges_col_content__challenge_date
  • Must precede other dates (if they're not null)
    • test: test_challenges_row_rule__rebuttal_date_not_before_challenge_date
    • test: test_challenges_row_rule__resolution_date_not_before_challenge_date

• rebuttal_date:

  • Must be in ISO 8601 extended date format.
    • test: test_challenges_col_content__rebuttal_date
  • Must be later than challenge date and before or on resolution date.
    • test: test_challenges_row_rule__rebuttal_date_not_before_challenge_date
    • test: test_challenges_row_rule__resolution_date_not_before_rebuttal_date

• resolution_date:

  • Must be in ISO 8601 extended date format.
    • test: test_challenges_col_content__resolution_date
  • Must be later than challenge date and on or after rebuttal_date.
    • test: test_challenges_row_rule__resolution_date_not_before_challenge_date
    • test: test_challenges_row_rule__resolution_date_not_before_rebuttal_date

• disposition: Enum in [I, N, A, S, R, M]

  • test: test_DispositionsOfChallenge_validator

• provider_id: A 6-digit integer (ideally identifying an ISP in the FCC's providers data, but this only confirms that 100000 <= provider_id <= 999999).

  • test: test_FccProviderIdValidator
  • test: test_challenges_col_content__provider_id

• technology: Enum in [10, 40, 50, 60, 61, 70, 71, 72, 0]

  • test: test_Technology
  • test: test_challenges_col_content__technology

• location_id: a 10-digit integer $\ge 10^9$ and $\lt 10^{10}$

  • test: test_BSLLocationIdValidator
  • test: test_challenges_col_content__location_id

• reason_code: Enum in [1, 2, 3, 4, 5, 6, 8, 9]

  • test: test_challenges_col_content__reason_code

• evidence_file_id: "File names for evidence and responses must only contain US ASCII letters, the digits 0-9, hyphens (-) and underscore (_) characters. File names are not case sensitive. The file name extension must be pdf (PDF files); other file types may be permitted in the future."

  • test: test_FileNameNullableValidator
  • test: test_challenges_col_content__evidence_file_id

• response_file_id: "File names for evidence and responses must only contain US ASCII letters, the digits 0-9, hyphens (-) and underscore (_) characters. File names are not case sensitive. The file name extension must be pdf (PDF files); other file types may be permitted in the future."

  • test: test_FileNameNullableValidator
  • test: test_challenges_col_content__response_file_id

• resolution: Free text, "required for E, I, S, and R" (presumably challenge_types, but there isn't a type I and R is a CAI type)

• advertised_download_speed: Non-negative integer, Mbps.

  • test: test_challenges_col_content__advertised_download_speed

• download_speed: Non-negative number, Mbps.

  • test: test_challenges_col_content__download_speed

• advertised_upload_speed: Non-negative integer, Mbps.

  • test: test_challenges_col_content__advertised_upload_speed

• upload_speed: Non-negative number, Mbps.

  • test: test_challenges_col_content__upload_speed

• latency: Non-negative number, ms.

  • test: test_challenges_col_content__latency

Columns Required to be Not Null

• challenge: test: test_challenges_column_not_null_validations
• challenge_type: test test_ChallengeType
• challenge_date: test test_challenges_col_content__challenge_date
• resolution_date: test test_challenges_col_content__resolution_date
• disposition: test: test_challenges_col_content__disposition
• location_id: test: test_challenges_col_content__location_id

Columns that can be Conditionally Null

• challenger:
  • Can be null if challenge_type is in [V, F, M, X, Y, Z]
    • test test_ChallengeType
    • test test_challenges_row_rule__challenger_given_challenge_type
  • noted but not checked: null if "[challenge_type] E or N challenge was part of pre-challenge process deduplication."
• rebuttal_date: Cannot be null if there was a rebuttal (i.e., response_file_id should also be not-null)
  • test: test_challenges_row_rule__rebuttal_date_and_file
• provider_id: Cannot be null for any challenge_type except for "P" (planned service)
  • test: test_challenges_row_rule__provider_id_given_challenge_type
• technology: Cannot be null for challenge_type except for "N" (not part of an enforceable commitment)
  • test: test_challenges_row_rule__technology_given_challenge_type
• reason_code: Cannot be null if challenge_type is "A" (availability)
  • test: test_challenges_row_rule__reason_code_given_challenge_type
• evidence_file_id: Cannot be null unless challenge_types in [E, V] (enforceable commitment or DSL)
  • test: test_challenges_row_rule__evidence_file_id_given_challenge_type
• response_file_id: Must be null if rebuttal_date is null, and must be not-null if rebuttal_date is not-null
  • test: test_challenges_row_rule__rebuttal_date_and_file
• resolution: Cannot be null for challenge_types in [E, I, S, R]
  • test: test_challenges_row_rule__resolution_given_challenge_type__disposition
  • Noted but not checked: "For E, include name of broadband funding program or other source of the commitment, as applicable." (per pg 30 of the policy notice)
• advertised_download_speed: Cannot be null except for challenge_type "N" (not part of an enforceable commitment)
  • test: test_challenges_row_rule__advertised_download_speed_given_challenge_type
• download_speed: Cannot be null for challenge_types in [S, M]
  • test: test_challenges_row_rule__download_speed_given_challenge_type
• advertised_upload_speed: Cannot be null except for challenge_type "N" (not part of an enforceable commitment)
  • test: test_challenges_row_rule__advertised_upload_speed_given_challenge_type
• upload_speed: Cannot be null for challenge_types in [S, M]
  • test: test_challenges_row_rule__upload_speed_given_challenge_type
• latency: Cannot be null for challenge_types in [L, M]
  • test: test_challenges_row_rule__latency_given_challenge_type

Multi-File Column Rules

• challenger: Uniquely identifies a challenger in the table challenger.csv (not case sensitive)
  • test: ToDo: implement test

Intentionally noted but not checked by this project

• provider_id: Must be a valid provider in the state.
• location_id: Must be in the Fabric for that state.