Adding input files to gcamdata: Column type error

[ian-pimenta]

After adding new input .csv files to GCAM data, I ran into the following error while trying to load them in a new chunk:

Error : (converted from warning) One or more parsing issues, see `problems()` for details                                                            

Error in load_csv_files(unfound_chunk$input, unfound_chunk$optional, quiet = TRUE) : 
  Error or warning while reading ASM_2011_31AS101_with_ann.csv

In case anyone runs into this problem: this error indicates a column type error.
When specifying column types in the header, if you assign the "n" data type to a column that contains even a single character data cell, you will get this error.

[ian-pimenta]

This error can be resolved by removing the character data cells from the "n" data columns.

Here is a way to determine which columns are causing the error:

Running the following code would tell you the data type for each column.
If you find a char column where you weren't expecting it, that would likely be the cause of the error.
(For this example, let's imagine your file is an external, raw data file for gcam usa)
readr::read_csv("inst/extdata/gcam-usa/your_file_name", comment = "#") -> random_variable_of_your_choice str(random_variable_of_your_choice)

[zarrarkhan]

Sometimes we get similar issues when you edit a csv or open it in excel and close it again. In this case excel saves the csv but puts in quotation marks in empty cells. These are read in as populated cells and throw errors.
One way to check for this is to open the .csv file in notepad and make sure there are no empty cells which will show up as empty quotation marks (""). Once removed and saved the error is resolved.

[PaulWolfram]

That's very useful. Thanks Ian and Zarrar!

[marideeweber]

This error can also be thrown if there are "#" in the input file, excluding the start of the header. If there is a "#" in a cell, read_csv ignores everything that comes after it, so the character needs to be removed or replaced.

[bpbond]

@ssmithClimate asked about having a clearer error message.

Just so this doesn't get lost in the Slack thread...one solution would be to have load_csv_files() check for this problem, e.g. starting at line 74 in utils.R add something like:

# Check that our comment character doesn't appear in the data
preread <- read_lines(fqfn)
preread <- preread[grep(paste0("^", COMMENT_CHAR), preread, invert = TRUE)] # remove lines with "#" at beginning
badlines <- grep(COMMENT_CHAR, preread) # identify any remaining lines with "#"
if(length(badlines)) stop("Comment character ", COMMENT_CHAR, " appears in data!")