Is it possible to parse big csv file line by line?

[alfonz19]

Maybe I'm overlooking this option somehow, if so please be so kind and point to related documentation/code.

Scenario: In deserializing to some java dto, csv file with first line being header(potentially invalid not relating to dto), following lines are records, potentially invalid, either for deserialization to dto or bean validation. I'd like to process the whole file and report all lines which contains error.

Currently I'm solving it by reading first line (header) which I store in variable and while reading remaining lines I'm creating 2LOC csv documents, which I parse separately. It's nasty, but I don't see another option. Please advise

[cowtowncoder]

Sounds like a discussion, not issue, converting.

[cowtowncoder]

So, something like:

ObjectReader reader = new CsvMapper()
                .readerFor(ImportCsvDataWithBooleanPrimitive.class)
                .with(CsvSchema.emptySchema().withHeader().withArrayElementSeparator("|"))
                .with(CsvParser.Feature.EMPTY_STRING_AS_NULL)
                .with(CsvParser.Feature.FAIL_ON_MISSING_COLUMNS);
MappingIterator<ImportCsvDataWithBooleanPrimitive> it;
try {
   it = reader.readValues();
} catch (IOException e) {
    throw new RuntimeException("Bad header row, root cause: "+e.getMessage(), e);
} 

while (it.hasNextValue()) {
  try {
     ImportCsvDataWithBooleanPrimitive value = it.nextValue();
     // process value
  } catch (IOException e) {
     // or handle problem
     ++errorCount;
  } 
}

Hope this helps.

[alfonz19]

TLDR: Yes, it helped me a lot, fixed. Thank you very much.

Uff, now I get it. Apologies, I have no idea how could I overlook the possibility of passing whole csv data into com.fasterxml.jackson.databind.ObjectReader#readValues(java.lang.String). I overcomplicated it for myself with BufferedReader and then was surprised that it does not work. Thank you very much for your help.

full adapted test with assertions with same input as we had it before looks like following:

@Test
    void testReadingWithCollectingErrorPerLine() {
        String input = """
                "A","B"
                "valid1", true
                "invalid1", true, "this", "is", "too", "much!"
                "invalid2", "false"
                "invalid3"
                """;
        List<String> errors = new ArrayList<>();
        List<ImportCsvDataWithBooleanPrimitive> values = new ArrayList<>();

        ObjectReader reader = new CsvMapper()
                .readerFor(ImportCsvDataWithBooleanPrimitive.class)
                .with(CsvSchema.emptySchema().withHeader().withArrayElementSeparator("|"))
                .with(CsvParser.Feature.EMPTY_STRING_AS_NULL)
                .with(CsvParser.Feature.FAIL_ON_MISSING_COLUMNS);
        try(MappingIterator<ImportCsvDataWithBooleanPrimitive> it = reader.readValues(input)) {
            int lineNumber = 1;
            while (it.hasNextValue()) {
                lineNumber++;
                try {
                    values.add(it.nextValue());
                } catch (JacksonException e) {
                    errors.add(String.format("#%d:%d %s", lineNumber, e.getLocation().getColumnNr(), e.getOriginalMessage()));
                } catch (Exception e) {
                    errors.add(String.format("#%d %s", lineNumber, e.getMessage()));
                }
            }
        } catch (IOException e) {
            throw new RuntimeException("Bad header row, root cause: "+e.getMessage(), e);
        }


        assertThat(values.size(), is(1));
        assertThat(errors.size(), is(3));

        assertThat(values.getFirst(), is(new ImportCsvDataWithBooleanPrimitive("valid1", true)));

        String expected = """
                #3:18 Too many entries: expected at most 2 (value #2 (6 chars) ""this"")
                #4:12 Cannot deserialize value of type `boolean` from String ""false"": only "true"/"True"/"TRUE" or "false"/"False"/"FALSE" recognized
                #5:1 Not enough column values: expected 2, found 1""";
        assertThat(String.join("\n", errors), is(expected));
    }

if you would like to add it to your how-to page, with test cases covering bad header etc. I will expand it and finish it to cover all cases. Would you like me to do that 'for you' and others? I can put there better input demonstrating also array splitting. Just tell me; you helped me, I can create demonstration of this which would look like you want it. And please advise where to put it if you want me to create MR, otherwise I can put it here.

[cowtowncoder]

@alfonz19 Great! Glad it was just bit of misunderstanding, b/w our assumptions.

On improvement: I appreciate the offer & think this would be great addition to csv/README.md (as seen at https://github.com/FasterXML/jackson-dataformats-text/tree/2.19/csv ), along with any fixes? Change would be done by a simple pull request.

[alfonz19]

(please be patient with me; I have some deadline quite close which would cost me some money. I'd either have to rush it, which I don't want or I have to try to do it a little later. I will definitely try to do that, probably in a week. Sorry for delay).

[cowtowncoder]

@alfonz19 No problem at all. Take your time.