Unable to Correctly Extract Chinese Characters from PDF Documents

[xujryan]

Description

When using Marker to extract Chinese characters from some PDF documents, some characters are not extracted at all, while others are extracted as garbled text. Below are three example files, including the original PDF files and the parsed Markdown files.

Example

• PDF Files: test1.pdf, test2.pdf, test3.pdf

• Parsed Markdown Files: test1.md, test2.md, test3.md

Do you have any suggestions or recommendations for handling or parsing these types of Chinese documents?

[lqhl]

I think #153 is related.

[VikParuchuri]

Do these PDFs have garbled text in them? Does setting OCR_ALL_PAGES to force OCR change anything?

[VikParuchuri]

I get better results by settings OCR_ALL_PAGES=true, which forces OCR. I think this PDF has bad/garbled text in it (at least the first one, which is the one I tested).

[elgarteo]

I ran into the same issue while extracting Chinese text from a PDF document edited by macOS’ built-in PDF reader Preview, which turns out to be the culprit. Garbled text only appears when extracting from the edited PDF, not the original file.

I did some tracing and the garbled text are produced by the upstream package Pdfium, specifically the FPDFText_GetUnicode method. Someone actually raised this issue as early as 2017 but nothing has been done.

The solution for now is to avoid passing in PDF files edited with Preview (and potentially other PDF editors).