Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Marker Improvements and Bugfixes #403

Merged
merged 41 commits into from
Dec 3, 2024
Merged

Marker Improvements and Bugfixes #403

merged 41 commits into from
Dec 3, 2024

Conversation

iammosespaulr
Copy link
Collaborator

@iammosespaulr iammosespaulr commented Nov 30, 2024 •
edited
Loading

  • Add CLI option for Paginating the output --paginate_output
  • Add CLI option to disable image extraction --disable_image_extraction as well as a config option extract_image which defaults to True
  • Add ListProcessor for merging lists across pages and columns
  • Clean up dangling newlines in the outputs
  • Configurable OUTPUT_ENCODING
  • Fix bug with PageFooter being moved to the top of the page
  • Fix Section Hierarchy bug
  • Fix text inputs from all sources, the PdfProvider, surya OCR and tabled
  • Standardize DebugProcessor outputs
  • Add BlockquoteProcessor
  • Add support for nested list handling in ListProcessor

iammosespaulr and others added 24 commits November 27, 2024 18:14
@iammosespaulr
ignore unassociated newlines from the pdf provider [skip ci]
f348349
@iammosespaulr
ftfy ocr builder [skip ci]
5bfd4aa
@iammosespaulr
ftfy table cells
b92793c
@iammosespaulr
debug improvements and bugfix
c82f11a
@iammosespaulr
switch blockTypes to strEnum
89ebc6d
@VikParuchuri
Add texify batch size
93549e5
@iammosespaulr
fixed test
921d866
@iammosespaulr
fix section hierarchy leak
577bb73
@iammosespaulr
don't move page footers to the top
c962038
@iammosespaulr
account for line merging now that page footers are at the bottom of t…
9e4811e 
…he page
@iammosespaulr
add extract_images config opt
a1065cb
@iammosespaulr
integrate extract_images into json output as well
9602cb4
@VikParuchuri
Force utf8
4f75544
@VikParuchuri
Finalize merge
00d24b8
@iammosespaulr
refactor next block logic for reuse with list item processor
a3d5ec2
@iammosespaulr
bugfix [skip ci]
3a4dec1
@iammosespaulr
add list joining processor
eb1550d
@iammosespaulr
adjust heuristics
6a786be
@iammosespaulr
make column break threshold more generous
d9eb4dd
@iammosespaulr
restricting column break threshold doesn't work quite as well for lis…
9d0f0eb 
…t groups, loosen it
@VikParuchuri
Merge pull request #402 from VikParuchuri/dev-mose/list-joining-misc
1047d9c 
Add `ListGroup` joining processor and refactor `Text` joining processor
@VikParuchuri
Version bump
c086efe
@VikParuchuri
Merge pull request #397 from VikParuchuri/vik_fixes
7f869ee 
Misc fixes
@iammosespaulr
Add CLI option to disable image extraction
76ab1de
@VikParuchuri VikParuchuri merged commit f446e56 into master Dec 3, 2024
2 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Dec 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@VikParuchuri VikParuchuri Awaiting requested review from VikParuchuri

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@iammosespaulr @VikParuchuri