Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional py parser for .md to json format conversion #21

Merged
merged 15 commits into from
Sep 27, 2024

Conversation

monk1337
Copy link
Contributor

@monk1337 monk1337 commented Aug 6, 2024

python parser and validator

This pull request introduces improvements to the markdown parsing and task conversion process. The changes affect two main files: parser.py and validate.py.

Changes in parser.py

  1. Robust Error Checking: Added comprehensive checks for all required sections in the markdown files (name, description, modality, examples, and tags).
  2. Custom Exception Handling: Introduced a MarkdownParsingError class for more specific error reporting.
  3. Multiple Examples Support: The parser now handles any number of examples in a markdown file.
  4. Detailed Error Messages: Provides specific error messages for missing or improperly formatted sections.
  5. Flexible Section Extraction: Implemented a helper function for consistent extraction of different sections.

Changes in validate.py

  1. Integration with Enhanced Parser: Updated to use the new robust parsing function from parser.py.
  2. Improved File Processing:
    • Checks if JSON files need updating based on modification times.
    • Creates the JSON output folder if it doesn't exist.
  3. Error Handling and Reporting:
    • Catches and reports both parsing errors and unexpected exceptions.
    • Provides informative console output for successful conversions and skipped files.

How to Use

  1. Ensure both parser.py and validate.py are in the same directory.
  2. Run validate.py:
    python validate.py
    
  3. The script will process all .md files in the tasks folder and output .json files to the tasks-json folder.

Copy link

netlify bot commented Aug 6, 2024

Deploy Preview for openreasoningtasks ready!

Name Link
🔨 Latest commit 682491f
🔍 Latest deploy log https://app.netlify.com/sites/openreasoningtasks/deploys/66b88838e7c21b000834fd83
😎 Deploy Preview https://deploy-preview-21--openreasoningtasks.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@N8python
Copy link
Contributor

N8python commented Aug 7, 2024

This is absolutely beautiful - wonderful improvement to my band-aid system. Only thing - perhaps validate.py should be remained to parseall.py or markdown-to-json.py as it does a lot more than just validating?

@teknium1 teknium1 merged commit 35df700 into NousResearch:main Sep 27, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants