Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature #600]: Use multiprocessing to speed up the parsing #601

Open
wants to merge 27 commits into
base: develop
Choose a base branch
from
+220 −28
Open

[Feature #600]: Use multiprocessing to speed up the parsing #601

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
0c0c945
Move Mastr initialization
AlexandraImbrisca Jan 22, 2025
4a5e427
Use a ProcessPoolExecutor to process multiple files at once
AlexandraImbrisca Jan 25, 2025
470a90c
Move the tables creation inside of the process_xml_file
AlexandraImbrisca Jan 26, 2025
418a554
Set the maximum number of processes to 3 as it was shown optimal in t…
AlexandraImbrisca Jan 26, 2025
76502cf
Add unit test for new function
AlexandraImbrisca Jan 26, 2025
436bb54
Remove print
AlexandraImbrisca Jan 26, 2025
ccf6b55
Set default number of processes to 1 and add warning for values too l…
AlexandraImbrisca Jan 28, 2025
6719194
Fix PostgreSQL incompatibility and add comments for database options
AlexandraImbrisca Jan 28, 2025
98cff60
Add USE_RECOMMENDED_NUMBER_OF_PROCESSES
AlexandraImbrisca Jan 28, 2025
e1a199d
Add check to introduce the new column only if not defined already
AlexandraImbrisca Jan 28, 2025
ee9b1af
Catch & ignore "duplicate column name" exception
AlexandraImbrisca Jan 28, 2025
5aecc8f
Add message introducing parallelized processing
AlexandraImbrisca Jan 28, 2025
d76db32
Update CHANGELOG.md
AlexandraImbrisca Jan 28, 2025
8efc3d8
Add missing columns in comments
AlexandraImbrisca Jan 28, 2025
2b196e8
Update docs to include new environment variables
AlexandraImbrisca Jan 28, 2025
139bedd
Remove unnecessary import
AlexandraImbrisca Jan 28, 2025
506e270
Separate SQLite-only options
AlexandraImbrisca Jan 28, 2025
1d90469
Adapted timeout option per engine
AlexandraImbrisca Jan 28, 2025
d844c2b
Replace obfuscated password
AlexandraImbrisca Jan 30, 2025
136abe8
Add quotes to comments
AlexandraImbrisca Jan 30, 2025
351faaa
Use regex to generalize password replacement
AlexandraImbrisca Jan 30, 2025
bfe1285
Fix regex in pw replacement
nesnoj Jan 31, 2025
572349c
Fix processing of NUMBER_OF_PROCESSES
nesnoj Jan 31, 2025
fd08980
Add try catch in process_xml_file
AlexandraImbrisca Feb 7, 2025
68ddeac
Use ProcessPoolExecutor only if the user has opted for parallelisation
AlexandraImbrisca Feb 25, 2025
30df437
Add note about if __name__ == "__main__"
AlexandraImbrisca Feb 25, 2025
26a3e82
Escape __ in documentation
AlexandraImbrisca Feb 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update CHANGELOG.md
@AlexandraImbrisca
AlexandraImbrisca committed Jan 28, 2025
commit d76db32934761ff5e9225bb8af3cca84a4cbca70
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -19,6 +19,8 @@ and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/
[#593](https://github.com/OpenEnergyPlatform/open-MaStR/pull/593)
- Delete Google Fonts from Documentation Page
[#599](https://github.com/OpenEnergyPlatform/open-MaStR/pull/599)
- Use multiprocessing to speed up parsing
[#601](https://github.com/OpenEnergyPlatform/open-MaStR/pull/601)
### Removed
- Moved old code artefacts from `scripts` folder to paper specific
[repository](https://github.com/FlorianK13/verify-marktstammdaten)