Yada #3
Yada #3
Conversation
…onal test dependencies
- Introduced a high-level interface for downloading IDX files, simplifying usage. - Encapsulated IDX downloading operations within the `IdxDownloader` class for better organization. - Enhanced error handling and logging during file downloads and cleanup processes. - Implemented concurrent download support for multiple IDX files. - Preserved original implementations for reference while updating methods for clarity and efficiency. This refactor addresses previous issues with scattered code and lack of batch processing capabilities, ensuring a more robust and maintainable solution.
WalkthroughThis pull request introduces a comprehensive enhancement to the SEC EDGAR Bulker project, focusing on establishing a robust configuration and testing framework. The changes include new configuration files like Changes
Sequence DiagramsequenceDiagram
participant CLI as CLI
participant Config as Configuration
participant Downloader as Edgar Downloader
participant ProxyManager as Proxy Manager
participant HeaderGenerator as Header Generator
participant IdxDownloader as Index Downloader
CLI->>Config: Load Configuration
Config-->>CLI: Validated Config
CLI->>ProxyManager: Initialize Proxies
CLI->>HeaderGenerator: Generate Headers
CLI->>IdxDownloader: Download Index Files
IdxDownloader-->>Downloader: Provide Submission Data
Downloader->>Downloader: Process Submissions
Downloader-->>CLI: Download Results
Poem
Tip CodeRabbit's docstrings feature is now available as part of our Early Access Program! Simply use the command Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 63
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (27)
.coveragerc(1 hunks).cursorrules(1 hunks).gitignore(1 hunks)config.yaml(1 hunks)pyproject.toml(4 hunks)pytest.ini(1 hunks)requirements.txt(1 hunks)src/sec_edgar_bulker/__init__.py(1 hunks)src/sec_edgar_bulker/cli/__init__.py(1 hunks)src/sec_edgar_bulker/cli/main.py(1 hunks)src/sec_edgar_bulker/core/__init__.py(1 hunks)src/sec_edgar_bulker/core/downloader.py(1 hunks)src/sec_edgar_bulker/core/exceptions.py(1 hunks)src/sec_edgar_bulker/core/header_generator.py(1 hunks)src/sec_edgar_bulker/core/idx_downloader.py(1 hunks)src/sec_edgar_bulker/core/models.py(1 hunks)src/sec_edgar_bulker/core/proxy_manager.py(1 hunks)src/sec_edgar_bulker/core/utils.py(1 hunks)tests/__init__.py(1 hunks)tests/conftest.py(1 hunks)tests/test_cli/test_main.py(1 hunks)tests/test_core/test_exceptions.py(1 hunks)tests/test_core/test_header_generator.py(1 hunks)tests/test_core/test_idx_downloader.py(1 hunks)tests/test_core/test_models.py(1 hunks)tests/test_core/test_proxy_manager.py(1 hunks)tests/test_core/test_utils.py(1 hunks)
🧰 Additional context used
🪛 Ruff (0.8.2)
tests/test_core/test_exceptions.py
15-15: Trailing comma missing
Add trailing comma
(COM812)
18-18: Missing return type annotation for public function test_timeout_exception
Add return type annotation: None
(ANN201)
25-25: Missing return type annotation for public function test_no_proxies_available_error
Add return type annotation: None
(ANN201)
32-32: Missing return type annotation for public function test_invalid_proxy_format_error
Add return type annotation: None
(ANN201)
39-39: Missing return type annotation for public function test_proxy_file_error
Add return type annotation: None
(ANN201)
46-46: Missing return type annotation for public function test_cache_error
Add return type annotation: None
(ANN201)
53-53: Missing return type annotation for public function test_config_error
Add return type annotation: None
(ANN201)
60-60: Missing return type annotation for public function test_parsing_error
Add return type annotation: None
(ANN201)
67-67: Missing return type annotation for public function test_download_mode_error
Add return type annotation: None
(ANN201)
74-74: Missing return type annotation for public function test_invalid_form_type_error
Add return type annotation: None
(ANN201)
81-81: Missing return type annotation for public function test_missing_metadata_error
Add return type annotation: None
(ANN201)
88-88: Missing return type annotation for public function test_file_integrity_error
Add return type annotation: None
(ANN201)
95-95: Missing return type annotation for public function test_exception_inheritance
Add return type annotation: None
(ANN201)
108-108: Trailing comma missing
Add trailing comma
(COM812)
114-114: Missing return type annotation for public function test_exception_with_nested_exception
Add return type annotation: None
(ANN201)
118-118: Abstract raise to an inner function
(TRY301)
118-118: Avoid specifying long messages outside the exception class
(TRY003)
120-120: Avoid specifying long messages outside the exception class
(TRY003)
122-122: Found assertion on exception e in except block, use pytest.raises() instead
(PT017)
123-123: Found assertion on exception e in except block, use pytest.raises() instead
(PT017)
125-125: Missing return type annotation for public function test_exception_without_message
Add return type annotation: None
(ANN201)
141-141: Trailing comma missing
Add trailing comma
(COM812)
143-143: Missing return type annotation for public function test_exception_creation
Add return type annotation: None
(ANN201)
143-143: Missing type annotation for function argument exception_class
(ANN001)
150-150: Missing return type annotation for public function test_exception_with_details
Add return type annotation: None
(ANN201)
157-157: Missing return type annotation for public function test_chained_exceptions
Add return type annotation: None
(ANN201)
162-162: Abstract raise to an inner function
(TRY301)
162-162: Avoid specifying long messages outside the exception class
(TRY003)
164-164: Avoid specifying long messages outside the exception class
(TRY003)
166-166: Avoid specifying long messages outside the exception class
(TRY003)
168-168: Found assertion on exception e in except block, use pytest.raises() instead
(PT017)
169-169: Found assertion on exception e in except block, use pytest.raises() instead
(PT017)
170-170: Found assertion on exception e in except block, use pytest.raises() instead
(PT017)
tests/test_core/test_models.py
22-22: Trailing comma missing
Add trailing comma
(COM812)
25-25: Missing return type annotation for public function test_proxy_settings_validation
Add return type annotation: None
(ANN201)
34-34: Trailing comma missing
Add trailing comma
(COM812)
43-43: Missing return type annotation for public function test_download_settings_validation
Add return type annotation: None
(ANN201)
51-51: Trailing comma missing
Add trailing comma
(COM812)
60-60: Missing return type annotation for public function test_processing_settings_validation
Add return type annotation: None
(ANN201)
65-65: Trailing comma missing
Add trailing comma
(COM812)
70-70: datetime.datetime.now() called without a tz argument
(DTZ005)
73-73: Missing return type annotation for public function test_header_generator_settings_validation
Add return type annotation: None
(ANN201)
80-80: Trailing comma missing
Add trailing comma
(COM812)
81-81: Trailing comma missing
Add trailing comma
(COM812)
86-86: Missing return type annotation for public function test_logging_settings_validation
Add return type annotation: None
(ANN201)
92-92: Trailing comma missing
Add trailing comma
(COM812)
100-100: Missing return type annotation for public function test_directory_settings_validation
Add return type annotation: None
(ANN201)
109-109: Trailing comma missing
Add trailing comma
(COM812)
114-114: Missing return type annotation for public function test_filtering_settings_validation
Add return type annotation: None
(ANN201)
122-122: Trailing comma missing
Add trailing comma
(COM812)
132-132: Trailing comma missing
Add trailing comma
(COM812)
135-135: Missing return type annotation for public function test_submission_model
Add return type annotation: None
(ANN201)
145-145: Trailing comma missing
Add trailing comma
(COM812)
151-151: Missing return type annotation for public function test_header_model
Add return type annotation: None
(ANN201)
158-158: datetime.datetime.now() called without a tz argument
(DTZ005)
159-159: Trailing comma missing
Add trailing comma
(COM812)
164-164: Missing return type annotation for public function test_document_metadata_model
Add return type annotation: None
(ANN201)
171-171: Trailing comma missing
Add trailing comma
(COM812)
176-176: Missing return type annotation for public function test_document_model
Add return type annotation: None
(ANN201)
183-183: Trailing comma missing
Add trailing comma
(COM812)
187-187: Trailing comma missing
Add trailing comma
(COM812)
192-192: Missing return type annotation for public function test_excluded_document_model
Add return type annotation: None
(ANN201)
199-199: Trailing comma missing
Add trailing comma
(COM812)
203-203: Trailing comma missing
Add trailing comma
(COM812)
208-208: Missing return type annotation for public function test_progress_model
Add return type annotation: None
(ANN201)
213-213: datetime.datetime.now() called without a tz argument
(DTZ005)
213-213: Trailing comma missing
Add trailing comma
(COM812)
218-218: Missing return type annotation for public function test_master_progress_model
Add return type annotation: None
(ANN201)
223-223: datetime.datetime.now() called without a tz argument
(DTZ005)
224-224: Trailing comma missing
Add trailing comma
(COM812)
230-230: Missing return type annotation for public function test_config_model_full
Add return type annotation: None
(ANN201)
230-230: Missing type annotation for function argument sample_config_dict
(ANN001)
239-239: Wrong type passed to first argument of pytest.mark.parametrize; expected tuple
Use a tuple for the first argument
(PT006)
246-246: Missing return type annotation for public function test_config_validation_cases
Add return type annotation: None
(ANN201)
246-246: Missing type annotation for function argument sample_config_dict
(ANN001)
246-246: Missing type annotation for function argument field
(ANN001)
246-246: Missing type annotation for function argument value
(ANN001)
246-246: Missing type annotation for function argument should_raise
(ANN001)
src/sec_edgar_bulker/cli/main.py
123-123: main is too complex (23 > 10)
(C901)
147-147: Abstract raise to an inner function
(TRY301)
147-147: Avoid specifying long messages outside the exception class
(TRY003)
155-155: Abstract raise to an inner function
(TRY301)
155-155: Avoid specifying long messages outside the exception class
(TRY003)
159-159: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
159-159: Avoid specifying long messages outside the exception class
(TRY003)
161-161: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
161-161: Avoid specifying long messages outside the exception class
(TRY003)
161-161: Use explicit conversion flag
Replace with conversion flag
(RUF010)
162-162: Do not catch blind exception: Exception
(BLE001)
163-163: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
163-163: Avoid specifying long messages outside the exception class
(TRY003)
163-163: Use explicit conversion flag
Replace with conversion flag
(RUF010)
168-168: Do not catch blind exception: Exception
(BLE001)
169-169: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
169-169: Avoid specifying long messages outside the exception class
(TRY003)
169-169: Use explicit conversion flag
Replace with conversion flag
(RUF010)
175-178: Abstract raise to an inner function
(TRY301)
175-178: Avoid specifying long messages outside the exception class
(TRY003)
177-177: Trailing comma missing
Add trailing comma
(COM812)
185-185: datetime.datetime.now() called without a tz argument
(DTZ005)
187-190: Avoid specifying long messages outside the exception class
(TRY003)
189-189: Trailing comma missing
Add trailing comma
(COM812)
193-196: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
193-196: Avoid specifying long messages outside the exception class
(TRY003)
195-195: Trailing comma missing
Add trailing comma
(COM812)
202-205: Avoid specifying long messages outside the exception class
(TRY003)
204-204: Trailing comma missing
Add trailing comma
(COM812)
208-211: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
208-211: Avoid specifying long messages outside the exception class
(TRY003)
210-210: Trailing comma missing
Add trailing comma
(COM812)
216-216: Do not catch blind exception: Exception
(BLE001)
217-217: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
217-217: Avoid specifying long messages outside the exception class
(TRY003)
217-217: Use explicit conversion flag
Replace with conversion flag
(RUF010)
223-223: Do not catch blind exception: Exception
(BLE001)
224-224: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
224-224: Avoid specifying long messages outside the exception class
(TRY003)
224-224: Use explicit conversion flag
Replace with conversion flag
(RUF010)
229-229: Do not catch blind exception: Exception
(BLE001)
230-230: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
230-230: Avoid specifying long messages outside the exception class
(TRY003)
230-230: Use explicit conversion flag
Replace with conversion flag
(RUF010)
236-236: Do not catch blind exception: Exception
(BLE001)
237-237: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
237-237: Logging statement uses f-string
(G004)
237-237: Use explicit conversion flag
Replace with conversion flag
(RUF010)
238-238: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
238-238: Avoid specifying long messages outside the exception class
(TRY003)
238-238: Use explicit conversion flag
Replace with conversion flag
(RUF010)
242-242: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
243-243: Use explicit conversion flag
Replace with conversion flag
(RUF010)
245-245: Do not catch blind exception: Exception
(BLE001)
246-246: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
246-246: Logging statement uses f-string
(G004)
246-246: Use explicit conversion flag
Replace with conversion flag
(RUF010)
247-247: Use explicit conversion flag
Replace with conversion flag
(RUF010)
tests/test_core/test_proxy_manager.py
5-5: pathlib.Path imported but unused
Remove unused import: pathlib.Path
(F401)
10-10: Trailing comma missing
Add trailing comma
(COM812)
14-14: Missing return type annotation for public function proxy_manager
(ANN201)
14-14: Missing type annotation for function argument sample_config
(ANN001)
18-18: Missing return type annotation for public function test_proxy_manager_init
Add return type annotation: None
(ANN201)
18-18: Missing type annotation for function argument proxy_manager
(ANN001)
18-18: Missing type annotation for function argument sample_config
(ANN001)
26-26: Missing return type annotation for public function test_proxy_manager_disabled
Add return type annotation: None
(ANN201)
26-26: Missing type annotation for function argument sample_config
(ANN001)
33-33: Missing return type annotation for public function test_load_proxies_valid
Add return type annotation: None
(ANN201)
33-33: Missing type annotation for function argument sample_config
(ANN001)
33-33: Missing type annotation for function argument sample_proxy_file
(ANN001)
43-43: Missing return type annotation for public function test_load_proxies_invalid_format
Add return type annotation: None
(ANN201)
43-43: Missing type annotation for function argument sample_config
(ANN001)
43-43: Missing type annotation for function argument tmp_path
(ANN001)
53-53: Missing return type annotation for public function test_load_proxies_missing_file
Add return type annotation: None
(ANN201)
53-53: Missing type annotation for function argument sample_config
(ANN001)
61-61: Missing return type annotation for public function test_get_random_proxy_random_mode
Add return type annotation: None
(ANN201)
61-61: Missing type annotation for function argument sample_config
(ANN001)
61-61: Missing type annotation for function argument sample_proxy_file
(ANN001)
73-73: Missing return type annotation for public function test_get_random_proxy_lru_mode
Add return type annotation: None
(ANN201)
73-73: Missing type annotation for function argument sample_config
(ANN001)
73-73: Missing type annotation for function argument sample_proxy_file
(ANN001)
84-84: Missing return type annotation for public function test_proxy_usage_limit
Add return type annotation: None
(ANN201)
84-84: Missing type annotation for function argument sample_config
(ANN001)
84-84: Missing type annotation for function argument sample_proxy_file
(ANN001)
103-103: Missing return type annotation for public function test_proxy_auth_security
Add return type annotation: None
(ANN201)
103-103: Missing type annotation for function argument sample_config
(ANN001)
103-103: Missing type annotation for function argument sample_proxy_file
(ANN001)
121-121: Missing return type annotation for public function test_proxy_rotation
Add return type annotation: None
(ANN201)
121-121: Missing type annotation for function argument sample_config
(ANN001)
121-121: Missing type annotation for function argument sample_proxy_file
(ANN001)
135-135: Missing return type annotation for public function test_proxy_reset_after_limit
Add return type annotation: None
(ANN201)
135-135: Missing type annotation for function argument sample_config
(ANN001)
135-135: Missing type annotation for function argument sample_proxy_file
(ANN001)
161-161: Missing return type annotation for public function test_proxy_format_variations
Add return type annotation: None
(ANN201)
161-161: Missing type annotation for function argument sample_config
(ANN001)
161-161: Missing type annotation for function argument tmp_path
(ANN001)
161-161: Missing type annotation for function argument proxy_line
(ANN001)
tests/test_cli/test_main.py
11-11: Missing return type annotation for public function cli_runner
(ANN201)
16-16: Missing return type annotation for public function config_file
(ANN201)
16-16: Missing type annotation for function argument tmp_path
(ANN001)
16-16: Missing type annotation for function argument sample_config_dict
(ANN001)
23-23: Missing return type annotation for public function test_cli_help
Add return type annotation: None
(ANN201)
23-23: Missing type annotation for function argument cli_runner
(ANN001)
33-33: Missing return type annotation for public function test_cli_missing_config
Add return type annotation: None
(ANN201)
33-33: Missing type annotation for function argument cli_runner
(ANN001)
40-40: Missing return type annotation for public function test_cli_invalid_config_format
Add return type annotation: None
(ANN201)
40-40: Missing type annotation for function argument cli_runner
(ANN001)
40-40: Missing type annotation for function argument tmp_path
(ANN001)
49-49: Missing return type annotation for public function test_cli_valid_config
Add return type annotation: None
(ANN201)
49-49: Missing type annotation for function argument cli_runner
(ANN001)
49-49: Missing type annotation for function argument config_file
(ANN001)
49-49: Missing type annotation for function argument mocker
(ANN001)
58-58: Missing return type annotation for public function test_cli_download_mode_override
Add return type annotation: None
(ANN201)
58-58: Missing type annotation for function argument cli_runner
(ANN001)
58-58: Missing type annotation for function argument config_file
(ANN001)
58-58: Missing type annotation for function argument mocker
(ANN001)
64-64: Trailing comma missing
Add trailing comma
(COM812)
73-73: Missing return type annotation for public function test_cli_years_override
Add return type annotation: None
(ANN201)
73-73: Missing type annotation for function argument cli_runner
(ANN001)
73-73: Missing type annotation for function argument config_file
(ANN001)
73-73: Missing type annotation for function argument mocker
(ANN001)
79-79: Trailing comma missing
Add trailing comma
(COM812)
88-88: Missing return type annotation for public function test_cli_quarters_override
Add return type annotation: None
(ANN201)
88-88: Missing type annotation for function argument cli_runner
(ANN001)
88-88: Missing type annotation for function argument config_file
(ANN001)
88-88: Missing type annotation for function argument mocker
(ANN001)
94-94: Trailing comma missing
Add trailing comma
(COM812)
103-103: Missing return type annotation for public function test_cli_invalid_download_mode
Add return type annotation: None
(ANN201)
103-103: Missing type annotation for function argument cli_runner
(ANN001)
103-103: Missing type annotation for function argument config_file
(ANN001)
107-107: Trailing comma missing
Add trailing comma
(COM812)
113-113: Missing return type annotation for public function test_cli_invalid_years_format
Add return type annotation: None
(ANN201)
113-113: Missing type annotation for function argument cli_runner
(ANN001)
113-113: Missing type annotation for function argument config_file
(ANN001)
117-117: Trailing comma missing
Add trailing comma
(COM812)
122-122: Missing return type annotation for public function test_cli_invalid_quarters_format
Add return type annotation: None
(ANN201)
122-122: Missing type annotation for function argument cli_runner
(ANN001)
122-122: Missing type annotation for function argument config_file
(ANN001)
126-126: Trailing comma missing
Add trailing comma
(COM812)
131-131: Missing return type annotation for public function test_cli_invalid_quarters_range
Add return type annotation: None
(ANN201)
131-131: Missing type annotation for function argument cli_runner
(ANN001)
131-131: Missing type annotation for function argument config_file
(ANN001)
135-135: Trailing comma missing
Add trailing comma
(COM812)
140-140: Missing return type annotation for public function test_cli_directory_creation
Add return type annotation: None
(ANN201)
140-140: Missing type annotation for function argument cli_runner
(ANN001)
140-140: Missing type annotation for function argument config_file
(ANN001)
140-140: Missing type annotation for function argument tmp_path
(ANN001)
140-140: Missing type annotation for function argument mocker
(ANN001)
142-142: Local variable mock_run is assigned to but never used
Remove assignment to unused variable mock_run
(F841)
163-163: Missing return type annotation for public function test_cli_logging_setup
Add return type annotation: None
(ANN201)
163-163: Missing type annotation for function argument cli_runner
(ANN001)
163-163: Missing type annotation for function argument config_file
(ANN001)
163-163: Missing type annotation for function argument tmp_path
(ANN001)
163-163: Unused function argument: tmp_path
(ARG001)
163-163: Missing type annotation for function argument mocker
(ANN001)
166-166: Local variable mock_run is assigned to but never used
Remove assignment to unused variable mock_run
(F841)
178-178: Missing return type annotation for public function test_cli_run_downloader
Add return type annotation: None
(ANN201)
178-178: Missing type annotation for function argument cli_runner
(ANN001)
178-178: Missing type annotation for function argument config_file
(ANN001)
178-178: Missing type annotation for function argument mocker
(ANN001)
190-190: Missing return type annotation for public function test_cli_config_validation_error
Add return type annotation: None
(ANN201)
190-190: Missing type annotation for function argument cli_runner
(ANN001)
190-190: Missing type annotation for function argument config_file
(ANN001)
190-190: Missing type annotation for function argument mocker
(ANN001)
194-194: Trailing comma missing
Add trailing comma
(COM812)
tests/test_core/test_idx_downloader.py
6-6: asyncio imported but unused
Remove unused import: asyncio
(F401)
7-7: unittest.mock.Mock imported but unused
Remove unused import
(F401)
7-7: unittest.mock.patch imported but unused
Remove unused import
(F401)
9-9: sec_edgar_bulker.core.idx_downloader.IdxDownloader imported but unused
Remove unused import: sec_edgar_bulker.core.idx_downloader.IdxDownloader
(F401)
10-10: sec_edgar_bulker.core.models.Config imported but unused
Remove unused import: sec_edgar_bulker.core.models.Config
(F401)
13-13: Missing return type annotation for public function test_download_idx_file_success
Add return type annotation: None
(ANN201)
13-13: Missing type annotation for function argument sample_config
(ANN001)
13-13: Missing type annotation for function argument mock_aioresponse
(ANN001)
32-32: Missing return type annotation for public function test_download_idx_file_failure
Add return type annotation: None
(ANN201)
32-32: Missing type annotation for function argument sample_config
(ANN001)
32-32: Missing type annotation for function argument mock_aioresponse
(ANN001)
50-50: Missing return type annotation for public function test_download_idx_file_yearly
Add return type annotation: None
(ANN201)
50-50: Missing type annotation for function argument sample_config
(ANN001)
50-50: Missing type annotation for function argument mock_aioresponse
(ANN001)
69-69: Missing return type annotation for public function test_download_idx_file_network_error
Add return type annotation: None
(ANN201)
69-69: Missing type annotation for function argument sample_config
(ANN001)
69-69: Missing type annotation for function argument mock_aioresponse
(ANN001)
85-85: Missing return type annotation for public function test_download_and_process_idx_files_success
Add return type annotation: None
(ANN201)
85-85: Missing type annotation for function argument sample_config
(ANN001)
85-85: Missing type annotation for function argument mock_aioresponse
(ANN001)
99-99: Undefined name idx_type
(F821)
104-104: Missing return type annotation for public function test_download_and_process_idx_files_partial_failure
Add return type annotation: None
(ANN201)
104-104: Missing type annotation for function argument sample_config
(ANN001)
104-104: Missing type annotation for function argument mock_aioresponse
(ANN001)
120-120: Undefined name idx_type
(F821)
125-125: Undefined name idx_type
(F821)
129-129: Missing return type annotation for public function test_download_idx_file_creates_directory
Add return type annotation: None
(ANN201)
129-129: Missing type annotation for function argument sample_config
(ANN001)
129-129: Missing type annotation for function argument mock_aioresponse
(ANN001)
152-152: Missing return type annotation for public function test_download_idx_file_overwrites_existing
Add return type annotation: None
(ANN201)
152-152: Missing type annotation for function argument sample_config
(ANN001)
152-152: Missing type annotation for function argument mock_aioresponse
(ANN001)
175-175: Missing return type annotation for public function test_download_idx_file_large_content
Add return type annotation: None
(ANN201)
175-175: Missing type annotation for function argument sample_config
(ANN001)
175-175: Missing type annotation for function argument mock_aioresponse
(ANN001)
197-197: Missing return type annotation for public function test_download_idx_file_unicode_content
Add return type annotation: None
(ANN201)
197-197: Missing type annotation for function argument sample_config
(ANN001)
197-197: Missing type annotation for function argument mock_aioresponse
(ANN001)
src/sec_edgar_bulker/__init__.py
10-10: Trailing comma missing
Add trailing comma
(COM812)
15-23: __all__ is not sorted
Apply an isort-style sorting to __all__
(RUF022)
22-22: Trailing comma missing
Add trailing comma
(COM812)
tests/test_core/test_utils.py
5-5: pathlib.Path imported but unused
Remove unused import: pathlib.Path
(F401)
9-9: sec_edgar_bulker.core.models.Config imported but unused
Remove unused import: sec_edgar_bulker.core.models.Config
(F401)
11-11: Missing return type annotation for public function test_setup_logging_with_file
Add return type annotation: None
(ANN201)
11-11: Missing type annotation for function argument tmp_path
(ANN001)
18-18: Trailing comma missing
Add trailing comma
(COM812)
38-38: Missing return type annotation for public function test_setup_logging_without_file
Add return type annotation: None
(ANN201)
44-44: Trailing comma missing
Add trailing comma
(COM812)
55-55: Missing return type annotation for public function test_setup_logging_levels
Add return type annotation: None
(ANN201)
55-55: Missing type annotation for function argument level
(ANN001)
61-61: Trailing comma missing
Add trailing comma
(COM812)
67-67: Missing return type annotation for public function test_setup_logging_custom_format
Add return type annotation: None
(ANN201)
74-74: Trailing comma missing
Add trailing comma
(COM812)
81-81: Missing return type annotation for public function test_validate_config_valid
Add return type annotation: None
(ANN201)
81-81: Missing type annotation for function argument sample_config
(ANN001)
85-85: Missing return type annotation for public function test_validate_config_no_years
Add return type annotation: None
(ANN201)
85-85: Missing type annotation for function argument sample_config
(ANN001)
91-91: Missing return type annotation for public function test_validate_config_no_quarters
Add return type annotation: None
(ANN201)
91-91: Missing type annotation for function argument sample_config
(ANN001)
97-97: Missing return type annotation for public function test_validate_config_company_filter_no_identifiers
Add return type annotation: None
(ANN201)
97-97: Missing type annotation for function argument sample_config
(ANN001)
105-105: Missing return type annotation for public function test_validate_config_date_filter_no_range
Add return type annotation: None
(ANN201)
105-105: Missing type annotation for function argument sample_config
(ANN001)
113-113: Missing return type annotation for public function test_generate_batch_id
Add return type annotation: None
(ANN201)
119-119: datetime.datetime.now() called without a tz argument
(DTZ005)
123-123: Naive datetime constructed using datetime.datetime.strptime() without %z
(DTZ007)
130-130: Trailing comma missing
Add trailing comma
(COM812)
132-132: Missing return type annotation for public function test_generate_batch_id_formats
Add return type annotation: None
(ANN201)
132-132: Missing type annotation for function argument format_str
(ANN001)
136-136: Naive datetime constructed using datetime.datetime.strptime() without %z
(DTZ007)
140-140: Missing return type annotation for public function test_setup_logging_file_creation
Add return type annotation: None
(ANN201)
140-140: Missing type annotation for function argument tmp_path
(ANN001)
147-147: Trailing comma missing
Add trailing comma
(COM812)
156-156: Missing return type annotation for public function test_setup_logging_directory_creation
Add return type annotation: None
(ANN201)
156-156: Missing type annotation for function argument tmp_path
(ANN001)
164-164: Trailing comma missing
Add trailing comma
(COM812)
173-173: Missing return type annotation for public function test_setup_logging_rotation
Add return type annotation: None
(ANN201)
173-173: Missing type annotation for function argument tmp_path
(ANN001)
180-180: Trailing comma missing
Add trailing comma
(COM812)
184-184: Local variable handler is assigned to but never used
Remove assignment to unused variable handler
(F841)
197-197: Trailing comma missing
Add trailing comma
(COM812)
199-199: Missing return type annotation for public function test_generate_batch_id_invalid_format
Add return type annotation: None
(ANN201)
199-199: Missing type annotation for function argument invalid_format
(ANN001)
201-201: pytest.raises(ValueError) is too broad, set the match parameter or use a more specific exception
(PT011)
204-204: Missing return type annotation for public function test_validate_config_filtering_dates
Add return type annotation: None
(ANN201)
204-204: Missing type annotation for function argument sample_config
(ANN001)
tests/conftest.py
4-4: pathlib.Path imported but unused
Remove unused import: pathlib.Path
(F401)
8-8: Missing return type annotation for public function sample_config_dict
(ANN201)
20-20: Trailing comma missing
Add trailing comma
(COM812)
28-28: Trailing comma missing
Add trailing comma
(COM812)
36-36: Trailing comma missing
Add trailing comma
(COM812)
41-41: Trailing comma missing
Add trailing comma
(COM812)
48-48: Trailing comma missing
Add trailing comma
(COM812)
52-52: Trailing comma missing
Add trailing comma
(COM812)
53-53: Trailing comma missing
Add trailing comma
(COM812)
61-61: Trailing comma missing
Add trailing comma
(COM812)
69-69: Trailing comma missing
Add trailing comma
(COM812)
78-78: Trailing comma missing
Add trailing comma
(COM812)
79-79: Trailing comma missing
Add trailing comma
(COM812)
91-91: Trailing comma missing
Add trailing comma
(COM812)
92-92: Trailing comma missing
Add trailing comma
(COM812)
93-93: Trailing comma missing
Add trailing comma
(COM812)
97-97: Missing return type annotation for public function sample_config
(ANN201)
97-97: Missing type annotation for function argument sample_config_dict
(ANN001)
102-102: Missing return type annotation for public function sample_proxy_file
(ANN201)
102-102: Missing type annotation for function argument tmp_path
(ANN001)
113-113: mock_aioresponse is too complex (12 > 10)
(C901)
113-113: Missing return type annotation for public function mock_aioresponse
(ANN201)
116-116: Missing return type annotation for special method __init__
Add return type annotation: None
(ANN204)
116-116: Missing type annotation for function argument status
(ANN001)
116-116: Missing type annotation for function argument body
(ANN001)
120-120: Missing return type annotation for special method __aenter__
(ANN204)
123-123: Missing return type annotation for special method __aexit__
(ANN204)
123-123: Missing type annotation for function argument exc_type
(ANN001)
123-123: Missing type annotation for function argument exc_val
(ANN001)
123-123: Missing type annotation for function argument exc_tb
(ANN001)
126-126: Missing return type annotation for private function text
(ANN202)
129-129: Missing return type annotation for private function read
(ANN202)
133-133: Missing return type annotation for special method __init__
Add return type annotation: None
(ANN204)
136-136: Missing return type annotation for private function get
Add return type annotation: None
(ANN202)
136-136: Missing type annotation for function argument url
(ANN001)
136-136: Missing type annotation for function argument status
(ANN001)
136-136: Missing type annotation for function argument body
(ANN001)
136-136: Missing type annotation for function argument exception
(ANN001)
142-142: Missing return type annotation for special method __call__
(ANN204)
142-142: Missing type annotation for function argument method
(ANN001)
142-142: Unused method argument: method
(ARG002)
142-142: Missing type annotation for function argument url
(ANN001)
142-142: Missing type annotation for **kwargs
(ANN003)
142-142: Unused method argument: kwargs
(ARG002)
src/sec_edgar_bulker/core/exceptions.py
5-5: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
9-9: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
13-13: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
17-17: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
21-21: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
25-25: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
29-29: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
33-33: Missing return type annotation for special method __init__
Add return type annotation: None
(ANN204)
33-33: Missing type annotation for function argument message
(ANN001)
33-33: Missing type annotation for function argument details
(ANN001)
39-39: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
43-43: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
47-47: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
src/sec_edgar_bulker/core/utils.py
34-34: Abstract raise to an inner function
(TRY301)
34-34: Avoid specifying long messages outside the exception class
(TRY003)
54-54: os.chmod setting a permissive mask 0o755 on file or directory
(S103)
65-65: Trailing comma missing
Add trailing comma
(COM812)
76-76: Consider moving this statement to an else block
(TRY300)
84-84: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
84-84: Logging statement uses f-string
(G004)
100-101: Use a single if statement instead of nested if statements
(SIM102)
110-110: Naive datetime constructed using datetime.datetime.strptime() without %z
(DTZ007)
111-111: Naive datetime constructed using datetime.datetime.strptime() without %z
(DTZ007)
133-133: Avoid specifying long messages outside the exception class
(TRY003)
137-137: datetime.datetime() called without a tzinfo argument
(DTZ001)
141-141: Naive datetime constructed using datetime.datetime.strptime() without %z
(DTZ007)
143-143: Abstract raise to an inner function
(TRY301)
143-143: Avoid specifying long messages outside the exception class
(TRY003)
145-145: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
145-145: Avoid specifying long messages outside the exception class
(TRY003)
145-145: Use explicit conversion flag
Replace with conversion flag
(RUF010)
148-148: datetime.datetime.now() called without a tz argument
(DTZ005)
153-153: Abstract raise to an inner function
(TRY301)
153-153: Avoid specifying long messages outside the exception class
(TRY003)
155-155: Consider moving this statement to an else block
(TRY300)
160-160: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
160-160: Avoid specifying long messages outside the exception class
(TRY003)
160-160: Use explicit conversion flag
Replace with conversion flag
(RUF010)
168-168: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
src/sec_edgar_bulker/core/idx_downloader.py
9-9: typing.Dict is deprecated, use dict instead
(UP035)
9-9: typing.List is deprecated, use list instead
(UP035)
9-9: typing.Tuple is deprecated, use tuple instead
(UP035)
9-9: typing.Any imported but unused
Remove unused import: typing.Any
(F401)
15-15: .exceptions.DownloadModeError imported but unused
Remove unused import
(F401)
16-16: .exceptions.FileIntegrityError imported but unused
Remove unused import
(F401)
17-17: .exceptions.InvalidProxyFormatError imported but unused
Remove unused import
(F401)
19-19: .exceptions.ProxyFileError imported but unused
Remove unused import
(F401)
52-52: Do not catch blind exception: Exception
(BLE001)
53-53: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
53-53: Logging statement uses f-string
(G004)
73-73: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
116-116: Logging statement uses f-string
(G004)
116-116: Trailing comma missing
Add trailing comma
(COM812)
132-132: os.chmod setting a permissive mask 0o755 on file or directory
(S103)
133-133: Logging statement uses f-string
(G004)
134-134: Do not catch blind exception: Exception
(BLE001)
135-135: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
135-135: Logging statement uses f-string
(G004)
148-148: Trailing comma missing
Add trailing comma
(COM812)
152-152: Trailing comma missing
Add trailing comma
(COM812)
157-157: Consider moving this statement to an else block
(TRY300)
159-159: Do not catch blind exception: Exception
(BLE001)
160-160: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
160-160: Logging statement uses f-string
(G004)
176-176: Logging statement uses f-string
(G004)
177-177: Do not catch blind exception: Exception
(BLE001)
178-178: Logging statement uses f-string
(G004)
184-184: _download_file is too complex (24 > 10)
(C901)
218-218: Logging statement uses f-string
(G004)
219-219: Do not catch blind exception: Exception
(BLE001)
220-220: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
220-220: Logging statement uses f-string
(G004)
225-225: os.chmod setting a permissive mask 0o755 on file or directory
(S103)
226-226: Logging statement uses f-string
(G004)
227-227: Do not catch blind exception: Exception
(BLE001)
228-228: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
228-228: Logging statement uses f-string
(G004)
243-243: Do not catch blind exception: Exception
(BLE001)
244-244: Logging statement uses f-string
(G004)
261-261: Logging statement uses f-string
(G004)
261-261: Trailing comma missing
Add trailing comma
(COM812)
267-267: Logging statement uses f-string
(G004)
273-274: Logging statement uses f-string
(G004)
274-274: Trailing comma missing
Add trailing comma
(COM812)
285-285: Logging statement uses f-string
(G004)
288-288: Use explicit conversion flag
Replace with conversion flag
(RUF010)
293-293: Logging statement uses f-string
(G004)
293-293: Trailing comma missing
Add trailing comma
(COM812)
298-298: Logging statement uses f-string
(G004)
309-309: Do not catch blind exception: Exception
(BLE001)
310-312: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
311-311: Logging statement uses f-string
(G004)
311-311: Trailing comma missing
Add trailing comma
(COM812)
322-322: Logging statement uses f-string
(G004)
322-322: Trailing comma missing
Add trailing comma
(COM812)
324-324: Consider moving this statement to an else block
(TRY300)
325-325: Do not catch blind exception: Exception
(BLE001)
326-328: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
327-327: Logging statement uses f-string
(G004)
327-327: Trailing comma missing
Add trailing comma
(COM812)
331-331: Logging statement uses f-string
(G004)
336-336: Logging statement uses f-string
(G004)
338-338: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
338-338: Avoid specifying long messages outside the exception class
(TRY003)
341-341: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
341-341: Logging statement uses f-string
(G004)
354-354: Use list instead of List for type annotation
Replace with list
(UP006)
355-355: Use list instead of List for type annotation
Replace with list
(UP006)
356-356: Trailing comma missing
Add trailing comma
(COM812)
357-357: Use tuple instead of Tuple for type annotation
Replace with tuple
(UP006)
386-386: Logging statement uses f-string
(G004)
408-408: Logging statement uses f-string
(G004)
430-430: Loop control variable filename not used within loop body
Rename unused filename to _filename
(B007)
439-439: Logging statement uses f-string
(G004)
440-440: Do not catch blind exception: Exception
(BLE001)
441-441: Logging statement uses f-string
(G004)
442-442: Do not catch blind exception: Exception
(BLE001)
443-443: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
443-443: Logging statement uses f-string
(G004)
453-453: Use list instead of List for type annotation
Replace with list
(UP006)
454-454: Use list instead of List for type annotation
Replace with list
(UP006)
455-455: Trailing comma missing
Add trailing comma
(COM812)
456-456: Use tuple instead of Tuple for type annotation
Replace with tuple
(UP006)
466-470: __all__ is not sorted
Apply an isort-style sorting to __all__
(RUF022)
469-469: Trailing comma missing
Add trailing comma
(COM812)
tests/test_core/test_header_generator.py
5-5: sec_edgar_bulker.core.models.Config imported but unused
Remove unused import: sec_edgar_bulker.core.models.Config
(F401)
8-8: Missing return type annotation for public function header_generator
(ANN201)
8-8: Missing type annotation for function argument sample_config
(ANN001)
12-12: Missing return type annotation for public function test_header_generator_init
Add return type annotation: None
(ANN201)
12-12: Missing type annotation for function argument header_generator
(ANN001)
12-12: Missing type annotation for function argument sample_config
(ANN001)
17-17: Missing return type annotation for public function test_get_random_headers_with_generator
Add return type annotation: None
(ANN201)
17-17: Missing type annotation for function argument header_generator
(ANN001)
28-28: Missing return type annotation for public function test_get_random_headers_without_generator
Add return type annotation: None
(ANN201)
28-28: Missing type annotation for function argument sample_config
(ANN001)
35-35: Missing return type annotation for public function test_get_random_headers_without_random_settings
Add return type annotation: None
(ANN201)
35-35: Missing type annotation for function argument sample_config
(ANN001)
43-43: Missing return type annotation for public function test_header_generator_no_sensitive_data
Add return type annotation: None
(ANN201)
43-43: Missing type annotation for function argument header_generator
(ANN001)
52-52: Missing return type annotation for public function test_header_generator_unique_headers
Add return type annotation: None
(ANN201)
52-52: Missing type annotation for function argument header_generator
(ANN001)
58-58: Missing return type annotation for public function test_header_generator_required_fields
Add return type annotation: None
(ANN201)
58-58: Missing type annotation for function argument header_generator
(ANN001)
66-66: Trailing comma missing
Add trailing comma
(COM812)
70-70: Missing return type annotation for public function test_header_generator_valid_email_format
Add return type annotation: None
(ANN201)
70-70: Missing type annotation for function argument header_generator
(ANN001)
80-80: Missing return type annotation for public function test_header_generator_consistent_host
Add return type annotation: None
(ANN201)
80-80: Missing type annotation for function argument header_generator
(ANN001)
86-86: Missing return type annotation for public function test_header_generator_encoding_options
Add return type annotation: None
(ANN201)
86-86: Missing type annotation for function argument header_generator
(ANN001)
86-86: Missing type annotation for function argument encoding
(ANN001)
src/sec_edgar_bulker/core/proxy_manager.py
4-4: typing.List is deprecated, use list instead
(UP035)
4-4: typing.Dict is deprecated, use dict instead
(UP035)
4-4: typing.Tuple is deprecated, use tuple instead
(UP035)
4-4: typing.Set is deprecated, use set instead
(UP035)
8-8: os imported but unused
Remove unused import: os
(F401)
18-18: Missing return type annotation for special method __init__
Add return type annotation: None
(ANN204)
30-30: Use list instead of List for type annotation
Replace with list
(UP006)
30-30: Use tuple instead of Tuple for type annotation
Replace with tuple
(UP006)
31-31: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
32-32: Use set instead of Set for type annotation
Replace with set
(UP006)
36-37: Use a single if statement instead of nested if statements
(SIM102)
39-39: Avoid specifying long messages outside the exception class
(TRY003)
44-44: load_proxies is too complex (19 > 10)
(C901)
55-55: Logging statement uses f-string
(G004)
57-57: Abstract raise to an inner function
(TRY301)
57-57: Avoid specifying long messages outside the exception class
(TRY003)
58-58: Abstract raise to an inner function
(TRY301)
58-58: Avoid specifying long messages outside the exception class
(TRY003)
60-60: Unnecessary open mode parameters
Remove open mode parameters
(UP015)
66-66: Abstract raise to an inner function
(TRY301)
67-67: Abstract raise to an inner function
(TRY301)
72-72: Local variable ip_pattern is assigned to but never used
Remove assignment to unused variable ip_pattern
(F841)
78-78: Logging statement uses f-string
(G004)
79-79: Abstract raise to an inner function
(TRY301)
79-79: Avoid specifying long messages outside the exception class
(TRY003)
85-85: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
85-85: Logging statement uses f-string
(G004)
86-86: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
86-86: Avoid specifying long messages outside the exception class
(TRY003)
90-90: Logging statement uses f-string
(G004)
91-91: Abstract raise to an inner function
(TRY301)
91-91: Avoid specifying long messages outside the exception class
(TRY003)
100-100: Abstract raise to an inner function
(TRY301)
100-100: Avoid specifying long messages outside the exception class
(TRY003)
106-106: Trailing comma missing
Add trailing comma
(COM812)
111-111: Logging statement uses f-string
(G004)
113-113: Logging statement uses f-string
(G004)
121-121: Abstract raise to an inner function
(TRY301)
121-121: Avoid specifying long messages outside the exception class
(TRY003)
124-124: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
124-124: Logging statement uses f-string
(G004)
126-126: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
126-126: Avoid specifying long messages outside the exception class
(TRY003)
127-127: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
127-127: Avoid specifying long messages outside the exception class
(TRY003)
130-130: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
130-130: Avoid specifying long messages outside the exception class
(TRY003)
132-132: Do not catch blind exception: Exception
(BLE001)
133-133: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
133-133: Logging statement uses f-string
(G004)
135-135: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
135-135: Avoid specifying long messages outside the exception class
(TRY003)
136-136: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
136-136: Avoid specifying long messages outside the exception class
(TRY003)
143-143: Use tuple instead of Tuple for type annotation
Replace with tuple
(UP006)
154-154: Avoid specifying long messages outside the exception class
(TRY003)
165-165: Avoid specifying long messages outside the exception class
(TRY003)
174-174: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
176-176: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
187-187: Local variable port is assigned to but never used
Remove assignment to unused variable port
(F841)
189-189: Logging statement uses f-string
(G004)
192-192: Use tuple instead of Tuple for type annotation
Replace with tuple
(UP006)
203-203: Avoid specifying long messages outside the exception class
(TRY003)
217-217: Avoid specifying long messages outside the exception class
(TRY003)
234-234: Local variable port is assigned to but never used
Remove assignment to unused variable port
(F841)
236-236: Logging statement uses f-string
(G004)
251-251: Local variable port is assigned to but never used
Remove assignment to unused variable port
(F841)
src/sec_edgar_bulker/core/__init__.py
15-15: Trailing comma missing
Add trailing comma
(COM812)
21-21: Trailing comma missing
Add trailing comma
(COM812)
25-45: __all__ is not sorted
Apply an isort-style sorting to __all__
(RUF022)
44-44: Trailing comma missing
Add trailing comma
(COM812)
src/sec_edgar_bulker/core/header_generator.py
5-5: typing.Dict is deprecated, use dict instead
(UP035)
7-7: .models.HeaderGeneratorSettings imported but unused
Remove unused import
(F401)
7-7: .models.StaticHeaders imported but unused
Remove unused import
(F401)
47-47: Trailing comma missing
Add trailing comma
(COM812)
51-51: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
66-66: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
69-69: Avoid specifying long messages outside the exception class
(TRY003)
71-71: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
74-74: Trailing comma missing
Add trailing comma
(COM812)
80-80: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
89-89: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
src/sec_edgar_bulker/core/models.py
3-3: typing.List is deprecated, use list instead
(UP035)
3-3: typing.Dict is deprecated, use dict instead
(UP035)
3-3: typing.Any imported but unused
Remove unused import: typing.Any
(F401)
82-82: Use list instead of List for type annotation
Replace with list
(UP006)
82-82: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
90-90: Use list instead of List for type annotation
Replace with list
(UP006)
122-122: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
134-134: Trailing comma missing
Add trailing comma
(COM812)
144-144: Use list instead of List for type annotation
Replace with list
(UP006)
193-193: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
203-203: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
203-203: Use list instead of List for type annotation
Replace with list
(UP006)
212-212: Use list instead of List for type annotation
Replace with list
(UP006)
213-213: Use list instead of List for type annotation
Replace with list
(UP006)
259-259: Avoid specifying long messages outside the exception class
(TRY003)
263-263: Unused method argument: info
(ARG002)
309-309: Use list instead of List for type annotation
Replace with list
(UP006)
310-310: Use list instead of List for type annotation
Replace with list
(UP006)
315-315: Unused method argument: info
(ARG002)
329-329: Avoid specifying long messages outside the exception class
(TRY003)
331-331: Naive datetime constructed using datetime.datetime.strptime() without %z
(DTZ007)
332-332: Naive datetime constructed using datetime.datetime.strptime() without %z
(DTZ007)
334-334: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
334-334: Avoid specifying long messages outside the exception class
(TRY003)
356-356: Use list instead of List for type annotation
Replace with list
(UP006)
357-357: Use list instead of List for type annotation
Replace with list
(UP006)
362-362: Trailing comma missing
Add trailing comma
(COM812)
367-367: Trailing comma missing
Add trailing comma
(COM812)
370-370: Use list instead of List for type annotation
Replace with list
(UP006)
371-371: Use list instead of List for type annotation
Replace with list
(UP006)
372-372: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
372-372: Use list instead of List for type annotation
Replace with list
(UP006)
381-381: Use list instead of List for type annotation
Replace with list
(UP006)
381-381: Use list instead of List for type annotation
Replace with list
(UP006)
394-394: Avoid specifying long messages outside the exception class
(TRY003)
399-399: Use list instead of List for type annotation
Replace with list
(UP006)
399-399: Use list instead of List for type annotation
Replace with list
(UP006)
411-411: datetime.datetime.now() called without a tz argument
(DTZ005)
413-413: Avoid specifying long messages outside the exception class
(TRY003)
496-496: Use list instead of List for type annotation
Replace with list
(UP006)
src/sec_edgar_bulker/core/downloader.py
9-9: typing.Dict is deprecated, use dict instead
(UP035)
9-9: typing.List is deprecated, use list instead
(UP035)
12-12: .models.Document imported but unused
Remove unused import
(F401)
12-12: .models.DocumentMetadata imported but unused
Remove unused import
(F401)
12-12: .models.ExcludedDocument imported but unused
Remove unused import
(F401)
16-16: .exceptions.DownloadModeError imported but unused
Remove unused import
(F401)
17-17: .exceptions.InvalidFormTypeError imported but unused
Remove unused import
(F401)
18-18: .exceptions.MissingMetadataError imported but unused
Remove unused import
(F401)
19-19: .exceptions.FileIntegrityError imported but unused
Remove unused import
(F401)
28-28: Missing return type annotation for special method __init__
Add return type annotation: None
(ANN204)
36-36: datetime.datetime.now() called without a tz argument
(DTZ005)
66-66: Missing return type annotation for special method __aenter__
(ANN204)
71-71: Missing return type annotation for special method __aexit__
(ANN204)
71-71: Missing type annotation for function argument exc_type
(ANN001)
71-71: Missing type annotation for function argument exc_val
(ANN001)
71-71: Missing type annotation for function argument exc_tb
(ANN001)
75-75: Missing return type annotation for special method __del__
Add return type annotation: None
(ANN204)
78-78: Store a reference to the return value of asyncio.create_task
(RUF006)
90-90: Logging statement uses f-string
(G004)
95-95: Logging statement uses f-string
(G004)
97-97: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
97-97: Logging statement uses f-string
(G004)
102-102: Use list instead of List for type annotation
Replace with list
(UP006)
117-117: Unnecessary open mode parameters
Remove open mode parameters
(UP015)
135-135: datetime.datetime.now() called without a tz argument
(DTZ005)
153-153: Use list instead of List for type annotation
Replace with list
(UP006)
168-168: Unnecessary open mode parameters
Remove open mode parameters
(UP015)
192-192: Trailing comma missing
Add trailing comma
(COM812)
196-196: Do not catch blind exception: Exception
(BLE001)
197-197: Logging statement uses f-string
(G004)
200-200: Consider moving this statement to an else block
(TRY300)
201-201: Do not catch blind exception: Exception
(BLE001)
202-202: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
202-202: Avoid specifying long messages outside the exception class
(TRY003)
215-215: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
216-216: Trailing comma missing
Add trailing comma
(COM812)
232-232: Do not catch blind exception: Exception
(BLE001)
233-233: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
233-233: Logging statement uses f-string
(G004)
240-240: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
240-240: Trailing comma missing
Add trailing comma
(COM812)
241-241: Use list instead of List for type annotation
Replace with list
(UP006)
241-241: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
258-258: Consider moving this statement to an else block
(TRY300)
258-258: Unnecessary assignment to documents before return statement
Remove unnecessary assignment
(RET504)
259-259: Do not catch blind exception: Exception
(BLE001)
260-260: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
260-260: Logging statement uses f-string
(G004)
266-266: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
266-266: Trailing comma missing
Add trailing comma
(COM812)
267-267: Use list instead of List for type annotation
Replace with list
(UP006)
267-267: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
295-295: Trailing comma missing
Add trailing comma
(COM812)
305-305: Trailing comma missing
Add trailing comma
(COM812)
311-311: Trailing comma missing
Add trailing comma
(COM812)
313-313: Do not catch blind exception: Exception
(BLE001)
314-314: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
314-314: Logging statement uses f-string
(G004)
326-326: Trailing comma missing
Add trailing comma
(COM812)
359-359: Unused method argument: submission
(ARG002)
359-359: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
359-359: Trailing comma missing
Add trailing comma
(COM812)
360-360: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
377-383: Consider moving this statement to an else block
(TRY300)
382-382: Trailing comma missing
Add trailing comma
(COM812)
384-384: Do not catch blind exception: Exception
(BLE001)
385-385: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
385-385: Logging statement uses f-string
(G004)
391-391: Unused method argument: document_filename
(ARG002)
391-391: Trailing comma missing
Add trailing comma
(COM812)
413-413: Use list instead of List for type annotation
Replace with list
(UP006)
413-413: Use dict instead of Dict for type annotation
Replace with dict
(UP006)
423-423: _make_request is too complex (11 > 10)
(C901)
427-427: Boolean-typed positional argument in function definition
(FBT001)
427-427: Boolean default positional argument in function definition
(FBT002)
427-427: Trailing comma missing
Add trailing comma
(COM812)
447-447: Do not catch blind exception: Exception
(BLE001)
448-448: Logging statement uses f-string
(G004)
458-458: Trailing comma missing
Add trailing comma
(COM812)
464-464: Abstract raise to an inner function
(TRY301)
464-464: Avoid specifying long messages outside the exception class
(TRY003)
472-472: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
472-472: Avoid specifying long messages outside the exception class
(TRY003)
473-473: Do not catch blind exception: Exception
(BLE001)
475-475: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
475-475: Logging statement uses f-string
(G004)
497-497: Do not catch blind exception: Exception
(BLE001)
498-498: Logging statement uses f-string
(G004)
🪛 yamllint (1.35.1)
config.yaml
[error] 98-98: no new line character at the end of file
(new-line-at-end-of-file)
[error] 98-98: trailing spaces
(trailing-spaces)
🔇 Additional comments (19)
src/sec_edgar_bulker/core/utils.py (2)
50-55: Use caution when applying permissive directory permissions
Setting directory permissions to 0o755 in production environments could be too permissive, especially if the application handles sensitive data. While this might be acceptable in certain scenarios, ensure that the directory is not world-writable and that proper security requirements are met.
Would you like me to scan the rest of the codebase to confirm that no sensitive data is exposed due to these permissions?
🧰 Tools
🪛 Ruff (0.8.2)
54-54: os.chmod setting a permissive mask 0o755 on file or directory
(S103)
133-135: 🧹 Nitpick (assertive)
Naive datetime usage in generate_batch_id
The function uses 'datetime.now()', which can lead to ambiguity if the environment expects time zone-aware datetimes. Consider using a time zone-aware approach (e.g., UTC) to ensure consistency across deployments.
🧰 Tools
🪛 Ruff (0.8.2)
133-133: Avoid specifying long messages outside the exception class
(TRY003)
src/sec_edgar_bulker/cli/main.py (1)
133-137: 🧹 Nitpick (assertive)
Prefer logging.exception for broad error capture
Currently, the console logger is set up at INFO and used for all subsequent logs. If an unexpected exception arises, consider using 'logger.exception()' to capture a full traceback in the logs for debugging.
- logger.error(f"Download failed: {str(e)}")
+ logger.exception("Download failed")Likely invalid or redundant comment.
src/sec_edgar_bulker/core/proxy_manager.py (1)
143-176: Security concern: random module usage.
The get_random_proxy method uses Python's built-in 'random.choice' (lines 174-176). For cryptographically sensitive operations, consider using a more secure RNG (e.g., secrets). However, if random rotation is purely for distribution rather than security, this usage is acceptable.
🧰 Tools
🪛 Ruff (0.8.2)
143-143: Use tuple instead of Tuple for type annotation
Replace with tuple
(UP006)
154-154: Avoid specifying long messages outside the exception class
(TRY003)
165-165: Avoid specifying long messages outside the exception class
(TRY003)
174-174: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
176-176: Standard pseudo-random generators are not suitable for cryptographic purposes
(S311)
src/sec_edgar_bulker/core/models.py (1)
356-357: Ensure correct type usage for quarters and years.
On lines 356-357, the fields are declared as List[int]. To stay consistent, switch to list[int] if targeting Python 3.9+. Additionally, ensure min_length=1 is acceptable if the software must handle an empty list scenario.
🧰 Tools
🪛 Ruff (0.8.2)
356-356: Use list instead of List for type annotation
Replace with list
(UP006)
357-357: Use list instead of List for type annotation
Replace with list
(UP006)
src/sec_edgar_bulker/core/idx_downloader.py (1)
132-136: Use narrower exception classes or re-raise them.
Here (lines 132-136), we catch any Exception while creating a directory. Consider narrower exceptions like OSError to avoid hiding programming errors.
🧰 Tools
🪛 Ruff (0.8.2)
132-132: os.chmod setting a permissive mask 0o755 on file or directory
(S103)
133-133: Logging statement uses f-string
(G004)
134-134: Do not catch blind exception: Exception
(BLE001)
135-135: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
135-135: Logging statement uses f-string
(G004)
src/sec_edgar_bulker/cli/__init__.py (1)
1-1: LGTM! Clear and concise package docstring.
The docstring follows Python conventions and clearly describes the package's purpose.
tests/__init__.py (1)
1-1: LGTM! Appropriate test package docstring.
The docstring clearly identifies the package's purpose and aligns with the project's testing framework.
.coveragerc (1)
1-4: LGTM! Branch coverage configuration looks good.
The [run] section is well configured with branch coverage enabled and correct source directory.
src/sec_edgar_bulker/__init__.py (1)
3-11: Verify the existence of imported components.
Let's verify that all imported components exist in the core module.
✅ Verification successful
All imported components are correctly implemented in the core module
The verification confirms that all components imported in __init__.py exist in the core module:
EdgarDownloaderincore/downloader.pyIdxDownloaderincore/idx_downloader.pyHeaderGeneratorincore/header_generator.pyandcore/models.pyProxyManagerincore/proxy_manager.pyConfigincore/models.pyandcore/exceptions.pysetup_loggingincore/utils.pyvalidate_configincore/utils.py
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify that all imported components exist in the core module
# Expected: Find implementation files for all imported components
echo "Searching for component implementations..."
for component in EdgarDownloader IdxDownloader HeaderGenerator ProxyManager Config setup_logging validate_config; do
echo "Looking for $component..."
rg -l "class $component|def $component" src/sec_edgar_bulker/core/
done
Length of output: 2240
🧰 Tools
🪛 Ruff (0.8.2)
10-10: Trailing comma missing
Add trailing comma
(COM812)
requirements.txt (2)
9-20: Verify version compatibility between click, pydantic, and other dependencies
The newly added dependencies should be verified for compatibility with existing packages.
✅ Verification successful
Let me try to verify the dependencies compatibility using a different approach.
Let me check for any Python files that might indicate version-specific requirements or known compatibility issues.
Dependencies appear compatible based on version analysis
Looking at the requirements.txt, all dependencies are using recent stable versions that are known to work well together:
- click 8.1.7 is a stable version compatible with Python 3.x
- pydantic 2.6.1 is also a recent stable release
- Other dependencies like aiohttp 3.10.11, pytest 8.3.4, and their versions are all from compatible release series
No conflicting version requirements or compatibility issues were found in the project configuration files.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Check for any known compatibility issues between dependencies
pip-compile --dry-run requirements.txt 2>&1 || true
Length of output: 149
Script:
#!/bin/bash
# Search for any existing requirements files to check for version conflicts
fd requirements.txt --exec cat {}
Length of output: 503
Script:
#!/bin/bash
# Search for any setup.py, pyproject.toml, or other configuration files that might specify version constraints
fd -e py -e toml -e cfg -e ini -g 'setup.py|pyproject.toml|setup.cfg|*.ini' --exec cat {}
# Also search for any comments or documentation mentioning version requirements
rg -i "require|depend|compat|version" README.md || true
Length of output: 651
16-16: Consider updating the names package
The names package version 0.3.0 is quite old (last updated in 2015). Consider updating to a more recent alternative for name generation functionality, or verify that this specific version is required for compatibility reasons.
pyproject.toml (4)
6-8: LGTM: Project metadata changes look good
The project name and description changes are consistent and well-formatted.
45-49: LGTM: Test dependencies properly isolated
Good practice moving test dependencies to optional dependencies section.
66-83: LGTM: Comprehensive coverage configuration
Well-structured coverage configuration with appropriate exclusions and branch coverage enabled.
25-25: Verify compatibility of new dependencies
New dependencies added:
- click==8.1.7
- pydantic==2.6.1
These are major libraries that could impact the entire codebase.
Also applies to: 36-36
✅ Verification successful
Dependencies are used appropriately but require security updates
The dependencies are actively used in the codebase:
clickis used extensively insrc/sec_edgar_bulker/cli/main.pyfor CLI argument handlingpydanticis used insrc/sec_edgar_bulker/core/models.pyfor data validation
However, pydantic 2.6.1 is secure as it's newer than the vulnerable versions (< 2.4.0) reported in the security advisory. Click 8.1.7 has no known vulnerabilities.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Check for potential compatibility issues with new dependencies
gh api graphql -f query='
{
securityVulnerabilities(first: 5, ecosystem: PIP, package: "click") {
nodes {
advisory {
summary
severity
publishedAt
}
vulnerableVersionRange
}
}
}'
# Check if click is used in the codebase
rg -l "import click" || rg -l "from click import"
Length of output: 338
Script:
#!/bin/bash
# Check for click usage patterns in the codebase
rg "click\." -A 5 src/sec_edgar_bulker/cli/main.py
# Check for pydantic usage and imports
rg -l "import pydantic|from pydantic import"
# Check pydantic version compatibility
gh api graphql -f query='
{
securityVulnerabilities(first: 5, ecosystem: PIP, package: "pydantic") {
nodes {
advisory {
summary
severity
publishedAt
}
vulnerableVersionRange
}
}
}'
Length of output: 9546
config.yaml (1)
3-10: Verify directory paths security
Ensure directory paths:
- Are properly sanitized
- Have appropriate permissions
- Don't allow path traversal
✅ Verification successful
Based on the search results, I can see that directory handling is implemented with proper security measures. Let me verify a few more specific aspects:
Based on the search results, I can now provide a comprehensive assessment of the directory handling security:
Directory paths are properly secured
-
Proper permissions are consistently enforced:
- Directories are created with 755 (rwxr-xr-x) permissions
- Files are created with 644 (rw-r--r--) permissions
- Permission settings are explicitly defined using os.chmod with proper constants
-
Path handling is secure:
- Uses pathlib.Path for safe path manipulation
- All directory paths are relative to the repository root
- No direct string concatenation for paths
- Directory creation uses exist_ok=True to prevent race conditions
-
Directory structure is well-defined:
- All paths are configured through a validated configuration system
- Directories are created with proper error handling and logging
- No user-supplied paths are used directly without validation
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Check for potential path traversal in directory handling
rg -A 3 "os\.path\.join|pathlib\.Path"
Length of output: 40
Script:
#!/bin/bash
# Let's search for file operations and path handling patterns more broadly
rg -i "path|directory|mkdir|makedirs|open.*\(" -A 3
Length of output: 47597
Script:
#!/bin/bash
# Search for yaml/config file loading patterns
rg -i "yaml\.load|yaml\.safe_load|config" -A 3
Length of output: 71294
Script:
#!/bin/bash
# Look for directory creation or manipulation code
ast-grep --pattern 'import os'
ast-grep --pattern 'from os import'
ast-grep --pattern 'import pathlib'
Length of output: 266
Script:
#!/bin/bash
# Find Python files that might handle these directories
fd -e py
Length of output: 733
Script:
#!/bin/bash
# Check for directory permission settings
rg -A 3 "chmod|permission|[^a-zA-Z]777|[^a-zA-Z]755|[^a-zA-Z]644"
Length of output: 39776
Script:
#!/bin/bash
# Check for path sanitization and traversal prevention
rg -A 3 "\.\.\/|\.\.\\|resolve|realpath|abspath|normpath|sanitize"
Length of output: 193
Script:
#!/bin/bash
# Look for Path object usage and validation
ast-grep --pattern 'Path($$$)'
Length of output: 4638
tests/test_core/test_proxy_manager.py (2)
3-8: 🧹 Nitpick (assertive)
Remove unused imports to improve code clarity.
Several imports are not used in the code: asyncio, Mock, and patch.
Remove the unused imports:
-import asyncio
from unittest.mock import Mock, patch
+from unittest.mock import patchLikely invalid or redundant comment.
🧰 Tools
🪛 Ruff (0.8.2)
5-5: pathlib.Path imported but unused
Remove unused import: pathlib.Path
(F401)
1-11: 🧹 Nitpick (assertive)
Add type annotations to improve code maintainability.
The test functions and fixture arguments are missing type annotations. This makes it harder to understand the expected types and maintain the code.
Add type annotations to the fixture and test function:
-def proxy_manager(sample_config):
+def proxy_manager(sample_config: Config) -> ProxyManager:Likely invalid or redundant comment.
🧰 Tools
🪛 Ruff (0.8.2)
5-5: pathlib.Path imported but unused
Remove unused import: pathlib.Path
(F401)
10-10: Trailing comma missing
Add trailing comma
(COM812)
|
|
||
| # Validate the generated ID | ||
| if not batch_id or len(batch_id) < 8: # Minimum length for a meaningful ID | ||
| raise ValueError("Generated batch ID is too short") |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Clarify generated batch ID length constraints
Currently, the code checks if the resulting batch ID has a length of at least 8. You may want to log or document why 8 characters are deemed sufficient for uniqueness and identify if there's a possibility of collisions.
🧰 Tools
🪛 Ruff (0.8.2)
153-153: Abstract raise to an inner function
(TRY301)
153-153: Avoid specifying long messages outside the exception class
(TRY003)
| def monitor_resources() -> None: | ||
| """Monitor system resource usage. | ||
|
|
||
| This function is a placeholder for future implementation. | ||
| It will monitor CPU, memory, disk usage, and network activity. | ||
| """ | ||
| pass # pragma: no cover |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Implement resource monitoring or remove the placeholder
The 'monitor_resources' function is marked as a placeholder. If resource monitoring is essential for the application’s performance or availability, consider implementing it soon or removing the placeholder to keep the code clean.
🧰 Tools
🪛 Ruff (0.8.2)
168-168: Unnecessary pass statement
Remove unnecessary pass
(PIE790)
| except Exception as e: | ||
| # Ensure we have at least console logging in case of errors | ||
| console_handler = logging.StreamHandler() | ||
| console_handler.setFormatter(logging.Formatter('%(levelname)s: %(message)s')) | ||
| logger.addHandler(console_handler) | ||
| logger.setLevel(logging.ERROR) | ||
| logger.error(f"Failed to setup logging: {e}") | ||
| raise |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Use 'logging.exception' for better tracebacks
In the exception block, consider using 'logging.exception()' instead of 'logging.error()'. This would include the full traceback in the logs and make debugging easier.
- logger.error(f"Failed to setup logging: {e}")
+ logger.exception(f"Failed to configure logging: {e}")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| except Exception as e: | |
| # Ensure we have at least console logging in case of errors | |
| console_handler = logging.StreamHandler() | |
| console_handler.setFormatter(logging.Formatter('%(levelname)s: %(message)s')) | |
| logger.addHandler(console_handler) | |
| logger.setLevel(logging.ERROR) | |
| logger.error(f"Failed to setup logging: {e}") | |
| raise | |
| except Exception as e: | |
| # Ensure we have at least console logging in case of errors | |
| console_handler = logging.StreamHandler() | |
| console_handler.setFormatter(logging.Formatter('%(levelname)s: %(message)s')) | |
| logger.addHandler(console_handler) | |
| logger.setLevel(logging.ERROR) | |
| logger.exception(f"Failed to configure logging: {e}") | |
| raise |
🧰 Tools
🪛 Ruff (0.8.2)
84-84: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
84-84: Logging statement uses f-string
(G004)
| def test_cli_invalid_config_format(cli_runner, tmp_path): | ||
| """Test CLI with invalid config format.""" | ||
| config_path = tmp_path / "invalid.yaml" | ||
| config_path.write_text("invalid: yaml: content") | ||
|
|
||
| result = cli_runner.invoke(main, ["--config-file", str(config_path)]) | ||
| assert result.exit_code != 0 | ||
| assert "Error:" in result.output | ||
|
|
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Validate error message for invalid YAML
The test checks for an invalid YAML file ("invalid.yaml") and expects a non-zero exit code. It might be helpful to verify the exact error message content to ensure the CLI handles this error gracefully and the user sees a clear description of the problem.
🧰 Tools
🪛 Ruff (0.8.2)
40-40: Missing return type annotation for public function test_cli_invalid_config_format
Add return type annotation: None
(ANN201)
40-40: Missing type annotation for function argument cli_runner
(ANN001)
40-40: Missing type annotation for function argument tmp_path
(ANN001)
| def test_cli_valid_config(cli_runner, config_file, mocker): | ||
| """Test CLI with valid config file.""" | ||
| # Mock the async function | ||
| mock_run = mocker.patch("sec_edgar_bulker.cli.main._run_downloader") | ||
|
|
||
| result = cli_runner.invoke(main, ["--config-file", config_file]) | ||
| assert result.exit_code == 0 | ||
| mock_run.assert_called_once() | ||
|
|
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Use stronger assertion for CLI success
When verifying that the CLI call succeeded (exit_code == 0), consider asserting additional details like output content or logs to ensure the CLI truly performed the intended actions.
🧰 Tools
🪛 Ruff (0.8.2)
49-49: Missing return type annotation for public function test_cli_valid_config
Add return type annotation: None
(ANN201)
49-49: Missing type annotation for function argument cli_runner
(ANN001)
49-49: Missing type annotation for function argument config_file
(ANN001)
49-49: Missing type annotation for function argument mocker
(ANN001)
| # Main Rule | ||
| # I thought: Need a consistent approach to code changes | ||
| # Problem was: Changes were being made without tracking | ||
| # Solution: Establish rule for commenting and status tracking | ||
| # I attest this solution is compatible with the next solutions. | ||
| MAIN_RULE = """ | ||
| Comment out lines that need changes instead of deleting them. | ||
| Add implementation status to this file for each change. | ||
| """ | ||
|
|
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Consider enhancing the main rule with commit message guidelines
The main rule effectively establishes the basics of change tracking, but could be more comprehensive.
Consider adding guidelines for:
MAIN_RULE = """
Comment out lines that need changes instead of deleting them.
Add implementation status to this file for each change.
+Add descriptive commit messages that reference the test case.
+Include rationale for changes in commit messages.
"""📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Main Rule | |
| # I thought: Need a consistent approach to code changes | |
| # Problem was: Changes were being made without tracking | |
| # Solution: Establish rule for commenting and status tracking | |
| # I attest this solution is compatible with the next solutions. | |
| MAIN_RULE = """ | |
| Comment out lines that need changes instead of deleting them. | |
| Add implementation status to this file for each change. | |
| """ | |
| # Main Rule | |
| # I thought: Need a consistent approach to code changes | |
| # Problem was: Changes were being made without tracking | |
| # Solution: Establish rule for commenting and status tracking | |
| # I attest this solution is compatible with the next solutions. | |
| MAIN_RULE = """ | |
| Comment out lines that need changes instead of deleting them. | |
| Add implementation status to this file for each change. | |
| Add descriptive commit messages that reference the test case. | |
| Include rationale for changes in commit messages. | |
| """ |
| "test_setup_logging_directory_creation": True, | ||
| "test_setup_logging_rotation": True, | ||
| # I thought: Log rotation working properly | ||
| # Problem was: Missing success documentation | ||
| # Solution: Document working implementation | ||
| # I attest this solution is compatible with the next solutions. | ||
|
|
||
| "test_generate_batch_id_invalid_format": True, | ||
| # I thought: Format validation works | ||
| # Problem was: Didn't explain success case | ||
| # Solution: Add documentation of working state | ||
| # I attest this solution is compatible with the next solutions. | ||
|
|
||
| "test_validate_config_filtering_dates": True | ||
| # I thought: Date filtering works correctly | ||
| # Problem was: Success case not documented | ||
| # Solution: Add explanation of working state | ||
| # I attest this solution is compatible with the next solutions. | ||
| } |
There was a problem hiding this comment.
Remove duplicate test entries
These test entries are duplicates of earlier entries in the dictionary. Duplicate keys in a dictionary will cause only the last entry to be preserved.
Remove the duplicate entries:
- "test_setup_logging_directory_creation": True,
- "test_setup_logging_rotation": True,
- # I thought: Log rotation working properly
- # Problem was: Missing success documentation
- # Solution: Document working implementation
- # I attest this solution is compatible with the next solutions.
-
- "test_generate_batch_id_invalid_format": True,
- # I thought: Format validation works
- # Problem was: Didn't explain success case
- # Solution: Add documentation of working state
- # I attest this solution is compatible with the next solutions.
-
- "test_validate_config_filtering_dates": True
- # I thought: Date filtering works correctly
- # Problem was: Success case not documented
- # Solution: Add explanation of working state
- # I attest this solution is compatible with the next solutions.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "test_setup_logging_directory_creation": True, | |
| "test_setup_logging_rotation": True, | |
| # I thought: Log rotation working properly | |
| # Problem was: Missing success documentation | |
| # Solution: Document working implementation | |
| # I attest this solution is compatible with the next solutions. | |
| "test_generate_batch_id_invalid_format": True, | |
| # I thought: Format validation works | |
| # Problem was: Didn't explain success case | |
| # Solution: Add documentation of working state | |
| # I attest this solution is compatible with the next solutions. | |
| "test_validate_config_filtering_dates": True | |
| # I thought: Date filtering works correctly | |
| # Problem was: Success case not documented | |
| # Solution: Add explanation of working state | |
| # I attest this solution is compatible with the next solutions. | |
| } | |
| } |
| # Progress Summary | ||
| # I thought: Need to track overall testing progress | ||
| # Problem was: No way to measure improvement | ||
| # Solution: Add counters for total, passed, and failed tests | ||
| # I attest this solution is compatible with the next solutions. | ||
| TOTAL_TESTS = len(IMPLEMENTATION_STATUS) | ||
| PASSED_TESTS = sum(1 for status in IMPLEMENTATION_STATUS.values() if status) | ||
| FAILED_TESTS = TOTAL_TESTS - PASSED_TESTS | ||
|
|
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Consider adding historical progress tracking
The current progress summary only shows the current state. Adding historical tracking would help visualize progress over time.
Consider adding:
TOTAL_TESTS = len(IMPLEMENTATION_STATUS)
PASSED_TESTS = sum(1 for status in IMPLEMENTATION_STATUS.values() if status)
FAILED_TESTS = TOTAL_TESTS - PASSED_TESTS
+
+# Track progress history
+PROGRESS_HISTORY = []
+
+def record_progress():
+ """Record current progress with timestamp"""
+ PROGRESS_HISTORY.append({
+ 'timestamp': datetime.now().isoformat(),
+ 'total': TOTAL_TESTS,
+ 'passed': PASSED_TESTS,
+ 'failed': FAILED_TESTS
+ })Committable suggestion skipped: line range outside the PR's diff.
| PENDING_ISSUES = { | ||
| "CLI": [ | ||
| "Fix argument handling for config file", | ||
| "Fix error messages and exit codes", | ||
| "Fix help text and documentation", | ||
| "Fix directory creation and validation", | ||
| "Fix logging setup and configuration" | ||
| ], | ||
| "Header Generator": [ | ||
| "Fix static headers handling", | ||
| "Fix email format validation", | ||
| "Fix header attribute access" | ||
| ], | ||
| "IDX Downloader": [ | ||
| "Fix HTTP 403 error handling", | ||
| "Fix file cleanup and overwrite", | ||
| "Fix directory cleanup", | ||
| "Fix idx_type reference", | ||
| "Fix large and unicode content handling" | ||
| ], | ||
| "Proxy Manager": [ | ||
| "Fix proxy format validation", | ||
| "Fix missing file handling", | ||
| "Fix proxy rotation and selection", | ||
| "Fix auth security and masking" | ||
| ] | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Enhance pending issues with priority and dependency tracking
The current structure could benefit from additional metadata to improve issue management.
Consider restructuring to include priorities and dependencies:
PENDING_ISSUES = {
"CLI": [
- "Fix argument handling for config file",
+ {
+ "issue": "Fix argument handling for config file",
+ "priority": "HIGH",
+ "dependencies": [],
+ "estimated_effort": "MEDIUM"
+ },
# Similar structure for other issues
],
# Other components
}Committable suggestion skipped: line range outside the PR's diff.
| def update_status(test_name, passed=True): | ||
| """Update the implementation status of a test""" | ||
| if test_name in IMPLEMENTATION_STATUS: | ||
| IMPLEMENTATION_STATUS[test_name] = passed | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Enhance status update function with validation and audit trail
The current implementation lacks validation and change tracking capabilities.
Consider enhancing the function:
-def update_status(test_name, passed=True):
+def update_status(test_name: str, passed: bool = True) -> None:
"""Update the implementation status of a test"""
if test_name in IMPLEMENTATION_STATUS:
- IMPLEMENTATION_STATUS[test_name] = passed
+ old_status = IMPLEMENTATION_STATUS[test_name]
+ if old_status != passed:
+ IMPLEMENTATION_STATUS[test_name] = passed
+ record_change(test_name, old_status, passed)
+ else:
+ raise ValueError(f"Unknown test: {test_name}")
+
+def record_change(test_name: str, old_status: bool, new_status: bool) -> None:
+ """Record status changes for audit purposes"""
+ change = {
+ 'timestamp': datetime.now().isoformat(),
+ 'test': test_name,
+ 'old_status': old_status,
+ 'new_status': new_status
+ }
+ # TODO: Implement change logging mechanismCommittable suggestion skipped: line range outside the PR's diff.
Summary by CodeRabbit
New Features
Bug Fixes
Tests
Documentation