fix: add support for pyproject.toml and env.yml/env.yaml files #550
Conversation
|
Please note that this PR in the safety_schemas repo is needed for this to work: https://github.com/pyupio/safety_schemas/pull/1 |
WalkthroughThe changes enhance the functionality of scanning commands in the Changes
Possibly related PRs
Suggested labels
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (5)
- pyproject.toml (1 hunks)
- safety/scan/command.py (2 hunks)
- safety/scan/ecosystems/python/dependencies.py (3 hunks)
- safety/scan/finder/file_finder.py (3 hunks)
- safety/scan/finder/handlers.py (3 hunks)
Additional context used
Ruff
safety/scan/finder/handlers.py
5-5:
typing.Tupleimported but unusedRemove unused import:
typing.Tuple(F401)
safety/scan/finder/file_finder.py
11-11:
.handlers.PyProjectTomlHandlerimported but unusedRemove unused import:
.handlers.PyProjectTomlHandler(F401)
161-161: Use
enumerate()for index variablelevelinforloop(SIM113)
Additional comments not posted (9)
safety/scan/finder/handlers.py (4)
112-113: LGTM!The empty implementation of the
download_required_assetsmethod is appropriate since there are no required assets to download for Safety project files, as indicated by the docstring.
116-162: LGTM!The implementation of the
PyProjectTomlHandlerclass looks good:
- It correctly extends the
FileHandlerclass and sets the ecosystem toEcosystem.PYPROJECT_TOML.- The
download_required_assetsmethod fetches both the full and partial Safety databases, consistent with thePythonFileHandlerclass.- The
can_handlemethod correctly checks for thepyproject.tomlfile name and returns the correspondingFileType.- The
handlemethod reads thepyproject.tomlfile using thetomllibrary and extracts dependencies from the relevant sections.- The extracted dependencies are returned as a set, which is an appropriate data structure for unique dependencies.
Great job!
177-177: LGTM!Adding the
PyProjectTomlHandlerto theECOSYSTEM_HANDLER_MAPPINGdictionary is necessary and correctly implemented. This associates the handler with the correspondingEcosystem.PYPROJECT_TOMLecosystem.
Line range hint
1-179: Overall, the changes in this file look great!The introduction of the
PyProjectTomlHandlerclass is a valuable addition to support handlingpyproject.tomlfiles. The implementation is consistent with the existingFileHandlerclasses and follows the necessary conventions. The class correctly extends theFileHandlerclass and implements the required methods such asdownload_required_assets,can_handle, andhandleappropriately.The integration of the
PyProjectTomlHandlerwith the existing codebase is smooth, and the class is correctly added to theECOSYSTEM_HANDLER_MAPPINGdictionary to associate it with the corresponding ecosystem.Apart from the minor issues mentioned in the previous comments, the overall changes in this file are well-structured and contribute positively to the project's functionality.
Great work!
Tools
Ruff
167-167: Redefinition of unused
download_required_assetsfrom line 121(F811)
safety/scan/finder/file_finder.py (1)
153-160: LGTM!The new code block correctly handles the special files
pyproject.toml,env.yml, andenv.yamlby adding them to thefilesdictionary under their respective file types. This enhances the functionality of the file finder to support these important configuration files.safety/scan/ecosystems/python/dependencies.py (2)
273-308: LGTM!The
read_pyproject_toml_dependenciesfunction correctly reads dependencies from apyproject.tomlfile and yieldsPythonDependencyobjects. It handles both versioned and unversioned dependencies in the relevant sections of the TOML file.The implementation integrates well with the existing
PythonDependencymodel and thePythonSpecificationclass, enabling support forpyproject.tomlfiles in the project.
330-331: LGTM!The changes to the
get_dependenciesfunction correctly integrate the handling ofFileType.PYPROJECT_TOMLfiles by calling the newly introducedread_pyproject_toml_dependenciesfunction.This modification ensures that dependencies from
pyproject.tomlfiles are properly processed and returned asPythonDependencyobjects, seamlessly integrating with the existing dependency resolution logic.safety/scan/command.py (2)
48-48: LGTM!The new member
PYPROJECT_TOMLis added correctly to theScannableEcosystemsenumeration to support scanning projects defined by apyproject.tomlfile.
Line range hint
1-1: Skipped reviewingsystem_scanfunction.The
system_scanfunction is not changed in the provided code, so there is nothing to review.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Outside diff range and nitpick comments (3)
safety/scan/finder/handlers.py (3)
112-113: Consider providing a meaningful implementation or documentation.In the
SafetyProjectFileHandler, thedownload_required_assetsmethod currently contains only apassstatement. If no assets are required, it's a good practice to document this explicitly.Consider adding a docstring:
def download_required_assets(self, session): + """ + No required assets to download for Safety project files. + """ pass
138-139: Remove debug print statement incan_handlemethod.The
print("recognized")statement is likely used for debugging purposes. Consider removing it or replacing it with appropriate logging.Apply this diff to remove the print statement:
def can_handle(self, root: str, file_name: str, include_files: Dict[FileType, List[Path]]) -> Optional[FileType]: if file_name == 'pyproject.toml': - print("recognized") return FileType.PYPROJECT_TOML return None
146-146: Remove debug print statement inhandlemethod.The
print("printing data", data)statement may not be suitable for production code. Consider removing it or using a logging framework.Apply this diff to remove the print statement:
with open(file_path, 'r') as file: data = toml.load(file) - print("printing data", data) dependencies = set()
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
- safety/scan/finder/handlers.py (3 hunks)
🧰 Additional context used
🔇 Additional comments (2)
safety/scan/finder/handlers.py (2)
55-55:⚠️ Potential issueCorrect the use of
NotImplementedErrorin abstract method.In the
FileHandlerclass, thedownload_required_assetsmethod should raiseNotImplementedErrorinstead of returning it. This ensures that subclasses are properly notified to implement this method.Apply this diff to fix the issue:
def download_required_assets(self, session): """ Abstract method to download required assets for handling files. Should be implemented by subclasses. Args: session: The session object for making network requests. - Returns: - Dict[str, str]: A dictionary of downloaded assets. """ - return NotImplementedError(NOT_IMPLEMENTED) + raise NotImplementedError(NOT_IMPLEMENTED)Likely invalid or redundant comment.
5-5: Verify necessity of all imported types fromtypingmodule.Ensure that all imported types (
Dict,List,Optional,Set) are used in the code. Unused imports can be removed to keep the code clean.Run the following script to check for unused imports:
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Outside diff range and nitpick comments (3)
safety/scan/finder/handlers.py (1)
Line range hint
55-65: Update docstring to match implementation.The docstring still indicates a
Dict[str, str]return type, but the method signature no longer specifies a return type and raisesNotImplementedError. Please update the docstring to reflect the current implementation.Apply this diff to fix the docstring:
def download_required_assets(self, session): """ Abstract method to download required assets for handling files. Should be implemented by subclasses. Args: session: The session object for making network requests. - - Returns: - Dict[str, str]: A dictionary of downloaded assets. """ return NotImplementedError(NOT_IMPLEMENTED)safety/scan/finder/file_finder.py (2)
154-160: Consider refactoring special files handling into dedicated handlers.The current implementation hardcodes special files processing separately from the handler-based approach. This creates maintenance overhead and potential inconsistencies. Consider:
- Creating dedicated handlers for these file types
- Integrating with the existing handler mechanism
- Adding validation for file contents
Example implementation approach:
class PyProjectTomlHandler(FileHandler): def can_handle(self, root: str, filename: str, include_files: Optional[Dict]) -> Optional[FileType]: if filename == 'pyproject.toml': return FileType('pyproject.toml') return None class EnvYamlHandler(FileHandler): def can_handle(self, root: str, filename: str, include_files: Optional[Dict]) -> Optional[FileType]: if filename in ['env.yml', 'env.yaml']: return FileType(filename) return NoneThen register these handlers in
ECOSYSTEM_HANDLER_MAPPINGinstead of using the hardcoded approach.
161-161: Remove or properly implement the level counter.The
levelcounter appears to be unused. Either:
- Remove it if not needed
- Use it meaningfully for tracking directory depth
- If tracking iteration count is needed, use
enumerate()If you need to track iterations, consider:
- level += 1Or if you need both the index and iteration:
for level, (root, dirs, filenames) in enumerate(os.walk(dir_path)): # ... rest of the code🧰 Tools
🪛 Ruff
161-161: Use
enumerate()for index variablelevelinforloop(SIM113)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (3)
- safety/scan/ecosystems/python/dependencies.py (3 hunks)
- safety/scan/finder/file_finder.py (2 hunks)
- safety/scan/finder/handlers.py (3 hunks)
🧰 Additional context used
🪛 Ruff
safety/scan/finder/file_finder.py
161-161: Use
enumerate()for index variablelevelinforloop(SIM113)
safety/scan/finder/handlers.py
5-5:
typing.Setimported but unusedRemove unused import:
typing.Set(F401)
6-6:
tomlimported but unusedRemove unused import:
toml(F401)
🔇 Additional comments (5)
safety/scan/finder/handlers.py (1)
Line range hint
116-120: Add PyProjectTomlHandler to ecosystem mapping.Given the PR objective to add support for pyproject.toml files, the handler for these files should be added to the
ECOSYSTEM_HANDLER_MAPPING. Currently, there's no implementation visible for handling pyproject.toml files.Let's verify if the PyProjectTomlHandler exists in the codebase:
Either:
- Implement a new PyProjectTomlHandler class, or
- Add the existing handler to the mapping:
ECOSYSTEM_HANDLER_MAPPING = MappingProxyType({ Ecosystem.PYTHON: PythonFileHandler, Ecosystem.SAFETY_PROJECT: SafetyProjectFileHandler, + Ecosystem.PYPROJECT_TOML: PyProjectTomlHandler, # Add handler for pyproject.toml })safety/scan/finder/file_finder.py (2)
78-78: Remove debugging print statement.This print statement appears to be for debugging purposes and should be removed before merging.
154-160: Verify integration with safety_schemas.Since this PR depends on changes in the safety_schemas repository (pyupio/safety_schemas#1), we should verify:
- The FileType values match those defined in safety_schemas
- The file content validation is properly integrated
✅ Verification successful
Let me gather more information to verify the FileType implementation and its integration.
Based on the gathered information, I can now generate the final response:
FileType integration with safety_schemas is properly implemented
The implementation correctly integrates with safety_schemas:
- The FileType enum is imported from safety_schemas.models in multiple files
- The package requires safety_schemas>=0.0.4 as a dependency
- The special files ('pyproject.toml', 'env.yml', 'env.yaml') are properly handled using the FileType enum from safety_schemas
- File content validation is handled through the safety_schemas models and validation system
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Check if FileType values in safety_schemas match our usage rg -A 1 "class FileType" rg "FileType\(" # Check for any validation logic for these file types rg -l "pyproject\.toml|env\.ya?ml"Length of output: 331
Script:
#!/bin/bash # Check FileType enum definition and its values ast-grep --pattern 'class FileType' # Check for any validation or schema related code for these files rg -A 5 "def validate|schema" # Look for imports related to safety_schemas rg "from safety_schemas|import safety_schemas"Length of output: 25706
safety/scan/ecosystems/python/dependencies.py (2)
5-5: Importing 'toml' module is appropriateThe addition of
import tomlis necessary for parsingpyproject.tomlfiles.
347-350: Integration with 'pyproject.toml' files is correctly implementedThe
get_dependenciesfunction now correctly handlesFileType.PYPROJECT_TOMLby invokingread_pyproject_toml_dependencies. This enhancement allows the system to parse dependencies frompyproject.tomlfiles.
| dep_name, dep_version = (dep.split("==") + [None])[:2] | ||
| print(dep_name) | ||
| print(dep_version) |
There was a problem hiding this comment.
Improve dependency parsing to handle various version specifiers
The current approach splits the dependency string on '==' to obtain dep_name and dep_version. This may not correctly handle dependencies with different version specifiers (e.g., '>=', '~=', '!=') or additional markers and extras. Consider using a robust parser, such as packaging.requirements.Requirement, to accurately parse dependency strings.
Apply this diff to parse dependencies using the packaging library:
+ from packaging.requirements import Requirement
...
for dep in dependencies:
- dep_name, dep_version = (dep.split("==") + [None])[:2]
- print(dep_name)
- print(dep_version)
+ req = Requirement(dep)
+ dep_name = req.name
+ dep_version = str(req.specifier) if req.specifier else None
...
yield PythonDependency(
name=dep_name,
version=dep_version,
specifications=[
- PythonSpecification(f"{dep_name}=={dep_version}" if dep_version else dep_name, found=file)
+ PythonSpecification(str(req), found=file)
],
...
)Committable suggestion was skipped due to low confidence.
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Outside diff range and nitpick comments (7)
safety/scan/ecosystems/python/dependencies.py (2)
5-5: Consider pinning the toml package versionTo ensure consistent behavior across different environments, consider specifying the required version of
tomlin your project's requirements.
273-320: Consider using established Python packaging librariesThe current implementation could benefit from leveraging established Python packaging libraries:
- Use
poetry.core.packagesfor parsing Poetry dependencies- Use
tomli(PEP 680) instead oftomlfor better TOML parsing- Consider supporting PEP 621 build backend interface for more standardized project metadata handling
This would provide more robust parsing and better compatibility with the Python packaging ecosystem.
safety/scan/command.py (5)
248-248: Remove Unnecessary Blank LineThere is an extra blank line at line 248. Removing it will improve code readability and maintain consistency.
Apply this diff to remove the blank line:
): - if not ctx.obj.metadata.authenticated:
373-373: Remove Unnecessary Empty LineAn unnecessary empty line is present at line 373. Consider removing it to keep the code clean.
Apply this diff:
lines = [] - if spec.remediation.recommended:
445-454: Improve Argument Alignment inprint_summaryCallThe arguments in the
print_summaryfunction call are misaligned, which affects readability. Align the arguments vertically for better clarity.Apply this diff:
print_summary( - console=console, - total_issues_with_duplicates=total_issues_with_duplicates, - total_ignored_issues=total_ignored_issues, - 450~ - project=ctx.obj.project, - 451~ - dependencies_count=count, - 452~ - fixes_count=fixes_count, - 453~ - resolved_vulns_per_fix=total_resolved_vulns, - 454~ - is_detailed_output=detailed_output, - ignored_vulns_data=ignored_vulns_data + console=console, + total_issues_with_duplicates=total_issues_with_duplicates, + total_ignored_issues=total_ignored_issues, + project=ctx.obj.project, + dependencies_count=count, + fixes_count=fixes_count, + resolved_vulns_per_fix=total_resolved_vulns, + is_detailed_output=detailed_output, + ignored_vulns_data=ignored_vulns_data )
Line range hint
798-806: Update Function Docstring for AccuracyThe docstring for
get_vulnerability_summarymentions that it accepts aReportModel, but the function actually processes a dictionary. Ensure that the docstring accurately reflects the function's input parameter.Update the docstring to indicate the correct parameter type:
def get_vulnerability_summary(report: Dict[str, Any]) -> Tuple[int, int]: """ - Summarize vulnerabilities from the given report. + Summarize vulnerabilities from the given report dictionary. Args: - report (ReportModel): The report containing vulnerability data. + report (Dict[str, Any]): The report dictionary containing vulnerability data. Returns: Tuple[int, int]: A tuple containing:
373-373: Consider Combining Consecutive Empty LinesMultiple empty lines have been added consecutively at various points in the code, such as at line 373. While spacing can improve readability, excessive empty lines may have the opposite effect.
Review the code to ensure that only necessary empty lines are included.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (3)
- safety/scan/command.py (4 hunks)
- safety/scan/ecosystems/python/dependencies.py (3 hunks)
- safety/scan/finder/file_finder.py (1 hunks)
🧰 Additional context used
🪛 Ruff
safety/scan/finder/file_finder.py
160-160: Use
enumerate()for index variablelevelinforloop(SIM113)
🔇 Additional comments (2)
safety/scan/ecosystems/python/dependencies.py (2)
342-344: LGTM!The implementation follows the existing pattern and cleanly integrates pyproject.toml support.
291-297:⚠️ Potential issueImprove handling of poetry dependencies
The current implementation doesn't handle complex dependency specifications in poetry dependencies (e.g., version constraints, extras, markers).
Consider using
poetry.core.packagesfor proper parsing:if 'tool' in data and 'poetry' in data['tool'] and 'dependencies' in data['tool']['poetry']: for dep, version in data['tool']['poetry']['dependencies'].items(): - if isinstance(version, str): - dependencies.append(f"{dep}=={version}") - else: - dependencies.append(dep) + if isinstance(version, dict): + # Handle complex version specifications + version_constraint = version.get('version', '') + extras = version.get('extras', []) + markers = version.get('markers', '') + dep_spec = f"{dep}{version_constraint}" + if extras: + dep_spec += f"[{','.join(extras)}]" + if markers: + dep_spec += f"; {markers}" + else: + dep_spec = f"{dep}{version}" + dependencies.append(dep_spec)Likely invalid or redundant comment.
safety/scan/finder/file_finder.py
Outdated
| special_files = {'pyproject.toml', 'env.yml', 'env.yaml'} | ||
| if file_name in special_files: | ||
| file_type = FileType(file_name) | ||
| inspectable_file = Path(root, file_name) | ||
| if file_type.value not in files or not files[file_type.value]: | ||
| files[file_type.value] = set() | ||
| files[file_type.value].add(inspectable_file) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider refactoring special files handling into a dedicated handler.
The current implementation duplicates logic and reduces maintainability. Consider these improvements:
- Create a dedicated
SpecialFileHandlerclass that implements theFileHandlerinterface - Move special files set to a configuration constant
- Add validation of file contents before processing
Example implementation:
# Define in a central configuration
SPECIAL_FILES = frozenset({'pyproject.toml', 'env.yml', 'env.yaml'})
class SpecialFileHandler(FileHandler):
def can_handle(self, root: str, file_name: str, include_files: Optional[Dict[FileType, List[Path]]]) -> Optional[FileType]:
return FileType(file_name) if file_name in SPECIAL_FILES else NoneThis approach would:
- Improve maintainability by centralizing special file definitions
- Follow the existing handler pattern
- Make it easier to add validation logic
- Reduce code duplication
💡 Codebase verification
Based on the search results, I can now generate a final response.
The ecosystem handlers need to be updated for the new file types
The search results reveal that while pyproject.toml and environment files (env.yml, env.yaml) are added to the special files list in file_finder.py, there are gaps in the implementation:
- The
FileTypeenum is used in multiple places butpyproject.tomland environment files are not included in theDEFAULT_FILE_TYPESlist inscan/command.py - While there's a handler for
pyproject.tomlinecosystems/python/dependencies.py(read_pyproject_toml_dependencies), there's no handler for environment files - The supported file types for fixes (
SUPPORTED_FILE_TYPESinsafety.py) need to be updated if these files should support remediation
Required updates:
- Add environment file types to the
DEFAULT_FILE_TYPESlist if they should be scanned by default - Implement dependency reading for environment files
- Update
SUPPORTED_FILE_TYPESif fixes should be supported for these file types
🔗 Analysis chain
Verify related ecosystem handlers are updated.
The implementation adds support for pyproject.toml and environment files, but we should verify that corresponding handlers in the ecosystem mapping can process these files correctly.
Let me try a different search strategy to verify the ecosystem handlers.
Let me try one final search strategy focusing on the FileType enum and its usage.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Search for handlers that might need updates for the new file types
echo "Searching for related ecosystem handlers..."
rg -l "pyproject.toml|env\.ya?ml" --type python
# Search for dependency processing functions
echo "Searching for dependency processing functions..."
ast-grep --pattern 'def read_$_dependencies($$$)'
Length of output: 336
Script:
#!/bin/bash
# Search for files containing ecosystem-related code
echo "Searching for ecosystem-related files..."
fd -e py ecosystem
# Search for files mentioning the new file types
echo "Searching for files mentioning the new file types..."
rg "pyproject\.toml|env\.ya?ml|env\.yml"
# Search for dependency processing related files
echo "Searching for dependency processing related files..."
fd -e py dependency
Length of output: 514
Script:
#!/bin/bash
# Search for FileType enum definition and usage
echo "Searching for FileType enum..."
rg -A 5 "class FileType"
# Search for files that handle file type processing
echo "Searching for file type processing..."
rg -B 2 -A 2 "FileType\."
# Search for any dependency or requirement parsing
echo "Searching for dependency parsing..."
rg -l "parse.*requirements|parse.*dependencies"
Length of output: 4851
| for dep in dependencies: | ||
| dep_name, dep_version = (dep.split("==") + [None])[:2] | ||
| yield PythonDependency( | ||
| name=dep_name, | ||
| version=dep_version, | ||
| specifications=[ | ||
| PythonSpecification(f"{dep_name}=={dep_version}" if dep_version else dep_name, found=file) | ||
| ], | ||
| found=file, | ||
| insecure_versions=[], | ||
| secure_versions=[], | ||
| latest_version=None, | ||
| latest_version_without_known_vulnerabilities=None, | ||
| more_info_url=None | ||
| ) |
There was a problem hiding this comment.
Use packaging library for robust dependency parsing
The current approach of splitting on == is fragile and won't handle various version specifiers correctly.
Use the packaging library that's already imported:
for dep in dependencies:
- dep_name, dep_version = (dep.split("==") + [None])[:2]
+ try:
+ req = Requirement(dep)
+ dep_name = req.name
+ dep_version = str(req.specifier) if req.specifier else None
+ except InvalidRequirement:
+ logger.warning(f"Invalid requirement format: {dep}")
+ continue
yield PythonDependency(
name=dep_name,
version=dep_version,
specifications=[
- PythonSpecification(f"{dep_name}=={dep_version}" if dep_version else dep_name, found=file)
+ PythonSpecification(str(req), found=file)
],
found=file,
insecure_versions=[],
secure_versions=[],
latest_version=None,
latest_version_without_known_vulnerabilities=None,
more_info_url=None
)Committable suggestion was skipped due to low confidence.
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
- safety/scan/ecosystems/python/dependencies.py (1 hunks)
- safety/scan/finder/file_finder.py (1 hunks)
🧰 Additional context used
🪛 Ruff
safety/scan/finder/file_finder.py
153-153: Use
enumerate()for index variablelevelinforloop(SIM113)
🔇 Additional comments (2)
safety/scan/finder/file_finder.py (2)
152-155: Remove unused level counter.The
levelcounter is:
- Not used anywhere in the code
- Redundant with the existing
current_depthvariable- Incorrectly tracks iteration count rather than depth
The previous review comment about removing this counter is still valid. Please refer to that comment for the suggested fix.
🧰 Tools
🪛 Ruff
153-153: Use
enumerate()for index variablelevelinforloop(SIM113)
Line range hint
1-200: Implementation missing for special files support.The PR objectives mention adding support for
pyproject.tomlandenv.yml/env.yamlfiles, but the implementation is not visible in this file. The file finder needs to be updated to recognize these special files.Let's verify if the special files implementation exists elsewhere:
Please ensure that:
- Special files are properly registered in the
FileTypeenum- Appropriate handlers are implemented for these file types
- The handlers are registered in
ECOSYSTEM_HANDLER_MAPPINGWould you like assistance in implementing the special files support?
🧰 Tools
🪛 Ruff
153-153: Use
enumerate()for index variablelevelinforloop(SIM113)
| @@ -285,10 +283,11 @@ def get_dependencies(f: InspectableFile) -> List[PythonDependency]: | |||
| return [] | |||
|
|
|||
| if f.file_type in [FileType.REQUIREMENTS_TXT, FileType.POETRY_LOCK, | |||
| FileType.PIPENV_LOCK]: | |||
| FileType.PIPENV_LOCK, FileType.PYPROJECT_TOML]: | |||
There was a problem hiding this comment.
Missing implementation for pyproject.toml parsing
While FileType.PYPROJECT_TOML has been added to the list of supported file types, there's no specialized parsing logic for pyproject.toml files. The current implementation will try to use the generic read_dependencies() function, which may not correctly handle the unique structure of pyproject.toml files (e.g., project.dependencies, tool.poetry.dependencies, etc.).
Add a specialized parser for pyproject.toml files:
def read_pyproject_toml_dependencies(file: Path) -> Generator[PythonDependency, None, None]:
"""
Reads dependencies from a pyproject.toml file.
Args:
file (Path): Path to the pyproject.toml file
Returns:
Generator[PythonDependency, None, None]: Generator of PythonDependency objects
"""
try:
import toml
data = toml.load(file)
dependencies = []
# Check build-system requires
if 'build-system' in data and 'requires' in data['build-system']:
dependencies.extend(data['build-system']['requires'])
# Check project dependencies
if 'project' in data:
if 'dependencies' in data['project']:
dependencies.extend(data['project']['dependencies'])
if 'optional-dependencies' in data['project']:
for group in data['project']['optional-dependencies'].values():
dependencies.extend(group)
# Check poetry dependencies
if 'tool' in data and 'poetry' in data['tool']:
if 'dependencies' in data['tool']['poetry']:
for dep, version in data['tool']['poetry']['dependencies'].items():
if isinstance(version, dict):
# Handle complex version specifications
dep_spec = f"{dep}{version.get('version', '')}"
else:
dep_spec = f"{dep}{version}"
dependencies.append(dep_spec)
if 'dev-dependencies' in data['tool']['poetry']:
for dep, version in data['tool']['poetry']['dev-dependencies'].items():
if isinstance(version, dict):
dep_spec = f"{dep}{version.get('version', '')}"
else:
dep_spec = f"{dep}{version}"
dependencies.append(dep_spec)
# Parse dependencies
from packaging.requirements import Requirement
for dep in dependencies:
try:
req = Requirement(dep)
yield PythonDependency(
name=req.name,
version=str(req.specifier) if req.specifier else None,
specifications=[PythonSpecification(str(req), found=file)],
found=file,
absolute_path=file,
insecure_versions=[],
secure_versions=[],
latest_version=None,
latest_version_without_known_vulnerabilities=None,
more_info_url=None
)
except Exception as e:
logger.warning(f"Failed to parse dependency {dep}: {str(e)}")
continue
except Exception as e:
logger.error(f"Failed to parse {file}: {str(e)}")
returnThen update the get_dependencies function to use this specialized parser:
if f.file_type in [FileType.REQUIREMENTS_TXT, FileType.POETRY_LOCK,
FileType.PIPENV_LOCK, FileType.PYPROJECT_TOML]:
+ if f.file_type == FileType.PYPROJECT_TOML:
+ return list(read_pyproject_toml_dependencies(Path(f.file.name)))
return list(read_dependencies(f.file, resolve=True))Add support for env.yml/env.yaml files
According to the PR objectives, support for env.yml and env.yaml files should be added, but this functionality is missing from the implementation.
Add support for conda environment files:
def read_conda_environment_dependencies(file: Path) -> Generator[PythonDependency, None, None]:
"""
Reads dependencies from a conda environment file (env.yml/env.yaml).
Args:
file (Path): Path to the environment file
Returns:
Generator[PythonDependency, None, None]: Generator of PythonDependency objects
"""
try:
import yaml
with open(file) as f:
data = yaml.safe_load(f)
if not data or 'dependencies' not in data:
return
for dep in data['dependencies']:
# Skip conda-forge and other channels
if isinstance(dep, dict) or not isinstance(dep, str):
continue
# Skip non-pip packages
if dep.startswith('pip:'):
continue
try:
from packaging.requirements import Requirement
req = Requirement(dep)
yield PythonDependency(
name=req.name,
version=str(req.specifier) if req.specifier else None,
specifications=[PythonSpecification(str(req), found=file)],
found=file,
absolute_path=file,
insecure_versions=[],
secure_versions=[],
latest_version=None,
latest_version_without_known_vulnerabilities=None,
more_info_url=None
)
except Exception as e:
logger.warning(f"Failed to parse dependency {dep}: {str(e)}")
continue
# Handle pip dependencies
pip_deps = next((deps for dep in data['dependencies']
if isinstance(dep, dict) and 'pip' in dep
for deps in dep['pip']), [])
for dep in pip_deps:
try:
from packaging.requirements import Requirement
req = Requirement(dep)
yield PythonDependency(
name=req.name,
version=str(req.specifier) if req.specifier else None,
specifications=[PythonSpecification(str(req), found=file)],
found=file,
absolute_path=file,
insecure_versions=[],
secure_versions=[],
latest_version=None,
latest_version_without_known_vulnerabilities=None,
more_info_url=None
)
except Exception as e:
logger.warning(f"Failed to parse pip dependency {dep}: {str(e)}")
continue
except Exception as e:
logger.error(f"Failed to parse {file}: {str(e)}")
returnThen update the get_dependencies function to handle these files:
if f.file_type in [FileType.REQUIREMENTS_TXT, FileType.POETRY_LOCK,
- FileType.PIPENV_LOCK, FileType.PYPROJECT_TOML]:
+ FileType.PIPENV_LOCK, FileType.PYPROJECT_TOML,
+ FileType.CONDA_ENVIRONMENT]:
+ if f.file_type == FileType.PYPROJECT_TOML:
+ return list(read_pyproject_toml_dependencies(Path(f.file.name)))
+ if f.file_type == FileType.CONDA_ENVIRONMENT:
+ return list(read_conda_environment_dependencies(Path(f.file.name)))
return list(read_dependencies(f.file, resolve=True))
|
After changes in our approach, the code changes we need are much simpler and it is easier to just open them in a new PR here: #621 |
Tested this manually by adding these files to the repo and conifrming they are picked up. This supports both versions: env.yml and env.yaml
Original issue: #505
Summary by CodeRabbit
New Features
pyproject.tomlfiles.pyproject.tomland environment files.Improvements
Bug Fixes