Plan to support setup.py and remaining Python dependency lockfiles

[rtfpessoa]

Currently the deps.dev UI and APIs have a big percentage of python versions maybe 20-30% which have no known dependencies.
For Python specifically, this is mostly due to lack of support for setup.py, setup.cfg and some other dependency lockfiles.

Is there a plan to support these files?

[rtfpessoa]

@cuixq maybe you can help?

[cuixq]

@PFCM knows more about this :)

[rtfpessoa]

Maybe while I wait for the answer I can provide some details from my side.
We are very interested in this support and are currently adding parsing capabilities to osv-scanner to:

• Support for parsing the install_requires key in setup.py and setup.cfg
• In setup.py it only supports arrays of plain string values being passed as a named argument
• Any dependencies described in other requires keys are not scanned
• Fails fast on unsupported inputs

What is the deps.dev source for deps? Does it download the binaries/releases and parses the source with osv-scanner?
If yes, @cuixq do you think osv-scanner would accept a PR to add support for this?

[cuixq]

I think deps.dev fetches data from upstream (see https://docs.deps.dev/api/v3/#data) and computes dependency graph based on that.

Contribution is always welcome, so feel free to send the PR to OSV-Scanner!

[rtfpessoa]

@cuixq Any idea about the code that does this for python? I tried to look for it but could not find it.

The information, at least in good part only exists in the sdist binaries on pypi, not through APIs for this cases.

[cuixq]

I don't think the code for Python is open sourced.

[PFCM]

Hello, sorry for the delay and thanks for getting in touch.

For reference, just to make sure we're all on the same page: for Python, deps.dev operates on release artifacts downloaded from PyPI, and the dependencies we provide are intended to be as close as practical to the dependencies that pip would install if you were to run pip install <package>==<version> on a reasonably modern x86_64 linux system. As Xueqin mentioned the code for that is not open source, although it could be without too much trouble if there's interest.

As you've noticed the biggest cause of missing dependencies is because they're specified in setup.py, this covers about 22% of all the artifacts we've checked (although only 17% of all versions, because many versions have a wheel that is easier to handle).

Despite this we don't currently have any plans to add support for setup.py specifically. The biggest reasons are:

• this is the first time anyone has asked about it :)
• over the past year, only 3.5% of published releases have used it as the only way to specify dependencies (probably why no one has asked, we don't seem to get too many people interested in historical data for whatever reason)
• relatedly, PEP 517 is starting to take over, which is slightly easier for us (could still ultimately require executing unknown Python code to cover all possible cases, but also PEP 517 builders tend to produce PKG-INFO files that actually have the dependencies in them, so we never need to look at the pyproject.toml etc.)
• it's quite awkward to do well; given the above the value of a partial solution (all that's really feasible without a great deal of effort) seemed low.

However, plans can change. With all that in mind I would be interested to know what the use case is for these more historic dependencies (and/or whether there's an appetite for publishing more of our python code here where we might accept PRs)