Use PURL for inputs

[pombredanne]

I would like to use a PURL as a project input like with pkg:pypi/[email protected]

[tdruez]

It would be nice to use fetchcode for this purpose instead of yet another implementation in ScanCode.io

This does not look supported:

>>> fetch('pkg:pypi/[email protected]')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Volumes/nexB/repos/scancode.io/.venv/lib/python3.12/site-packages/fetchcode/__init__.py", line 108, in fetch
    raise Exception("Not a supported/known scheme.")
Exception: Not a supported/known scheme.

The best I could find is a wheel distribution link using:

>>> list(package.info('pkg:pypi/[email protected]'))
[Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')]
>>> p = list(package.info('pkg:pypi/[email protected]'))[0]
>>> print(p)
Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')
>>> print(p.download_url)
https://files.pythonhosted.org/packages/a2/14/2767d963915f957c07f5d4c3d9c5c9a407415289f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl

@pombredanne @keshav-space could you add proper support for fetching PURLs in fetchcode? Also, it would be better to return a sdist link when possible, such as "https://files.pythonhosted.org/packages/c9/5e/dc6acaf46d78979d6b03458b7a1618a68e152a6776fce95daac5e0f0301b/psycopg2-2.9.9.tar.gz"

[keshav-space]

@tdruez

could you add proper support for fetching PURLs in fetchcode?

FetchCode already supports retrieving package metadata along with download_url using PURL. We can add support for more ecosystems there as needed.

Also, it would be better to return a sdist link when possible, such as

Yes, we should return the sdist link in the repository_download_url which is misisng here.

[tdruez]

FetchCode already supports retrieving package metadata along with download_url using PURL.

@keshav-space Great, could you provide a link to the documentation on how to get the download_url from the suggested PURL input: pkg:pypi/[email protected]?

[keshav-space]

Great, could you provide a link to the documentation on how to get the download_url from the suggested PURL input: pkg:pypi/[email protected]?

@tdruez fetchcode.package.info that you tried earlier is the way to get package metadata, and it includes the download_url field. We're already using this in PurlDB. We don't have proper documentation for fetchcode only the README, which needs to be updated.

>>> list(package.info('pkg:pypi/[email protected]'))
[Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')]
>>> p = list(package.info('pkg:pypi/[email protected]'))[0]
>>> print(p)
Package(type='pypi', namespace=None, name='psycopg2', version='2.9.9')
>>> print(p.download_url)
https://files.pythonhosted.org/packages/a2/14/2767d963915f957c07f5d4c3d9c5c9a407415289f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl

[tdruez]

@keshav-space

list(package.info('pkg:pypi/[email protected]'))[0].download_url

This is not a great API...

The following should be implemented:

fetch('pkg:pypi/[email protected]')

[keshav-space]

@tdruez

list(package.info('pkg:pypi/[email protected]'))[0].download_url

This is not a great API...

We can also call this API with version-less PURL, and in that case, it will return package metadata for all the existing versions of the package.

>>> list(package.info('pkg:pypi/psycopg2'))
[Package(type='pypi', namespace=None, name='psycopg2', version='2.0.10'),
 Package(type='pypi', namespace=None, name='psycopg2', version='2.0.11'),
 Package(type='pypi', namespace=None, name='psycopg2', version='2.0.12'),
 Package(type='pypi', namespace=None, name='psycopg2', version='2.0.13'),
 ...

The following should be implemented:

fetch('pkg:pypi/[email protected]')

So, how should the new API behave? Should it return only the download_url, or the full package metadata? And should the version-less PURL be allowed as an input?