Home

Python Distribution Metadata Library

A Python distribution used to create, read and update a metadata repository. The repository holds information in Metadata 1.2 (http://www.python.org/dev/peps/pep-0345/) format.

Initially, this package will be developed to build a repository for the locally installed packages. Later I hope to build a repository around http://pypi.pythong.org/pypi/ and other personal indexes.

The distribution repository idea has most likely come from my experience with aptitude. Many of my thoughts about how this project should work have come from using aptitude to install and examine installed packages.

Other wiki pages:

• UseCases
• Tools

Requirements

• Distribution dependency resolution MUST be done without downloading the dependency source.
• Distribution metadata MUST be stored locally to be searched. Live searches against a metadata http service SHOULD be discouraged.
• Distribution metadata MUST be stored in a database agnostic fashion. A file structure database SHOULD be provided as a means to store and metadata.
• Distribution metadata MUST be in http://www.python.org/dev/peps/pep-0345/ format.

Deliverables

• A local repository of Python Distribution metadata.
• A means to determine Distribution selection before source acquistion. And the following sub deliverables:
• Conflict resolution
• Provides suggestions
• A means to transform a buildout.cfg's distribution library to a pip requirement file.

Implementation Plan

1. Start with the perfect scenario and basic metadata repository structure. (done)
2. Build a draft API for the repository. Should answer the question, "How do I search for items in the repo?" (done)
3. Begin pulling packages from PyPI into the perfect scenario (developed under the http://bitbucket.org/pumazi/index2repo package). Good examples will be distributions that do a lot of pinning and have multiple releases to provide more visible relationships. By developing this as a separate package, we should be able to do things like make long running processes or cron tasks that update via RSS or some other means. Other possibilities could include using the index2repo library to build more intelligent package metadata extractors using something like Jenkins.
   • Develop a simple way to transform metadata to a metadata 1.0 or 1.2 ( http://www.python.org/dev/peps/pep-0345/) format. We are allowing metadata 1.0 because 1.2 is basically 1.0 with a few extra fields. We must transform metadata 1.1 because it is invalid and not standardized.
   • Find a way to add the transformed packages to the repository. Possible sollutions...
     • develop something similar to http://mirrorer.alioth.debian.org/
     • create a local repo and push or serve it (mercurial like).
   • Add custom PkgMeta values to the metadata. For example, PkgMeta-Download would be a field that could be used 0 or more times to specify the package's download location. The PkgMeta-* fields will need to be internally standardized in documentation.
   • Set up a pypi mirror app for this case.
4. Develop a way to update the existing repository when changes are made. Make a differential scheme to make repository update files.