pyproject.toml - trying to move old project to new and failing.

[wb8tyw]

I am trying to convert an old project, d-rats to to be current, and need to get the packaging working.

I have finally got the project pretty much functional under Python 3, even though I have a lot of things to cleanup, but now I need to start getting the packaging working properly for testing the changes.

I tried to create a pyproject.toml and a setup.py for the existing flat layout, and all I ended up getting installed was some of the python modules. And I am not finding answers in the documentation.
I am finding terms mentioned for fields in the pyproject.toml like 'project-dir", and can not find any example of how they should be used.

I can not find anything about how to specify where to put the non library modules of the project.
It appears to have taken the directory of the the library modules as the base directory of the project, when the program is expecting that to be a sub-directory under the project name.

I am missing all the data files needed to actually run the program.
I see the all the files in the tarball, but something is missing to actually get them into the installed venv.

├── build_pot.sh
├── changelog
├── COPYING
├── docs
│   ├── conf.py
│   ├── index.rst
│   ├── make.bat
│   ├── Makefile
│   ├── source
│   │   ├── d_rats.map.rst (rest of directory skipped)
├── d_rats (base of internal library modules)
│   ├── __init__.py
│   ├── map
│   │   ├── __init__.py
│   ├── sessions
│   │   ├── __init__.py
│   ├── static
│   │   └── socket.io.js
│   ├── ui
│   │   ├── __init__.py
├── d-rats.py
├── d-rats_repeater.py
├── forms
│   ├── default.xsl
│   ├── email.xml
│   ├── email.xsl
├── Form_Templates
│   ├── memo.xml
│   └── radiogram.xml
├── images
│   ├── aprs_pri.png
│   ├── aprs_sec.png
├── install.txt
├── internationalization_test.py
├── libexec
│   ├── LZHUF_1.EXE
│   ├── lzhuf.c
│   ├── lzhuf.h
│   ├── main.c
│   ├── makefile
│   └── makefile.wc
├── LICENSE
├── linux
│   └── d-rats_repeater_service
├── locale
│   ├── base.pot
│   ├── en
│   │   └── LC_MESSAGES
│   │       ├── D-RATS.mo
│   └── it
│       └── LC_MESSAGES
│           ├── D-RATS.mo
├── MANIFEST.in
├── msys2_packages.sh
├── PKG-INFO
├── pylintrc
├── pyproject.toml
├── README.md
├── requirements.txt
├── setup.cfg (currently all commented out)
├── setup.py (builds .po files, gzips man files, builds lzhuf)
├── share
│   ├── d-rats2.xpm
│   ├── d-rats.desktop
│   ├── d-rats.py.1
│   ├── d-rats_repeater.desktop
│   ├── d-rats_repeater.py.1
├── ui
│   ├── addport.glade
│   ├── addport.glade.h
│   ├── mainwindow.glade
│   ├── mainwindow.glade.h

Everything runs when I run it from the git checkout now.

I am not sure what directory structure that I should end up with in the venv.

I was expecting something like:

|---- bin
|------|----- d-rats.py
|------|----- d-rats_repeater.py
|---- lib/pythonx.y/site-packages/D_rats/ (most of the directory tree above)

With probably "share" directory tree for the man pages and the message catalogs.

Right now, I can not determine what to put in the setup files to get this going.

The lzhuf image has been put in the a platform specific generated tarball by the older setup.py
And for windows they were creating an EXE file, apparently containing a locally captured pythion environment.

For Windows support, I plan to use mysys2 python only initially.

[abravalheri]

Hi @wb8tyw sorry for the delayed response.

I tried to create a pyproject.toml and a setup.py for the existing flat layout, and all I ended up getting installed was some of the python modules. And I am not finding answers in the documentation.
I am finding terms mentioned for fields in the pyproject.toml like 'project-dir", and can not find any example of how they should be used.

The good news is that flat-layouts don't require you to specify project-dir :)

Usually I would advice the users to rely on the auto-discovery features, but since your project layout is very complex, you will have to either specify [tool.setuptools] packages manually or set [tool.setuptools.packages.find] with something like the following:

[tool.setuptools.packages.find]
include = ["d_rats*"]

I can not find anything about how to specify where to put the non library modules of the project.
It appears to have taken the directory of the the library modules as the base directory of the project, when the program is expecting that to be a sub-directory under the project name.

I am not entirely sure what you mean here by "non library modules". Are you referring to auxiliary files and directories that don't get installed when your project is installed?

In Python packages, usually you have only one top-level directory for the code that gets installed, and all files used during runtime (Python files, images, text files, data files ...) are contained inside this top-level directory. The exception being transient/mutable or user configuration files (which usually are created on-demand inside the user $HOME, with the help of libraries such as platformdirs and are not part of the installation process).

I am missing all the data files needed to actually run the program.
I see the all the files in the tarball, but something is missing to actually get them into the installed venv.

Are you including these auxiliary files to run the program inside the package directory? Maybe it is worth to have a look on these docs.

With probably "share" directory tree for the man pages and the message catalogs.

For some kind of files you could try to use the data-files configuration, however this configuration is very discouraged. There are very few use cases that justify the use of data-files¹ and man pages is a well known 'counter-example' in the community². If your Python package need some files to operate, maybe you could consider adding them directly inside the package folder (immutable files), or creating them on demand under $XDG_DATA_HOME, $XDG_CACHE_HOME, $XDG_CONFIG_HOME (or equivalent) obtained with the help of platformdirs (mutable files).

Footnotes

1. I only know one: sharing a huge volume of immutable files between two packages installed in the same virtual environment. ↩

2. If the package needs to integrate with platform/OS-specific features maybe it would be a good idea to use a platform/OS-specific package distribution format. Normally it is useful to think in terms of virtual environments, if it does not make sense to have a set of files inside a Python virtual environment, than maybe including them in a Python package is not the solution. ↩

[wb8tyw]

I got things sort of working by using the setup.py to build the platform specific binary and the build the message catalogs and place the files in the same locations as the previous locations.
This works except for the Mac-OSX tester who is not a programmer, and I have not had time to do remote troubleshooting with them.
One side effect is that the python -m build is that the setup.py is run twice.
The resulting tarball (except on Mac OSX) can be installed with pip into a virtual envrionment and run.

So in the future I am looking at converting the directory layout to the layout in the documents, and moving the lzhuf binary in a separate package to be installed separately. Not all users need it.
The last time I looked, the only source of the lzhuf packages was in the d-rats source and it is needed to communicate with a third party package over the internet named "winlink".

For most of the files I can then just keep them in them under a data directory.

Except for the message databases, those need to be built by the procedure to make the pip installable tarball.
They are not platform specific. They are binaries that are built from sources stored in git.

I would prefer not to have to wrap the "python -m build" command inside of another procedure just to get the message databases built.

In the future I would hope to have platform specific packages again. I am told the packages were dropped because d-rats still required python2.

[wb8tyw]

Also, for my development system, I am currently using Anti-x Linux which is only at python 3.7, so I can not use features that require a newer python version until Anti-X offers something newer.

[abravalheri]

One side effect is that the python -m build is that the setup.py is run twice.

I would say this is natural. According to the build-system specification, setuptools would run setup.py once to create a sdist and then again to generate a wheel (the wheel is generated from the sdist, and not directly from the project directory).

I would prefer not to have to wrap the "python -m build" command inside of another procedure just to get the message databases built.

I believe that you can create a custom build step for those. You can have a look on this discussion for some pointers: #2591 (comment).