Use multipackage for managing CI/CD scripts

[timburke]

This PR ports iotile_analytics over to using multipackage for managing its CI/CD scripts.

This PR was the result of running:

pip install -U multipackage

Then initializing the repository with:

multipackage init

Then updating the .multipackage/components.txt file with the names of the packages in the repo (This is what used to be scripts/components.py:

$ cat .multipackage/components.txt

# List of all component packages in this repository
#
# This is the master list that is used to determine which subfolders of
# this repository contain python packages that should be tested and released.
#
# There should be one component listed on each line in the following format:
# package_name: relative_path[, compatibility=(universal|python2|python3)]
#
# Whitespace is ignored.  The package name cannot contain a - character or
# other special characters.  The relative path should be the path to the
# folder containing the setup.py file for the package.
#
# If the package is not compatible with either python 2 or python 3, you
# can pass a compatibility flag, specifying which version of python it
# is compatible with.  The default is a universal package that is compatible
# with both python 2 and python 3.
iotile_analytics_core: ./iotile_analytics_core
iotile_analytics_interactive: ./iotile_analytics_interactive
iotile_analytics_offline: ./iotile_analytics_offline

Finally, all of the build scripts were installed using:

multipackage update

You can run multipackage update at any point in the future to update your build scripts to the latest version.

[dkarchmer]

I understand most code is auto-generated, but isn't there a way to still keep most of these code on a package that gets included by the auto-generated code?

[timburke]

@dkarchmer It definitely could be. The question is just philosophical. Do you want the build/release scripts to depend on multipackage explicitly or be standalone? I don't actually know the answer so a discussion would be useful for deciding the right approach here.

My thinking is that we don't want anything related to build/release to depend on what version of multipackage the user has installed. Basically the idea was that you only need multipackage to install or update scripts. For using the scripts the repo is entirely self-contained so you can reason about what it is going to do without needing to reference a potentially uncontrolled external package.

The downside though is that you have these large diffs when you install and potentially when you update as well.

Thoughts?

@mattrunchey Do you have a perspective here?

[dkarchmer]

@timburke I see the argument. As you say, it is a tradeoff.
I can talk of similar projects I use (like serverless) where it has definetely been a challenge to keep all my projects on the same release of the package. One way they solve it is to include a

frameworkVersion: ">=1.17.0 < 1.19.0"

inside the configuration file itself, ensuring that if I update the package, I get an error if it falls outside the allowed versions.

But in my opinion, if I am relying on a framework (and I want to one day be able to do an "update"), I need to come up with a way to manage my dependencies. A good requitements.txt should be enough IMO. Yes, something can change, but at least for Python, things tend to be fairly stable, and if you have good testing, I can trust the framework is protecting me (it is not like we cannot break the build in so many other ways).

[dkarchmer]

Seems this review was stuck some time ago