pythonPackages.apache-airflow: init at 1.10.3 #65026

costrouc · 2019-07-18T13:22:51Z

Motivation for this change

Packaging apache airflow. This PR closes this PR #44028

Things done

pkgs/development/python-modules/apache-airflow/default.nix

costrouc · 2019-07-19T01:05:44Z

So... I was able to build on both python2 and python3 and run airflow version without errors! Would appreciate if others would test. I have made every effort to have the tests running for each new package added. Which are quite a few.

costrouc · 2019-07-19T01:06:18Z

@GrahamcOfBorg build python2Packages.airflow python3Packages.airflow

costrouc · 2019-07-19T01:10:11Z

The package is exposed as pythonPackages.apache-airflow and apache-airflow. Thoughts? Should it be airflow (this is not the official pypi name)

costrouc · 2019-07-19T01:10:48Z

@GrahamcOfBorg build python2Packages.apache-airflow python3Packages.apache-airflow

jonringer

nix-review passes on NixOS
diff LGTM
binary at least displays usage (not familiar enough with application to test further)

would like to see some unit tests ran, just to make sure that dependency import paths don't change from what airflow assumes there to be.

Not related to this pr:
Someone should upstream the dependency bounds of the dependencies, some of the the code bases at work do this much pinning and it makes my soul hurt (I understand the pain of being broken downstream constantly, but some of these are overly restrictive)

   substituteInPlace setup.py \
     --replace "flask-caching>=1.3.3, <1.4.0" "flask-caching" \
     --replace "flask-appbuilder==1.12.3" "flask-appbuilder" \
     --replace "pendulum==1.4.4" "pendulum" \
     --replace "configparser>=3.5.0, <3.6.0" "configparser" \
     --replace "jinja2>=2.7.3, <=2.10.0" "jinja2" \
     --replace "funcsigs==1.0.0" "funcsigs" \
     --replace "flask-swagger==0.2.13" "flask-swagger" \
     --replace "python-daemon>=2.1.1, <2.2" "python-daemon" \
     --replace "alembic>=0.9, <1.0" "alembic" \
     --replace "markdown>=2.5.2, <3.0" "markdown" \
     --replace "future>=0.16.0, <0.17" "future" \
     --replace "tenacity==4.12.0" "tenacity" \
     --replace "werkzeug>=0.14.1, <0.15.0" "werkzeug"

costrouc · 2019-07-19T01:28:54Z

Result of nix-review pr 65026 1

35 package were build:

apache-airflow (python37Packages.apache-airflow)
python27Packages.apache-airflow (python37Packages.apache-airflow)
python27Packages.apispec (python37Packages.apache-airflow)
python27Packages.croniter (python37Packages.apache-airflow)
python27Packages.flask-admin (python37Packages.apache-airflow)
python27Packages.flask-appbuilder (python37Packages.apache-airflow)
python27Packages.flask-babelex (python37Packages.apache-airflow)
python27Packages.flask-mongoengine (python37Packages.apache-airflow)
python27Packages.flask-openid (python37Packages.apache-airflow)
python27Packages.json-merge-patch (python37Packages.apache-airflow)
python27Packages.marshmallow-enum (python37Packages.apache-airflow)
python27Packages.mongoengine (python37Packages.apache-airflow)
python27Packages.openapi-spec-validator (python37Packages.apache-airflow)
python27Packages.prance (python37Packages.apache-airflow)
python27Packages.prison (python37Packages.apache-airflow)
python27Packages.python-openid (python37Packages.apache-airflow)
python27Packages.sqlalchemy-citext (python37Packages.apache-airflow)
python27Packages.sqlalchemy-utils (python37Packages.apache-airflow)
python27Packages.wtf-peewee (python37Packages.apache-airflow)
python37Packages.apispec (python37Packages.apache-airflow)
python37Packages.croniter (python37Packages.apache-airflow)
python37Packages.flask-admin (python37Packages.apache-airflow)
python37Packages.flask-appbuilder (python37Packages.apache-airflow)
python37Packages.flask-babelex (python37Packages.apache-airflow)
python37Packages.flask-mongoengine (python37Packages.apache-airflow)
python37Packages.flask-openid (python37Packages.apache-airflow)
python37Packages.json-merge-patch (python37Packages.apache-airflow)
python37Packages.marshmallow-enum (python37Packages.apache-airflow)
python37Packages.mongoengine (python37Packages.apache-airflow)
python37Packages.openapi-spec-validator (python37Packages.apache-airflow)
python37Packages.prance (python37Packages.apache-airflow)
python37Packages.prison (python37Packages.apache-airflow)
python37Packages.sqlalchemy-citext (python37Packages.apache-airflow)
python37Packages.sqlalchemy-utils (python37Packages.apache-airflow)
python37Packages.wtf-peewee (python37Packages.apache-airflow)

@jonringer I agree it would be nice to get some unit tests working with apache-airflow. That way we have a better guarentee than airflow version worked. Looks like I can manually select some unit tests for example https://github.com/apache/airflow/blob/16d93c9e45e14179c7822fed248743f0c3fd935c/run_unit_tests.sh#L71. Do you have much experience with nose tests?

jonringer · 2019-07-19T03:31:41Z

I do not. In all honesty, I prefer not to touch python code. I'm forced to because of work ;(

risicle · 2019-07-19T21:24:33Z

pkgs/development/python-modules/flask-appbuilder/default.nix

+    # remove mockldap requirement
+    rm flask_appbuilder/tests/_test_ldapsearch.py
+    nosetests flask_appbuilder.api flask_appbuilder.tests.test_api
+  '';


Is it worth leaving these in when they don't actually solve the problem?

risicle · 2019-07-19T21:49:07Z

35 packages built on macos 10.13, py27 and py37 versions' executables both run. Excellent work here.

costrouc · 2019-07-22T13:42:07Z

@GrahamcOfBorg build python2Packages.airflow python3Packages.airflow

costrouc · 2019-07-22T13:42:33Z

Ready for merge. Test pass and builds on osx and linux

costrouc · 2019-07-26T17:01:16Z

Ping. Ready for merge

FRidh · 2019-07-28T08:12:26Z

pkgs/development/python-modules/apache-airflow/default.nix

+}:
+
+let # import error "from pendulum import Pendulum" due to 2.x
+    pendulum1 = pendulum.overrideAttrs (super: rec {


We cannot have multiple versions in python-packages.nix

Having the restriction that only 1 version is allowed. Should I create a patch that has support for the newer versions?

Yes, to get this in it needs to support the pendulum version we have in the package set.

To get around this restriction could I treat airflow as an application rather than a library? I think this could possibly be a hard patch to maintain

And not include it in pythonPackages

Treating airflow itself as an application rather than a library sounds like a good way to go, with all of the dependencies aside from pendulum being regular python packages in the distribution.

I'd really like to be able to use this from NixPkg and appreciate the work the two of you are doing, btw!

costrouc · 2019-08-26T17:41:25Z

@bhipple I am happy to package this simply as an application. Does this work with you @FRidh?

I am sure this question will come up once it is packaged as an application. How would one include additional packages with airflow then?

jonringer · 2019-08-26T17:53:54Z

@costrouc if it's very unlikely that another package will ever need to import this (E.g. it's an application), then I would package it as just an application, and may move it out of python modules (to discourage trying to import it). But I would still wait for @FRidh to input, as he probably has better insights into the correct actions.

costrouc · 2019-08-26T19:17:24Z

On a similar note I have a devious plan for "multiple" python versions.

Python only allows one version of each package at a time (whatever is found in the python path first). What if a tool was written to rewrite import statements for a given package? For example import numpy -> import ab12abc45_numpy_1_16 for example. This would allow there to be many versions of numpy being used in the same environment.

This would allow multiple versions of a packages 😁. i would like to see what would happen with a tool like this.

jonringer · 2019-08-26T19:40:29Z

On a similar note I have a devious plan for "multiple" python versions.

Python only allows one version of each package at a time (whatever is found in the python path first). What if a tool was written to rewrite import statements for a given package? For example import numpy -> import ab12abc45_numpy_1_16 for example. This would allow there to be many versions of numpy being used in the same environment.

This would allow multiple versions of a packages grin. i would like to see what would happen with a tool like this.

I'm assuming it would be a lot of complexity and hair-pulling in constructing a dependency house of cards.

However, if successful, i think it would be really convenient to use when there's mutually exclusive version bounds in the same closure.

costrouc · 2019-08-26T19:52:24Z

Yes I agree that It could add to complexity and would need to be very explicit what is happening. Having something like this would be extremely helpful in these cases were we need to break the rules of multiple packages in nixpkgs. For example here I only want to change the verisons of two packages https://github.com/NixOS/nixpkgs/blob/fd7eef6dcf212e8e164f2201b7e58bb63247f290/pkgs/development/python-modules/apache-airflow/default.nix#L53-L73

jonringer · 2019-08-26T19:54:02Z

@costrouc you may also want to take a look at #67422 then

costrouc · 2019-08-27T18:30:40Z

Okay so this PR is a working example of rewriting imports with https://github.com/nix-community/nixpkgs-pytools/#python-rewrite-imports. It allows multiple versions of python packages existing within the same PYTHONPATH.

How does it work? It uses rope https://github.com/python-rope/rope to refactor certain imports. Rope makes this approach quite robust. Here is an example of how it correctly refactors chaning the numpy import https://github.com/nix-community/nixpkgs-pytools/blob/master/tests/test_import_rewrite.py.

I have left the "tricks" done in rewriting the imports as explicit as possible in this PR. In hopes that if others think this is worth moving forward with we can create a wrapper function that will automatically apply these changes in a more user friendly way. This approach should have minimal impact on libraries that depend on it. Personally I think this is a feasable approach to different versions of python package coexisting. Would love to hear others thoughts.

In this PR I demonstrate that you can have packages that would normally break airflow in the propagated build inputs (pendulum, flask-appbuilder). Because of the import rewrites airflow does not use these packages and allows downstream packages to use the latest.

@FRidh what are your thoughts? Would this be an okay approach to having several versions of a python package available? (Obviously having 1 version is ideal and we should always strive for the latest).

nixos-discourse · 2019-08-27T18:34:14Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/scaling-the-python-package-set-in-nixpkgs/3749/18

… duplicates

costrouc · 2019-08-27T18:43:30Z

@GrahamcOfBorg build python2Packages.apache-airflow python3Packages.apache-airflow

FRidh · 2019-08-28T05:34:29Z

@costrouc please open a separate ticket or PR as demo, don't do it in something you actually want to get merged. Also, make it a minimal working example. Furthermore, post it on ideas or packaging to get additional feedback.

Ericson2314 · 2019-10-15T22:47:17Z

@FRidh Does the PR still have the controversial multiple versions? Or anything else you want changed? I might take this over so just tell me what to do :)

rasendubi · 2019-11-19T13:23:23Z

#73074 adds apache-airflow-1.10.5 and has been merged, so this can be closed?

costrouc requested a review from FRidh as a code owner July 18, 2019 13:22

ofborg bot added the 6.topic: python label Jul 18, 2019

jonringer reviewed Jul 18, 2019

View reviewed changes

pkgs/development/python-modules/apache-airflow/default.nix Outdated Show resolved Hide resolved

pkgs/development/python-modules/apache-airflow/default.nix Outdated Show resolved Hide resolved

costrouc force-pushed the python-airflow-init branch from 99dce52 to 4f587fb Compare July 19, 2019 01:03

costrouc changed the title ~~[WIP] pythonPackages.apache-airflow: init at 1.10.3~~ pythonPackages.apache-airflow: init at 1.10.3 Jul 19, 2019

ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Jul 19, 2019

jonringer approved these changes Jul 19, 2019

View reviewed changes

risicle reviewed Jul 19, 2019

View reviewed changes

costrouc force-pushed the python-airflow-init branch from d654f91 to fd7eef6 Compare July 22, 2019 13:41

FRidh requested changes Jul 28, 2019

View reviewed changes

FRidh added the 2.status: work-in-progress This PR isn't done label Jul 28, 2019

costrouc added 13 commits August 27, 2019 09:02

pythonPackages.sqlalchemy-citext: init at 1.3.0

c4448c6

pythonPackages.flask-admin: init at 1.5.3

2929aa3

pythonPackages.prison: init at 0.1.2

8c8f59e

pythonPackages.marshmallow-enum: init at 1.4.1

4c1c1ac

pythonPackages.python-openid: init at 2.2.5

9edd577

pythonPackages.flask-openid: init at 1.2.5

f17c21e

pythonPackages.prance: init at 0.15.0

6590c56

pythonPackages.apispec: init at 2.0.2

060d05e

pythonPackages.flask-appbuilder: init at 2.1.6

2d35ce1

apache-airflow: refactor add as python application

92d1266

pythonPackages.flask-bcrypt: init at 0.7.1

1489acf

pythonPackages.snakebite: init at 2.11.0

43728e7

python3Packages.nixpkgs-pytools: 1.0.1 -> 1.2.0

2f763cf

costrouc force-pushed the python-airflow-init branch from 3046b43 to 1f39697 Compare August 27, 2019 18:21

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 27, 2019

python3Packages.apache-airflow: refactor use nixpkgs-pytools to allow…

907fe9c

… duplicates

costrouc force-pushed the python-airflow-init branch from 1f39697 to 907fe9c Compare August 27, 2019 18:42

ingenieroariel mentioned this pull request Nov 8, 2019

python3Packages.apache-airflow init 1.10.5 #73074

Merged

7 tasks

jonringer closed this Nov 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.apache-airflow: init at 1.10.3 #65026

pythonPackages.apache-airflow: init at 1.10.3 #65026

costrouc commented Jul 18, 2019 •

edited

Loading

costrouc commented Jul 19, 2019

costrouc commented Jul 19, 2019

costrouc commented Jul 19, 2019 •

edited

Loading

costrouc commented Jul 19, 2019

jonringer left a comment

costrouc commented Jul 19, 2019 •

edited

Loading

jonringer commented Jul 19, 2019

risicle Jul 19, 2019 •

edited

Loading

risicle commented Jul 19, 2019

costrouc commented Jul 22, 2019

costrouc commented Jul 22, 2019

costrouc commented Jul 26, 2019

FRidh Jul 28, 2019

costrouc Jul 28, 2019

FRidh Jul 28, 2019

costrouc Jul 28, 2019

costrouc Jul 28, 2019

bhipple Aug 26, 2019 •

edited

Loading

costrouc commented Aug 26, 2019

jonringer commented Aug 26, 2019

costrouc commented Aug 26, 2019

jonringer commented Aug 26, 2019

costrouc commented Aug 26, 2019

jonringer commented Aug 26, 2019

costrouc commented Aug 27, 2019 •

edited

Loading

nixos-discourse commented Aug 27, 2019

costrouc commented Aug 27, 2019

FRidh commented Aug 28, 2019 •

edited

Loading

Ericson2314 commented Oct 15, 2019 •

edited

Loading

rasendubi commented Nov 19, 2019

pythonPackages.apache-airflow: init at 1.10.3 #65026

pythonPackages.apache-airflow: init at 1.10.3 #65026

Conversation

costrouc commented Jul 18, 2019 • edited Loading

Motivation for this change

Things done

costrouc commented Jul 19, 2019

costrouc commented Jul 19, 2019

costrouc commented Jul 19, 2019 • edited Loading

costrouc commented Jul 19, 2019

jonringer left a comment

Choose a reason for hiding this comment

costrouc commented Jul 19, 2019 • edited Loading

jonringer commented Jul 19, 2019

risicle Jul 19, 2019 • edited Loading

Choose a reason for hiding this comment

risicle commented Jul 19, 2019

costrouc commented Jul 22, 2019

costrouc commented Jul 22, 2019

costrouc commented Jul 26, 2019

FRidh Jul 28, 2019

Choose a reason for hiding this comment

costrouc Jul 28, 2019

Choose a reason for hiding this comment

FRidh Jul 28, 2019

Choose a reason for hiding this comment

costrouc Jul 28, 2019

Choose a reason for hiding this comment

costrouc Jul 28, 2019

Choose a reason for hiding this comment

bhipple Aug 26, 2019 • edited Loading

Choose a reason for hiding this comment

costrouc commented Aug 26, 2019

jonringer commented Aug 26, 2019

costrouc commented Aug 26, 2019

jonringer commented Aug 26, 2019

costrouc commented Aug 26, 2019

jonringer commented Aug 26, 2019

costrouc commented Aug 27, 2019 • edited Loading

nixos-discourse commented Aug 27, 2019

costrouc commented Aug 27, 2019

FRidh commented Aug 28, 2019 • edited Loading

Ericson2314 commented Oct 15, 2019 • edited Loading

rasendubi commented Nov 19, 2019

costrouc commented Jul 18, 2019 •

edited

Loading

costrouc commented Jul 19, 2019 •

edited

Loading

costrouc commented Jul 19, 2019 •

edited

Loading

risicle Jul 19, 2019 •

edited

Loading

bhipple Aug 26, 2019 •

edited

Loading

costrouc commented Aug 27, 2019 •

edited

Loading

FRidh commented Aug 28, 2019 •

edited

Loading

Ericson2314 commented Oct 15, 2019 •

edited

Loading