Python: making it easier for users to override installed packages

[gokceneraslan]

When we try to override a system package i.e. pip3 install --user pysam --force-reinstall and import it, current structure of sys.path doesn't allow us to import the new package directly because the system folder is higher in rank than the user folder:

['/home/jupyter-user/notebooks/whatever/edit',
 '/etc/jupyter/custom',
 '/usr/lib/python37.zip',
 '/usr/lib/python3.7',                                          <-------
 '/usr/lib/python3.7/lib-dynload', 
 '',
 '/home/jupyter-user/.local/lib/python3.7/site-packages',       <-------
 '/usr/local/lib/python3.7/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/usr/local/lib/python3.7/dist-packages/IPython/extensions',
 '/home/jupyter-user/.ipython']

Therefore we have to prepend user site-packages folder to sys.path in order to import the new package:

import sys
sys.path.insert(0, '/home/jupyter-user/.local/lib/python3.7')

which leads to

['/home/jupyter-user/.local/lib/python3.7',
 '/home/jupyter-user/notebooks/whatever/edit',
 '/etc/jupyter/custom',
 '/usr/lib/python37.zip',
 '/usr/lib/python3.7',
 '/usr/lib/python3.7/lib-dynload',
 '',
 '/home/jupyter-user/.local/lib/python3.7/site-packages',
 '/usr/local/lib/python3.7/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/usr/local/lib/python3.7/dist-packages/IPython/extensions',
 '/home/jupyter-user/.ipython']

Would it make sense to add the following line to the base image Dockerfile:

ENV PYTHONPATH $HOME/.local/lib/python3.7:${PYTHONPATH:+$PYTHONPATH}

to make it easier to override system packages?

As a use case, you can just try to upgrade pysam package which is already installed in the default runtime and import the updated version to reproduce the issue.

[gokceneraslan]

Here is an example just to show how painful it is now to modify/reinstall/recompile a preinstalled package.

Users shouldn't make new dockerfiles just to upgrade a few packages, this should be easier by default, IMO.

[Qi77Qi]

It makes sense to me... @gokceneraslan would you mind adding the change to your other open PR?