Mesosphere Jupyter Service 1.3.0-0.35.4

Release: 1.3.0-0.35.4

This release contains everything from 1.2.0-0.33.7 and includes and/or updates the following:

Major Features and Improvements

• Based on Debian 9.6

Package Additions

• boost
• conda-pack
• gensim
• h2oai::h2o
• ibis-framework
• ipyleaflet
• nbdime
• nbserverproxy
• numexpr
• openblas
• plotly
• pyomo
• pyomo.extras
• pyomo.solvers
• r-caret
• r-devtools
• r-forecast
• r-nycflights13
• r-plotly
• r-randomforest
• r-sqlite
• r-shiny
• r-sparklyr
• r-tidyverse
• s3cmd
• setproctitle
• typing
• pygdf
• quilt[img,pytorch,torchvision]

Package Bumps

• dask 1.0.0
• distributed 1.25.1
• hadoop 2.9.2
• horovod 0.15.2
• jupyterlab 0.35.4
• mlflow 0.8.1
• pyarrow 0.11.0
• pytorch 1.0.0
• r-base 3.5.1
• ray[debug,rllib] 0.6.1
• setuptools 40.6.3
• tensorflow 1.11.0
• tensorflowonspark 1.4.1
• toree 0.3.0-incubating
• torchvision 0.2.1

Jupyter Extensions Additions

• @jupyterlab/celltags
• dask-labextension
• jupyterlab/git
• jupyterlab-drawio
• jupyter-leaflet
• jupyterlab-kernelspy
• jupyterlab_iframe
• nbdime-jupyterlab
• nbserverproxy
• qgrid

NVIDIA Library Bumps

• cuDNN 7.4.1.5-1+cuda9.0
• NCCL 2.3.7-1+cuda9.0

Miscellaneous Bumps

• libmesos-bundle 1.12.0
• zmartzone/lua-resty-openidc 1.7.0

OpenID Connect

• Added support for specifying:
  • Authorization Parameters
  • Redirect After Logout URI
  • Redirect After Logout With ID Token Hint (default: true)
  • Refresh Session Interval (default: 3300 seconds)
  • Whether to renew Access Token on Expiry (default: true)

Breaking Changes

Configuration

• The OIDC_REDIRECT_URI environment variable must now be specified as an absolute URI since redirect_uri_path is deprecated
• Rename the OIDC_AUTH_METHOD environment variable to OIDC_TOKEN_ENDPOINT_AUTH_METHOD to disambiguate from the Introspection Endpoint Authentication method

Features

• Apache Toree, as of 0.3.0-incubating has removed support for PySpark and SparkR, only the Scala and SQL interpreters will remain available. The vanilla PySpark and SparkR kernels, however retain their ability to launch pre-configured Spark Jobs

Docker Images

• mesosphere/mesosphere-jupyter-service:1.3.0-0.35.4
• mesosphere/mesosphere-jupyter-service:1.3.0-0.35.4-gpu
• mesosphere/mesosphere-data-toolkit:1.0.0-1.0.0
• mesosphere/mesosphere-data-toolkit:1.0.0-1.0.0-gpu

dcos-jupyterlab-1.2.0-0.33.7

This release contains everything from 1.0.0-0.33.4 and includes and/or updates the following packages:

New packages

• MLFlow 0.4.1
  An open source platform for the machine learning lifecycle

Updates

• Apache Toree v0.2.0-incubating-rc6
• Dask 0.18.2
• Distributed 1.22.1
• JupyterLab 0.33.7
• PyTorch 0.4.1
• Bump cuDNN to 7.1.4.18-1, NCCL to 2.2.13-1 e1ab35b
• Bump to Debian 9.5 and pin major packages to channels and versions to improve build reproducability 52218ef
• Switch to an Apache 2.0 License 233dbdf

Improvements

• Add JUPYTER_CONF_URLS to conveniently download a common set of configuration files (e.g., hdfs-site.xml, core-site.xml, hive-site.xml, krb5.conf, jaas.conf)
• Set a sane base path for ~/.ivy a73ebea

Fixes

Fixup OIDC UPN AuthN+Z 3a15a2e a85b349
Fixup Spark History Server 7ff32cd

Documentation

How to use JupyterLab on Mesosphere DC/OS
Fast Data: Data Analytics with JupyterLab, Spark and TensorFlow

Docker Images

CPU: dcoslabs/dcos-jupyterlab:1.2.0-0.33.7
GPU: dcoslabs/dcos-jupyterlab:1.2.0-0.33.7-gpu

Mesosphere Service Catalog

JupyterLab

dcos-jupyterlab-1.0.0-0.32.1

This release contains:

• Apache Spark 2.2.1
  Apache Spark™ is a unified analytics engine for large-scale data processing.
• Apache Toree v0.2.0-incubating-rc5
  Toree is a kernel for the Jupyter Notebook platform providing interactively access to Apache Spark.
• BeakerX 1.0.0
  BeakerX is a collection of kernels and extensions to the Jupyter interactive computing environment. It provides JVM support, Spark cluster support, polyglot programming, interactive plots, tables, forms, publishing, and more.
• Dask 0.18.1
  Dask is a flexible parallel computing library for analytic computing.
• Distributed 1.22.0
  Dask.distributed is a lightweight library for distributed computing in Python. It extends both the concurrent.futures and dask APIs to moderate sized clusters.
• IBM GPU Enabler for Spark 2.0.0
  Provides GPU awareness to Spark
• JupyterLab 0.32.1
  JupyterLab is the next-generation web-based user interface for Project Jupyter.
• PyTorch 0.4.0
  Tensors and Dynamic neural networks in Python with strong GPU acceleration. PyTorch is a deep learning framework for fast, flexible experimentation.
• Ray 0.5.0
  Ray is a flexible, high-performance distributed execution framework.
  • Ray Tune: Hyperparameter Optimization Framework
  • Ray RLlib: Scalable Reinforcement Learning
• TensorFlow 1.9.0
  TensorFlow™ is an open source software library for high performance numerical computation.
• Yahoo TensorFlowOnSpark 1.3.2
  TensorFlowOnSpark brings TensorFlow programs onto Apache Spark clusters.
• XGBoost 0.72
  Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more.

Also includes support for:

• OpenID Connect Authentication and Authorization based on email address or User Principal Name (UPN) (for Windows Integrated Authentication and AD FS 4.0 with Windows Server 2016)
• HDFS connectivity
• S3 connectivity
• GPUs with the <image>:<tag>-gpu Docker Image variant built from Dockerfile-cuDNN

Pre-built JupyterLab Docker Images for Mesosphere DC/OS

Related Docker Images:

• Machine Learning Worker for Mesosphere DC/OS
• Apache Spark (with GPU support) for Mesosphere DC/OS

Pre-Release of the JupyterLab Notebook Service for Mesosphere DC/OS

This release contains:

Apache Spark 2.2.1
Apache Spark™ is a unified analytics engine for large-scale data processing.
BeakerX 1.0.0
BeakerX is a collection of kernels and extensions to the Jupyter interactive computing environment. It provides JVM support, Spark cluster support, polyglot programming, interactive plots, tables, forms, publishing, and more.
Dask 0.18.1
Dask is a flexible parallel computing library for analytic computing.
Distributed 1.22.0
Dask.distributed is a lightweight library for distributed computing in Python. It extends both the concurrent.futures and dask APIs to moderate sized clusters.
IBM GPU Enabler for Spark 2.0.0
Provides GPU awareness to Spark
JupyterLab 0.32.1
JupyterLab is the next-generation web-based user interface for Project Jupyter.
PyTorch 0.4.0
Tensors and Dynamic neural networks in Python with strong GPU acceleration. PyTorch is a deep learning framework for fast, flexible experimentation.
Ray 0.5.0
Ray is a flexible, high-performance distributed execution framework.
Ray Tune: Hyperparameter Optimization Framework
Ray RLlib: Scalable Reinforcement Learning
TensorFlow 1.8.0
TensorFlow™ is an open source software library for high performance numerical computation.
Yahoo TensorFlowOnSpark 1.3.0
TensorFlowOnSpark brings TensorFlow programs onto Apache Spark clusters.
XGBoost 0.72
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more.
Also includes support for:

OpenID Connect Authentication and Authorization based on email address or User Principal Name (UPN) (for Windows Integrated Authentication and AD FS 4.0 with Windows Server 2016)
HDFS connectivity
S3 connectivity
GPUs with the :-gpu Docker Image variant built from Dockerfile-cuDNN
Pre-built JupyterLab Docker Images for Mesosphere DC/OS: https://hub.docker.com/r/dcoslabs/dcos-jupyter/tags/

Related Docker Images:

Machine Learning Worker for Mesosphere DC/OS: https://hub.docker.com/r/dcoslabs/dcos-ml-worker/tags/
Apache Spark (with GPU support) for Mesosphere DC/OS: https://hub.docker.com/r/dcoslabs/dcos-spark/tags/