\n",
- "\n",
- "Use Dask dataframe of MNIST (df2) and follow these steps:\n",
- " \n",
- "1. Add a new column called `sum` to the dataframe which contains sum of all the pixels\n",
- "2. Use groupby to find the mean value for `sum` for each label\n",
- " \n",
- "\n",
- " \n",
- " → Hints\n",
- "\n",
- " * Columns 1 onwards are the pixels. You can access them with `pixels=df2.iloc[:, 1:]`\n",
- " * Instead of `df['sum']=pixels.sum()` try `df['sum']=pixels.sum(axis=1)` because we want to sum along columns, not rows\n",
- " * If the dask output is confusing, try with df1 first\n",
- " * to groupby use `df2.groupby('label').?`, where you replace the `?` with the aggregation operation\n",
- "\n",
- " \n",
- "\n",
- " \n",
- " \n",
- " \n",
- " \n",
- " → Solution\n",
- " \n",
- "\n",
- " ```python\n",
- " # With pandas\n",
- " pixels = df1.loc[:, ['pixel' in c for c in df1.columns]]\n",
- " df1['sum']=pixels.sum(axis=1)\n",
- " task = df1[['label','sum']].groupby('label').mean()\n",
- " print(result)\n",
- "\n",
- " # With dask\n",
- " pixels = df2.loc[:, ['pixel' in c for c in df2.columns]]\n",
- " df2['sum']=pixels.sum(axis=1)\n",
- " task = df2[['label','sum']].groupby('label').mean()\n",
- " with ProgressBar():\n",
- " result=task.compute() \n",
- " print(result)\n",
- " ```\n",
- "\n",
- " \n",
- "\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## When to use Dask DataFrame?\n",
- "\n",
- "Lets visit [the dask page](https://docs.dask.org/en/latest/dataframe.html#common-uses-and-anti-uses) to look at when we should use it\n",
- "\n",
- "It is harder so only if you dataset is larger than memory.\n",
- "\n",
- "If fact also consider:\n",
- "- a database (if you have lots of structured queries)\n",
- "- https://downloadmoreram.com/ ;p\n",
- "- dask array"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Dask Array\n",
- "Dask is not just used to replace pandas. There are also multiple numpy functions which can be replaced by Dask. Dask array is Dask equivalent of a numpy array. By doing so, we can perform the computations in parallel and get the results faster."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {
- "ExecuteTime": {
- "end_time": "2020-10-04T07:13:23.588550Z",
- "start_time": "2020-10-04T07:13:23.584811Z"
- }
- },
- "outputs": [],
- "source": [
- "from dask import array"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {
- "ExecuteTime": {
- "end_time": "2020-10-04T07:13:23.605397Z",
- "start_time": "2020-10-04T07:13:23.593664Z"
- }
- },
- "outputs": [],
- "source": [
- "big_array = array.random.normal(size=(10000000, 100), chunks=200000)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
+ "execution_count": 29,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:13:23.616314Z",
- "start_time": "2020-10-04T07:13:23.608703Z"
+ "end_time": "2020-10-14T02:46:19.562831Z",
+ "start_time": "2020-10-14T02:46:18.729080Z"
}
},
"outputs": [
{
"data": {
"text/html": [
- "
\n",
+ "\n",
+ "Use Dask dataframe of MNIST (df2) and follow these steps:\n",
+ " \n",
+ "1. Add a new column called `sum` to the dataframe which contains sum of all the pixels\n",
+ "2. Use groupby to find the mean value for `sum` for each label\n",
+ " \n",
+ "\n",
+ " \n",
+ " → Hints\n",
+ "\n",
+ " * Columns 1 onwards are the pixels. You can access them with `pixels=df2.iloc[:, 1:]`\n",
+ " * Instead of `df['sum']=pixels.sum()` try `df['sum']=pixels.sum(axis=1)` because we want to sum along columns, not rows\n",
+ " * If the dask output is confusing, try with df1 first\n",
+ " * to groupby use `df2.groupby('label').?`, where you replace the `?` with the aggregation operation\n",
+ "\n",
+ " \n",
+ "\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " → Solution\n",
+ " \n",
+ "\n",
+ " ```python\n",
+ " # With pandas\n",
+ " pixels = df1.loc[:, ['pixel' in c for c in df1.columns]]\n",
+ " df1['sum']=pixels.sum(axis=1)\n",
+ " task = df1[['label','sum']].groupby('label').mean()\n",
+ " print(result)\n",
+ "\n",
+ " # With dask\n",
+ " pixels = df2.loc[:, ['pixel' in c for c in df2.columns]]\n",
+ " df2['sum']=pixels.sum(axis=1)\n",
+ " task = df2[['label','sum']].groupby('label').mean()\n",
+ " with ProgressBar():\n",
+ " result=task.compute() \n",
+ " print(result)\n",
+ " ```\n",
+ "\n",
+ " \n",
+ "\n",
+ "\n",
+ "
\n",
- "\n",
- " Description:\n",
- "\n",
- "- Create two Dask random arrays of size 10,000,000-by-100. \n",
- "- Find the difference between the two `y = ..`\n",
- "- and pass it to `array.linalg.norm` using argument `axis=1`. \n",
- "- Calculate the result and create a histogram of it.\n",
- " \n",
- "\n",
- " \n",
- " → Hints\n",
- " \n",
- " Replace the question marks `?`\n",
- "\n",
- "```python\n",
- "a = array.random.normal(size=(10000000, 100), chunks=200000)\n",
- "b = array.random.normal(size=(10000000, 100), chunks=200000)\n",
- "r = array.linalg.norm(?, axis=1)\n",
- "r.?\n",
- "```\n",
- "\n",
- " \n",
- "\n",
- " \n",
- " \n",
- " \n",
- " \n",
- " → Solution\n",
- " \n",
+ "## When to use Dask DataFrame?\n",
"\n",
- " ```python\n",
- " x1 = array.random.random(size=(10000000,100))\n",
- " x2 = array.random.random(size=(10000000,100))\n",
- " y = x2-x1\n",
- " d = array.linalg.norm(y,axis=1)\n",
- " with ProgressBar():\n",
- " result = d.compute()\n",
- " hist(result,bins=100);\n",
- " ```\n",
+ "Lets visit [the dask page](https://docs.dask.org/en/latest/dataframe.html#common-uses-and-anti-uses) to look at when we should use it\n",
"\n",
- " \n",
+ "It is harder so only if you dataset is larger than memory.\n",
"\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Delayed\n",
- "Dask delayed is a method for parallelising code where you can't write your code directly as dataframe or array operation. `Dask.delayed` is an easy-to-use tool to quickly parallelise these tasks."
+ "If fact also consider:\n",
+ "- a database (if you have lots of structured queries)\n",
+ "- https://downloadmoreram.com/ ;p\n",
+ "- dask array"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Consider the following functions. The first one takes an input, waits for one second and returns the value. The second function takes two inputs, waits for one second and returns the sum. We are using these functions to represent tasks that are time consuming."
+ "## Dask Array\n",
+ "Dask is not just used to replace pandas. There are also multiple numpy functions which can be replaced by Dask. Dask array is Dask equivalent of a numpy array. By doing so, we can perform the computations in parallel and get the results faster."
]
},
{
"cell_type": "code",
- "execution_count": 32,
+ "execution_count": 57,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:13:53.300877Z",
- "start_time": "2020-10-04T07:13:53.289192Z"
+ "end_time": "2020-10-14T03:03:00.931400Z",
+ "start_time": "2020-10-14T03:03:00.929055Z"
}
},
"outputs": [],
"source": [
- "from time import sleep\n",
- "\n",
- "\n",
- "def task1(x):\n",
- " sleep(1)\n",
- " return x\n",
- "\n",
- "\n",
- "def task2(x, y):\n",
- " sleep(1)\n",
- " return x + y"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now, if we pass two values separately into the first function and then pass the results into the second function, we will have the following code:"
+ "from dask import array"
]
},
{
"cell_type": "code",
- "execution_count": 33,
+ "execution_count": 58,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:13:56.315870Z",
- "start_time": "2020-10-04T07:13:53.304926Z"
+ "end_time": "2020-10-14T03:03:01.407137Z",
+ "start_time": "2020-10-14T03:03:01.400443Z"
}
},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "CPU times: user 4.81 ms, sys: 74 µs, total: 4.89 ms\n",
- "Wall time: 3 s\n"
- ]
- }
- ],
- "source": [
- "%%time\n",
- "x1 = task1(1)\n",
- "x2 = task1(2)\n",
- "y = task2(x1,x2)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
+ "outputs": [],
"source": [
- "Since each of these functions are taking one second; therefore, the entire block takes three seconds. But the calculation for `x1` is totally independent of the calculation for `x2`. If we were able to do these operation simultaneously we could save time. This is where `Dask.delayed` comes into play. We need to convert the functions into `delayed` functions so Dask can handle parallelisation."
+ "big_array = array.random.normal(size=(10000000, 100), chunks=200000)"
]
},
{
"cell_type": "code",
- "execution_count": 34,
+ "execution_count": 59,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:13:56.325303Z",
- "start_time": "2020-10-04T07:13:56.319479Z"
+ "end_time": "2020-10-14T03:03:01.852375Z",
+ "start_time": "2020-10-14T03:03:01.846761Z"
}
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ " \n",
+ "
Array
Chunk
\n",
+ " \n",
+ " \n",
+ "
Bytes
8.00 GB
160.00 MB
\n",
+ "
Shape
(10000000, 100)
(200000, 100)
\n",
+ "
Count
50 Tasks
50 Chunks
\n",
+ "
Type
float64
numpy.ndarray
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "\n",
+ "
\n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ "dask.array"
+ ]
+ },
+ "execution_count": 59,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
- "task1_delayed = dask.delayed(task1)\n",
- "task2_delayed = dask.delayed(task2)"
+ "big_array"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "And now instead of the original function we use the delayed functions:"
+ "This data takes 8 GB if we wanted to store it in RAM. But Dask only generates the numbers in chunks when it needs them. So at each steps it has to deal with a chunk which is ~160 MB in this case."
]
},
{
"cell_type": "code",
- "execution_count": 35,
+ "execution_count": 60,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:13:56.338070Z",
- "start_time": "2020-10-04T07:13:56.329559Z"
+ "end_time": "2020-10-14T03:04:16.237275Z",
+ "start_time": "2020-10-14T03:04:06.749275Z"
}
},
"outputs": [
@@ -1500,96 +9029,113 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "CPU times: user 524 µs, sys: 0 ns, total: 524 µs\n",
- "Wall time: 465 µs\n"
+ "[########################################] | 100% Completed | 9.5s\n"
]
}
],
"source": [
- "%%time\n",
- "x1 = task1_delayed(1)\n",
- "x2 = task1_delayed(2)\n",
- "y = task2_delayed(x1,x2)"
+ "task = (big_array * big_array).mean(axis=1)\n",
+ "with ProgressBar():\n",
+ " res = task.compute()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can set the chunk size:"
]
},
{
"cell_type": "code",
- "execution_count": 36,
+ "execution_count": 61,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:13:58.355308Z",
- "start_time": "2020-10-04T07:13:56.341660Z"
+ "end_time": "2020-10-14T03:04:50.034869Z",
+ "start_time": "2020-10-14T03:04:50.028940Z"
}
},
"outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "CPU times: user 5.72 ms, sys: 0 ns, total: 5.72 ms\n",
- "Wall time: 2 s\n"
- ]
- },
{
"data": {
+ "text/html": [
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ " \n",
+ "
Array
Chunk
\n",
+ " \n",
+ " \n",
+ "
Bytes
8.00 GB
419.43 MB
\n",
+ "
Shape
(10000000, 100)
(524288, 100)
\n",
+ "
Count
20 Tasks
20 Chunks
\n",
+ "
Type
float64
numpy.ndarray
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "\n",
+ "
\n",
+ "
\n",
+ "
"
+ ],
"text/plain": [
- "3"
+ "dask.array"
]
},
- "execution_count": 36,
+ "execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
- "%%time\n",
- "y.compute()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we saved one second! `x1` and `x2` where calculated in parallel, and then `y` was calculated using `x1` and `x2`."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can directly create delayed functions using `dask.delayed` decorator."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 37,
- "metadata": {
- "ExecuteTime": {
- "end_time": "2020-10-04T07:13:58.365973Z",
- "start_time": "2020-10-04T07:13:58.358782Z"
- }
- },
- "outputs": [],
- "source": [
- "@dask.delayed\n",
- "def task1(x):\n",
- " sleep(1)\n",
- " return x\n",
- "\n",
- "\n",
- "@dask.delayed\n",
- "def task2(x, y):\n",
- " sleep(1)\n",
- " return x + y"
+ "big_array = array.random.normal(size=(10000000, 100), chunks=(2 ** 19, 100))\n",
+ "big_array"
]
},
{
"cell_type": "code",
- "execution_count": 38,
+ "execution_count": 62,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:00.383619Z",
- "start_time": "2020-10-04T07:13:58.369716Z"
+ "end_time": "2020-10-14T03:05:01.836720Z",
+ "start_time": "2020-10-14T03:04:51.480886Z"
}
},
"outputs": [
@@ -1597,239 +9143,294 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "CPU times: user 1.85 ms, sys: 3.8 ms, total: 5.65 ms\n",
- "Wall time: 2 s\n"
+ "[########################################] | 100% Completed | 10.3s\n"
]
- },
- {
- "data": {
- "text/plain": [
- "3"
- ]
- },
- "execution_count": 38,
- "metadata": {},
- "output_type": "execute_result"
}
],
"source": [
- "%%time\n",
- "x1 = task1(1)\n",
- "x2 = task1(2)\n",
- "y = task2(x1,x2)\n",
- "y.compute()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Introduction to Numba"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## What is Numba?\n",
- "\n",
- "Numba is a **just-in-time**, **type-specializing**, **function compiler** for accelerating **numerically-focused** Python. That's a long list, so let's break down those terms:\n",
- "\n",
- " * **function compiler**: Numba compiles Python functions, not entire applications, and not parts of functions. Numba does not replace your Python interpreter, but is just another Python module that can turn a function into a (usually) faster function. \n",
- " * **type-specializing**: Numba speeds up your function by generating a specialized implementation for the specific data types you are using. Python functions are designed to operate on generic data types, which makes them very flexible, but also very slow. In practice, you only will call a function with a small number of argument types, so Numba will generate a fast implementation for each set of types.\n",
- " * **just-in-time**: Numba translates functions when they are first called. This ensures the compiler knows what argument types you will be using. This also allows Numba to be used interactively in a Jupyter notebook just as easily as a traditional application\n",
- " * **numerically-focused**: Currently, Numba is focused on numerical data types, like `int`, `float`, and `complex`. There is very limited string processing support, and many string use cases are not going to work well on the GPU. To get best results with Numba, you will likely be using NumPy arrays.\n"
+ "task = (big_array * big_array).mean(axis=1)\n",
+ "with ProgressBar():\n",
+ " res = task.compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### First Steps\n",
- "\n",
- "Let's write our first Numba function and compile it for the **CPU**. The Numba compiler is typically enabled by applying a *decorator* to a Python function. Decorators are functions that transform Python functions. Here we will use the CPU compilation decorator:"
+ "We can also apply most of common numpy functions to the array."
]
},
{
"cell_type": "code",
- "execution_count": 39,
+ "execution_count": 63,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:01.058402Z",
- "start_time": "2020-10-04T07:14:00.387277Z"
+ "end_time": "2020-10-14T03:05:16.433299Z",
+ "start_time": "2020-10-14T03:05:01.838300Z"
}
},
- "outputs": [],
- "source": [
- "from numba import jit\n",
- "import math\n",
- "\n",
- "\n",
- "@jit\n",
- "def hypot(x, y):\n",
- " # Implementation from https://en.wikipedia.org/wiki/Hypot\n",
- " x = abs(x)\n",
- " y = abs(y)\n",
- " t = min(x, y)\n",
- " x = max(x, y)\n",
- " t = t / x\n",
- " return x * math.sqrt(1 + t * t)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[########################################] | 100% Completed | 14.6s\n"
+ ]
+ }
+ ],
"source": [
- "The above code is equivalent to writing:\n",
- "``` python\n",
- "def hypot(x, y):\n",
- " x = abs(x);\n",
- " y = abs(y);\n",
- " t = min(x, y);\n",
- " x = max(x, y);\n",
- " t = t / x;\n",
- " return x * math.sqrt(1+t*t)\n",
- " \n",
- "hypot = jit(hypot)\n",
- "```\n",
- "This means that the Numba compiler is just a function you can call whenever you want!\n",
- "\n",
- "Let's try out our hypotenuse calculation:"
+ "task = np.sin(big_array).mean(axis=0)\n",
+ "with ProgressBar():\n",
+ " res = task.compute()"
]
},
{
"cell_type": "code",
- "execution_count": 40,
+ "execution_count": 64,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:01.282029Z",
- "start_time": "2020-10-04T07:14:01.061236Z"
+ "end_time": "2020-10-14T03:05:16.439009Z",
+ "start_time": "2020-10-14T03:05:16.435073Z"
}
},
"outputs": [
{
"data": {
+ "text/html": [
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ " \n",
+ "
Array
Chunk
\n",
+ " \n",
+ " \n",
+ "
Bytes
800 B
800 B
\n",
+ "
Shape
(100,)
(100,)
\n",
+ "
Count
68 Tasks
1 Chunks
\n",
+ "
Type
float64
numpy.ndarray
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "\n",
+ "
\n",
+ "
\n",
+ "
"
+ ],
"text/plain": [
- "5.0"
+ "dask.array"
]
},
- "execution_count": 40,
+ "execution_count": 64,
"metadata": {},
"output_type": "execute_result"
- }
- ],
- "source": [
- "hypot(3.0, 4.0)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
+ }
+ ],
"source": [
- "The first time we call `hypot`, the compiler is triggered and compiles a machine code implementation for float inputs. Numba also saves the original Python implementation of the function in the `.py_func` attribute, so we can call the original Python code to make sure we get the same answer:"
+ "task"
]
},
{
"cell_type": "code",
- "execution_count": 41,
+ "execution_count": 65,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:01.289853Z",
- "start_time": "2020-10-04T07:14:01.284540Z"
+ "end_time": "2020-10-14T03:05:16.506778Z",
+ "start_time": "2020-10-14T03:05:16.440443Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
- "5.0"
+ "array([-4.96623569e-05, -9.21618151e-05, 1.77109021e-04, 2.67433735e-04,\n",
+ " -4.54797760e-05, 2.92740013e-04, 1.02261515e-04, -2.70215806e-04,\n",
+ " 5.37307664e-04, 2.86031036e-04, -6.97538808e-06, -3.66262279e-05,\n",
+ " 6.74122747e-05, 1.83273347e-04, 2.11252086e-04, 1.64897837e-04,\n",
+ " 3.59514362e-05, 1.71063759e-04, -9.70854531e-05, -1.43783037e-04,\n",
+ " -2.34059994e-04, 1.52175767e-04, -1.31447779e-04, 3.49086909e-04,\n",
+ " 4.92961515e-05, -6.34444433e-04, -5.88597059e-05, 2.33904797e-05,\n",
+ " 1.43739722e-04, 2.98193817e-04, -3.17477335e-05, -2.08925932e-04,\n",
+ " 2.05396399e-04, 6.85760116e-05, 1.15092378e-04, 1.36169168e-05,\n",
+ " -3.61674345e-04, 2.09824264e-04, -1.25670866e-04, 2.50631214e-04,\n",
+ " 4.12638901e-06, -3.03833410e-04, -6.33738351e-05, -1.26853903e-04,\n",
+ " 1.51523004e-04, -2.89482462e-04, -1.38021839e-04, -9.12122807e-05,\n",
+ " -1.00218247e-04, -4.24066366e-05, 2.81373079e-04, -1.40732558e-04,\n",
+ " 3.44148003e-04, 4.80461779e-04, 1.59566155e-04, -2.33837880e-04,\n",
+ " 1.94673201e-04, -1.12906923e-04, 1.65474798e-04, 3.24221883e-04,\n",
+ " -1.06062398e-04, 1.97985690e-04, -2.05774209e-04, 3.99499162e-05,\n",
+ " -6.56359293e-04, 1.74988815e-04, -1.26379504e-04, -1.44962937e-04,\n",
+ " -6.18747968e-05, 3.76777761e-04, -1.30070472e-05, 1.80178223e-04,\n",
+ " -1.27134975e-04, -6.65599327e-05, 7.29711967e-05, -1.86057964e-04,\n",
+ " -2.11766175e-04, 4.26616018e-05, 3.74094549e-04, -1.60232094e-05,\n",
+ " 7.24228099e-05, 2.65367915e-06, 8.58840574e-06, -1.23935576e-04,\n",
+ " -9.25305321e-05, -2.75633942e-04, 5.31312702e-05, 1.16043460e-04,\n",
+ " -4.12244514e-04, -1.11056734e-04, -7.11785755e-05, 1.73886249e-05,\n",
+ " -8.71786971e-05, 3.79871916e-04, 5.61180778e-05, -1.19167936e-04,\n",
+ " -3.42306665e-04, -1.49317602e-04, -3.93023952e-05, 6.51016327e-05])"
]
},
- "execution_count": 41,
+ "execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
- "hypot.py_func(3.0, 4.0)"
+ "res"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Benchmarking\n",
+ "### Exercise\n",
"\n",
- "An important part of using Numba is measuring the performance of your new code. Let's see if we actually sped anything up. The easiest way to do this in the Jupyter notebook is to use the `%timeit` magic function. Let's first measure the speed of the original Python:"
+ "- Create two Dask random arrays of size 10,000,000-by-100. \n",
+ "- Find the difference between the two `y = ..`\n",
+ "- and pass it to `array.linalg.norm` using argument `axis=1`. \n",
+ "- Calculate the result and create a histogram of it."
]
},
{
"cell_type": "code",
- "execution_count": 42,
+ "execution_count": 31,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:10.897755Z",
- "start_time": "2020-10-04T07:14:01.292519Z"
+ "end_time": "2020-10-04T07:13:53.285755Z",
+ "start_time": "2020-10-04T07:13:53.281047Z"
}
},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "1.18 µs ± 33.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
- ]
+ "outputs": [],
+ "source": [
+ "from matplotlib.pyplot import hist"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-04T07:04:25.331001Z",
+ "start_time": "2020-10-04T07:04:25.308406Z"
}
- ],
+ },
"source": [
- "%timeit hypot.py_func(3.0, 4.0)"
+ "
\n",
+ "
Exercise
\n",
+ "\n",
+ " Description:\n",
+ "\n",
+ "- Create two Dask random arrays of size 10,000,000-by-100. \n",
+ "- Find the difference between the two `y = ..`\n",
+ "- and pass it to `array.linalg.norm` using argument `axis=1`. \n",
+ "- Calculate the result and create a histogram of it.\n",
+ " \n",
+ "\n",
+ " \n",
+ " → Hints\n",
+ " \n",
+ " Replace the question marks `?`\n",
+ "\n",
+ "```python\n",
+ "a = array.random.normal(size=(10000000, 100), chunks=200000)\n",
+ "b = array.random.normal(size=(10000000, 100), chunks=200000)\n",
+ "r = array.linalg.norm(?, axis=1)\n",
+ "r.?\n",
+ "```\n",
+ "\n",
+ " \n",
+ "\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " → Solution\n",
+ " \n",
+ "\n",
+ " ```python\n",
+ " x1 = array.random.random(size=(10000000,100))\n",
+ " x2 = array.random.random(size=(10000000,100))\n",
+ " y = x2-x1\n",
+ " d = array.linalg.norm(y,axis=1)\n",
+ " with ProgressBar():\n",
+ " result = d.compute()\n",
+ " plt.hist(result,bins=100);\n",
+ " ```\n",
+ "\n",
+ " \n",
+ "\n",
+ "
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "The `%timeit` magic runs the statement many times to get an accurate estimate of the run time."
+ "## Delayed\n",
+ "Dask delayed is a method for parallelising code where you can't write your code directly as dataframe or array operation. `Dask.delayed` is an easy-to-use tool to quickly parallelise these tasks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Consider the following functions. The first one takes an input, waits for one second and returns the value. The second function takes two inputs, waits for one second and returns the sum. We are using these functions to represent tasks that are time consuming."
]
},
{
"cell_type": "code",
- "execution_count": 43,
+ "execution_count": 69,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:13.554235Z",
- "start_time": "2020-10-04T07:14:10.900691Z"
+ "end_time": "2020-10-14T03:15:10.324957Z",
+ "start_time": "2020-10-14T03:15:10.321859Z"
}
},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "322 ns ± 8.11 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "%timeit hypot(3.0, 4.0)"
+ "from time import sleep\n",
+ "\n",
+ "\n",
+ "def task1(x):\n",
+ " sleep(1)\n",
+ " return x\n",
+ "\n",
+ "\n",
+ "def task2(x, y):\n",
+ " sleep(1)\n",
+ " return x + y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Numba did a pretty good job with this function. It's 3x faster than the pure Python version.\n",
- "\n",
- "Of course, the `hypot` function is already present in the Python module:"
+ "Now, if we pass two values separately into the first function and then pass the results into the second function, we will have the following code:"
]
},
{
"cell_type": "code",
- "execution_count": 44,
+ "execution_count": 70,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:15.594561Z",
- "start_time": "2020-10-04T07:14:13.556497Z"
+ "end_time": "2020-10-14T03:15:14.035901Z",
+ "start_time": "2020-10-14T03:15:11.029409Z"
}
},
"outputs": [
@@ -1837,43 +9438,54 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "247 ns ± 1.24 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
+ "CPU times: user 0 ns, sys: 2.15 ms, total: 2.15 ms\n",
+ "Wall time: 3 s\n"
]
}
],
"source": [
- "%timeit math.hypot(3.0, 4.0)"
+ "%%time\n",
+ "x1 = task1(1)\n",
+ "x2 = task1(2)\n",
+ "y = task2(x1,x2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Python's built-in is even faster than Numba! This is because Numba does introduce some overhead to each function call that is larger than the function call overhead of Python itself. Extremely fast functions (like the above one) will be hurt by this.\n",
- "\n",
- "(However, if you call one Numba function from another one, there is very little function overhead, sometimes even zero if the compiler inlines the function into the other one.)"
+ "Since each of these functions are taking one second; therefore, the entire block takes three seconds. But the calculation for `x1` is totally independent of the calculation for `x2`. If we were able to do these operation simultaneously we could save time. This is where `Dask.delayed` comes into play. We need to convert the functions into `delayed` functions so Dask can handle parallelisation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 71,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:15:30.151401Z",
+ "start_time": "2020-10-14T03:15:30.148783Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "task1_delayed = dask.delayed(task1)\n",
+ "task2_delayed = dask.delayed(task2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### How does Numba work?\n",
- "\n",
- "The first time we called our Numba-wrapped `hypot` function, the following process was initiated:\n",
- "\n",
- "![Numba Flowchart](img/numba_flowchart.png \"The compilation process\")\n",
- "\n",
- "We can see the result of type inference by using the `.inspect_types()` method, which prints an annotated version of the source code:"
+ "And now instead of the original function we use the delayed functions:"
]
},
{
"cell_type": "code",
- "execution_count": 45,
+ "execution_count": 72,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:15.605026Z",
- "start_time": "2020-10-04T07:14:15.598417Z"
+ "end_time": "2020-10-14T03:16:02.623513Z",
+ "start_time": "2020-10-14T03:16:02.619917Z"
}
},
"outputs": [
@@ -1881,407 +9493,739 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "hypot (float64, float64)\n",
- "--------------------------------------------------------------------------------\n",
- "# File: \n",
- "# --- LINE 5 --- \n",
- "# label 0\n",
- "\n",
- "@jit\n",
- "\n",
- "# --- LINE 6 --- \n",
- "\n",
- "def hypot(x, y):\n",
- "\n",
- " # --- LINE 7 --- \n",
- "\n",
- " # Implementation from https://en.wikipedia.org/wiki/Hypot\n",
- "\n",
- " # --- LINE 8 --- \n",
- " # x = arg(0, name=x) :: float64\n",
- " # y = arg(1, name=y) :: float64\n",
- " # $0.1 = global(abs: ) :: Function()\n",
- " # $0.3 = call $0.1(x, func=$0.1, args=[Var(x, (8))], kws=(), vararg=None) :: (float64,) -> float64\n",
- " # del x\n",
- " # del $0.1\n",
- " # x.1 = $0.3 :: float64\n",
- " # del $0.3\n",
- "\n",
- " x = abs(x)\n",
- "\n",
- " # --- LINE 9 --- \n",
- " # $0.4 = global(abs: ) :: Function()\n",
- " # $0.6 = call $0.4(y, func=$0.4, args=[Var(y, (8))], kws=(), vararg=None) :: (float64,) -> float64\n",
- " # del y\n",
- " # del $0.4\n",
- " # y.1 = $0.6 :: float64\n",
- " # del $0.6\n",
- "\n",
- " y = abs(y)\n",
- "\n",
- " # --- LINE 10 --- \n",
- " # $0.7 = global(min: ) :: Function()\n",
- " # $0.10 = call $0.7(x.1, y.1, func=$0.7, args=[Var(x.1, (8)), Var(y.1, (9))], kws=(), vararg=None) :: (float64, float64) -> float64\n",
- " # del $0.7\n",
- " # t = $0.10 :: float64\n",
- " # del $0.10\n",
- "\n",
- " t = min(x, y)\n",
- "\n",
- " # --- LINE 11 --- \n",
- " # $0.11 = global(max: ) :: Function()\n",
- " # $0.14 = call $0.11(x.1, y.1, func=$0.11, args=[Var(x.1, (8)), Var(y.1, (9))], kws=(), vararg=None) :: (float64, float64) -> float64\n",
- " # del y.1\n",
- " # del x.1\n",
- " # del $0.11\n",
- " # x.2 = $0.14 :: float64\n",
- " # del $0.14\n",
- "\n",
- " x = max(x, y)\n",
- "\n",
- " # --- LINE 12 --- \n",
- " # $0.17 = t / x.2 :: float64\n",
- " # del t\n",
- " # t.1 = $0.17 :: float64\n",
- " # del $0.17\n",
- "\n",
- " t = t / x\n",
- "\n",
- " # --- LINE 13 --- \n",
- " # $0.19 = global(math: ) :: Module()\n",
- " # $0.20 = getattr(value=$0.19, attr=sqrt) :: Function()\n",
- " # del $0.19\n",
- " # $const0.21 = const(int, 1) :: Literal[int](1)\n",
- " # $0.24 = t.1 * t.1 :: float64\n",
- " # del t.1\n",
- " # $0.25 = $const0.21 + $0.24 :: float64\n",
- " # del $const0.21\n",
- " # del $0.24\n",
- " # $0.26 = call $0.20($0.25, func=$0.20, args=[Var($0.25, (13))], kws=(), vararg=None) :: (float64,) -> float64\n",
- " # del $0.25\n",
- " # del $0.20\n",
- " # $0.27 = x.2 * $0.26 :: float64\n",
- " # del x.2\n",
- " # del $0.26\n",
- " # $0.28 = cast(value=$0.27) :: float64\n",
- " # del $0.27\n",
- " # return $0.28\n",
- "\n",
- " return x * math.sqrt(1 + t * t)\n",
- "\n",
- "\n",
- "================================================================================\n"
+ "CPU times: user 522 µs, sys: 75 µs, total: 597 µs\n",
+ "Wall time: 439 µs\n"
]
}
],
"source": [
- "hypot.inspect_types()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note that Numba's type names tend to mirror the NumPy type names, so a Python `float` is a `float64` (also called \"double precision\" in other languages). Taking a look at the data types can sometimes be important in GPU code because the performance of `float32` and `float64` computations will be very different on CUDA devices. An accidental upcast can dramatically slow down a function."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### When Things Go Wrong\n",
- "\n",
- "Numba cannot compile all Python code. Some functions don't have a Numba-translation, and some kinds of Python types can't be efficiently compiled at all (yet). For example, Numba does not support `FrozenSet` (as of this tutorial):"
+ "%%time\n",
+ "x1 = task1_delayed(1)\n",
+ "x2 = task1_delayed(2)\n",
+ "y = task2_delayed(x1,x2)"
]
},
{
"cell_type": "code",
- "execution_count": 46,
+ "execution_count": 73,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:15.680101Z",
- "start_time": "2020-10-04T07:14:15.607250Z"
- },
- "scrolled": true
+ "end_time": "2020-10-14T03:16:05.539712Z",
+ "start_time": "2020-10-14T03:16:03.532253Z"
+ }
},
"outputs": [
{
- "name": "stderr",
+ "name": "stdout",
"output_type": "stream",
"text": [
- ":1: NumbaWarning: \u001b[1m\n",
- "Compilation is falling back to object mode WITH looplifting enabled because Function \"cannot_compile\" failed type inference due to: \u001b[1m\u001b[1mnon-precise type pyobject\u001b[0m\n",
- "\u001b[0m\u001b[1m[1] During: typing of argument at (3)\u001b[0m\n",
- "\u001b[1m\n",
- "File \"\", line 3:\u001b[0m\n",
- "\u001b[1mdef cannot_compile(x):\n",
- "\u001b[1m return \"a\" in x\n",
- "\u001b[0m \u001b[1m^\u001b[0m\u001b[0m\n",
- "\u001b[0m\n",
- " @jit\n",
- "/home/wassname/.pyenv/versions/jup3.7.3/lib/python3.7/site-packages/numba/object_mode_passes.py:178: NumbaWarning: \u001b[1mFunction \"cannot_compile\" was compiled in object mode without forceobj=True.\n",
- "\u001b[1m\n",
- "File \"\", line 2:\u001b[0m\n",
- "\u001b[1m@jit\n",
- "\u001b[1mdef cannot_compile(x):\n",
- "\u001b[0m\u001b[1m^\u001b[0m\u001b[0m\n",
- "\u001b[0m\n",
- " state.func_ir.loc))\n",
- "/home/wassname/.pyenv/versions/jup3.7.3/lib/python3.7/site-packages/numba/object_mode_passes.py:187: NumbaDeprecationWarning: \u001b[1m\n",
- "Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.\n",
- "\n",
- "For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit\n",
- "\u001b[1m\n",
- "File \"\", line 2:\u001b[0m\n",
- "\u001b[1m@jit\n",
- "\u001b[1mdef cannot_compile(x):\n",
- "\u001b[0m\u001b[1m^\u001b[0m\u001b[0m\n",
- "\u001b[0m\n",
- " warnings.warn(errors.NumbaDeprecationWarning(msg, state.func_ir.loc))\n"
+ "CPU times: user 2.72 ms, sys: 394 µs, total: 3.11 ms\n",
+ "Wall time: 2 s\n"
]
},
{
"data": {
"text/plain": [
- "True"
+ "3"
]
},
- "execution_count": 46,
+ "execution_count": 73,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
- "@jit\n",
- "def cannot_compile(x):\n",
- " return \"a\" in x\n",
- "\n",
- "\n",
- "cannot_compile(frozenset((\"a\", \"b\", \"c\")))"
+ "%%time\n",
+ "y.compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Wait, what happened?? By default, Numba will fall back to a mode, called \"object mode,\" which does not do type-specialization. Object mode exists to enable other Numba functionality, but in many cases, you want Numba to tell you if type inference fails. You can force \"nopython mode\" (the other compilation mode) by passing arguments to the decorator:"
+ "And we saved one second! `x1` and `x2` where calculated in parallel, and then `y` was calculated using `x1` and `x2`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can directly create delayed functions using `dask.delayed` decorator."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 74,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:19:36.667741Z",
+ "start_time": "2020-10-14T03:19:36.664494Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "@dask.delayed\n",
+ "def task1(x):\n",
+ " sleep(1)\n",
+ " return x\n",
+ "\n",
+ "\n",
+ "@dask.delayed\n",
+ "def task2(x, y):\n",
+ " sleep(1)\n",
+ " return x + y"
]
},
{
"cell_type": "code",
- "execution_count": 47,
+ "execution_count": 75,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:15.702891Z",
- "start_time": "2020-10-04T07:14:15.683207Z"
- },
- "scrolled": true
+ "end_time": "2020-10-14T03:19:44.870446Z",
+ "start_time": "2020-10-14T03:19:42.862419Z"
+ }
},
"outputs": [
{
- "name": "stderr",
+ "name": "stdout",
"output_type": "stream",
"text": [
- "ERROR:root:Failed in nopython mode pipeline (step: nopython frontend)\n",
- "\u001b[1m\u001b[1mnon-precise type pyobject\u001b[0m\n",
- "\u001b[0m\u001b[1m[1] During: typing of argument at (3)\u001b[0m\n",
- "\u001b[1m\n",
- "File \"\", line 3:\u001b[0m\n",
- "\u001b[1mdef cannot_compile(x):\n",
- "\u001b[1m return \"a\" in x\n",
- "\u001b[0m \u001b[1m^\u001b[0m\u001b[0m\n",
- "\n",
- "This error may have been caused by the following argument(s):\n",
- "- argument 0: \u001b[1mcannot determine Numba type of \u001b[0m\n",
- "\n",
- "This is not usually a problem with Numba itself but instead often caused by\n",
- "the use of unsupported features or an issue in resolving types.\n",
- "\n",
- "To see Python/NumPy features supported by the latest release of Numba visit:\n",
- "http://numba.pydata.org/numba-doc/latest/reference/pysupported.html\n",
- "and\n",
- "http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html\n",
- "\n",
- "For more information about typing errors and how to debug them visit:\n",
- "http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile\n",
- "\n",
- "If you think your code should work with Numba, please report the error message\n",
- "and traceback, along with a minimal reproducer at:\n",
- "https://github.com/numba/numba/issues/new\n",
- "Traceback (most recent call last):\n",
- " File \"\", line 6, in \n",
- " cannot_compile(frozenset((\"a\", \"b\", \"c\")))\n",
- " File \"/home/wassname/.pyenv/versions/jup3.7.3/lib/python3.7/site-packages/numba/dispatcher.py\", line 401, in _compile_for_args\n",
- " error_rewrite(e, 'typing')\n",
- " File \"/home/wassname/.pyenv/versions/jup3.7.3/lib/python3.7/site-packages/numba/dispatcher.py\", line 344, in error_rewrite\n",
- " reraise(type(e), e, None)\n",
- " File \"/home/wassname/.pyenv/versions/jup3.7.3/lib/python3.7/site-packages/numba/six.py\", line 668, in reraise\n",
- " raise value.with_traceback(tb)\n",
- "numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)\n",
- "\u001b[1m\u001b[1mnon-precise type pyobject\u001b[0m\n",
- "\u001b[0m\u001b[1m[1] During: typing of argument at (3)\u001b[0m\n",
- "\u001b[1m\n",
- "File \"\", line 3:\u001b[0m\n",
- "\u001b[1mdef cannot_compile(x):\n",
- "\u001b[1m return \"a\" in x\n",
- "\u001b[0m \u001b[1m^\u001b[0m\u001b[0m\n",
- "\n",
- "This error may have been caused by the following argument(s):\n",
- "- argument 0: \u001b[1mcannot determine Numba type of \u001b[0m\n",
- "\n",
- "This is not usually a problem with Numba itself but instead often caused by\n",
- "the use of unsupported features or an issue in resolving types.\n",
- "\n",
- "To see Python/NumPy features supported by the latest release of Numba visit:\n",
- "http://numba.pydata.org/numba-doc/latest/reference/pysupported.html\n",
- "and\n",
- "http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html\n",
- "\n",
- "For more information about typing errors and how to debug them visit:\n",
- "http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile\n",
- "\n",
- "If you think your code should work with Numba, please report the error message\n",
- "and traceback, along with a minimal reproducer at:\n",
- "https://github.com/numba/numba/issues/new\n",
- "\n"
+ "CPU times: user 3.28 ms, sys: 475 µs, total: 3.75 ms\n",
+ "Wall time: 2 s\n"
]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "3"
+ ]
+ },
+ "execution_count": 75,
+ "metadata": {},
+ "output_type": "execute_result"
}
],
"source": [
- "@jit(nopython=True)\n",
- "def cannot_compile(x):\n",
- " return \"a\" in x\n",
- "\n",
- "try:\n",
- " cannot_compile(frozenset((\"a\", \"b\", \"c\")))\n",
- "except Exception as e:\n",
- " logging.exception(e)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we get an exception when Numba tries to compile the function, with an error that says:\n",
- "```\n",
- "- argument 0: cannot determine Numba type of \n",
- "```\n",
- "which is the underlying problem. Numba doesn't know about frozenset. There are classes that we use regularly in our code but they might not be defined in Numba. An example of a common class that you cannot use in Numba is pandas data frames. Now the question is: what does Numba support? Some of the types/classes that are supported by Numba are listed below:\n",
- "* Numbers (integers, floats, etc)\n",
- "* Numpy arrays\n",
- "* Strings\n",
- "* Lists and tuples (note that a list/tuple of numbers or strings is supported but a list of lists is not)"
+ "%%time\n",
+ "x1 = task1(1)\n",
+ "x2 = task1(2)\n",
+ "y = task2(x1,x2)\n",
+ "y.compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "So, if we want the last example to be compiled successfully by Numba jit, we need to use a tuple or a list."
+ "# Xarray\n",
+ "\n",
+ "Xarray is pandas for N-dimensional data. It also has a [dask backend](http://xarray.pydata.org/en/stable/dask.html)"
]
},
{
"cell_type": "code",
- "execution_count": 48,
+ "execution_count": 77,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-04T07:14:16.550823Z",
- "start_time": "2020-10-04T07:14:15.708748Z"
+ "end_time": "2020-10-14T03:25:15.644058Z",
+ "start_time": "2020-10-14T03:25:15.588851Z"
}
},
"outputs": [
{
"data": {
+ "text/html": [
+ "
\n",
+ "
<xarray.Dataset>\n",
+ "Dimensions: (time: 36, x: 275, y: 205)\n",
+ "Coordinates:\n",
+ " * time (time) object 1980-09-16 12:00:00 ... 1983-08-17 00:00:00\n",
+ " xc (y, x) float64 dask.array<chunksize=(205, 275), meta=np.ndarray>\n",
+ " yc (y, x) float64 dask.array<chunksize=(205, 275), meta=np.ndarray>\n",
+ "Dimensions without coordinates: x, y\n",
+ "Data variables:\n",
+ " Tair (time, y, x) float64 dask.array<chunksize=(10, 205, 275), meta=np.ndarray>\n",
+ "Attributes:\n",
+ " title: /workspace/jhamman/processed/R1002RBRxaaa01a/l...\n",
+ " institution: U.W.\n",
+ " source: RACM R1002RBRxaaa01a\n",
+ " output_frequency: daily\n",
+ " output_mode: averaged\n",
+ " convention: CF-1.4\n",
+ " references: Based on the initial model of Liang et al., 19...\n",
+ " comment: Output from the Variable Infiltration Capacity...\n",
+ " nco_openmp_thread_number: 1\n",
+ " NCO: netCDF Operators version 4.7.9 (Homepage = htt...\n",
+ " history: Fri Aug 7 17:57:38 2020: ncatted -a bounds,,d...
\n",
+ "\n",
+ " 1. Look at the output of `ds.time.dt.season`\n",
+ " 2. Try grouping by season and getting the mean\n",
+ " 3. Plot each season (use the plotting code from above)\n",
+ " \n",
+ "\n",
+ " \n",
+ " → Hints\n",
+ "\n",
+ " * You do a for loop over groups `for season, ds_season in ds.groupby(ds.time.dt.season):`\n",
+ " * You need to remove the time dimension, with `.mean('time')`\n",
+ " * Use `mean['Tair'].plot.pcolormesh()` to plot\n",
+ "\n",
+ " \n",
+ "\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " → Solution\n",
+ " \n",
+ "\n",
+ " ```python\n",
+ " for season, ds_season in ds.groupby(ds.time.dt.season): \n",
+ " mean = ds_season.mean('time')\n",
+ " mean['Tair'].plot.pcolormesh(\n",
+ " vmin=-30, vmax=30)\n",
+ " plt.title(season)\n",
+ " plt.show()\n",
+ " ```\n",
+ "\n",
+ " \n",
+ "\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Introduction to Numba"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## What is Numba?\n",
+ "\n",
+ "Numba is a **just-in-time**, **type-specializing**, **function compiler** for accelerating **numerically-focused** Python. That's a long list, so let's break down those terms:\n",
+ "\n",
+ " * **function compiler**: Numba compiles Python functions, not entire applications, and not parts of functions. Numba does not replace your Python interpreter, but is just another Python module that can turn a function into a (usually) faster function. \n",
+ " * **type-specializing**: Numba speeds up your function by generating a specialized implementation for the specific data types you are using. Python functions are designed to operate on generic data types, which makes them very flexible, but also very slow. In practice, you only will call a function with a small number of argument types, so Numba will generate a fast implementation for each set of types.\n",
+ " * **just-in-time**: Numba translates functions when they are first called. This ensures the compiler knows what argument types you will be using. This also allows Numba to be used interactively in a Jupyter notebook just as easily as a traditional application\n",
+ " * **numerically-focused**: Currently, Numba is focused on numerical data types, like `int`, `float`, and `complex`. There is very limited string processing support, and many string use cases are not going to work well on the GPU. To get best results with Numba, you will likely be using NumPy arrays.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### First Steps\n",
+ "\n",
+ "Let's write our first Numba function and compile it for the **CPU**. The Numba compiler is typically enabled by applying a *decorator* to a Python function. Decorators are functions that transform Python functions. Here we will use the CPU compilation decorator:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:46:54.654222Z",
+ "start_time": "2020-10-14T03:46:54.652118Z"
+ }
+ },
+ "source": [
+ "\n",
+ "The length of the hypotenuse of a triangle is\n",
+ " \n",
+ "$r = \\sqrt{x^2 + y^2}.$\n",
+ "\n",
+ "However, the squares of very large or small values of x and y may exceed the range of machine precision when calculated on a computer, leading to an inaccurate result caused by arithmetic underflow and/or arithmetic overflow.\n",
+ "\n",
+ "$ hypot = |x| \\sqrt{1 + \\left(\\tfrac{y}{x}\\right)^2}$\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 103,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-15T23:28:05.117434Z",
+ "start_time": "2020-10-15T23:28:05.113335Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "from numba import jit\n",
+ "import math\n",
+ "\n",
+ "\n",
+ "@jit\n",
+ "def hypot(x, y):\n",
+ " # Implementation from https://en.wikipedia.org/wiki/Hypot\n",
+ " x = abs(x)\n",
+ " y = abs(y)\n",
+ " t = min(x, y)\n",
+ " x = max(x, y)\n",
+ " t = t / x\n",
+ " return x * math.sqrt(1 + t * t)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The above code is equivalent to writing:\n",
+ "``` python\n",
+ "def hypot(x, y):\n",
+ " x = abs(x);\n",
+ " y = abs(y);\n",
+ " t = min(x, y);\n",
+ " x = max(x, y);\n",
+ " t = t / x;\n",
+ " return x * math.sqrt(1+t*t)\n",
+ " \n",
+ "hypot = jit(hypot)\n",
+ "```\n",
+ "This means that the Numba compiler is just a function you can call whenever you want!\n",
+ "\n",
+ "Let's try out our hypotenuse calculation:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 93,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:47:40.979991Z",
+ "start_time": "2020-10-14T03:47:40.975581Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CPU times: user 6 µs, sys: 1 µs, total: 7 µs\n",
+ "Wall time: 10.3 µs\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "5.0"
+ ]
+ },
+ "execution_count": 93,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "%%time\n",
+ "hypot(3.0, 4.0)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The first time we call `hypot`, the compiler is triggered and compiles a machine code implementation for float inputs. Numba also saves the original Python implementation of the function in the `.py_func` attribute, so we can call the original Python code to make sure we get the same answer:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 94,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:47:42.474371Z",
+ "start_time": "2020-10-14T03:47:42.469826Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CPU times: user 11 µs, sys: 2 µs, total: 13 µs\n",
+ "Wall time: 16.2 µs\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "5.0"
+ ]
+ },
+ "execution_count": 94,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "%%time\n",
+ "hypot.py_func(3.0, 4.0)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Benchmarking\n",
+ "\n",
+ "An important part of using Numba is measuring the performance of your new code. Let's see if we actually sped anything up. The easiest way to do this in the Jupyter notebook is to use the `%timeit` magic function. Let's first measure the speed of the original Python:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 95,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:48:10.886616Z",
+ "start_time": "2020-10-14T03:48:05.339532Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "684 ns ± 7.08 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
+ ]
+ }
+ ],
+ "source": [
+ "%timeit hypot.py_func(3.0, 4.0)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The `%timeit` magic runs the statement many times to get an accurate estimate of the run time."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 96,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:48:12.520692Z",
+ "start_time": "2020-10-14T03:48:10.888168Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "201 ns ± 0.613 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
+ ]
+ }
+ ],
+ "source": [
+ "%timeit hypot(3.0, 4.0)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Numba did a pretty good job with this function. It's 3x faster than the pure Python version.\n",
+ "\n",
+ "Of course, the `hypot` function is already present in the Python module:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 97,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:49:28.961742Z",
+ "start_time": "2020-10-14T03:49:17.857136Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "137 ns ± 0.635 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n"
+ ]
+ }
+ ],
+ "source": [
+ "%timeit math.hypot(3.0, 4.0)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Python's built-in is even faster than Numba! This is because Numba does introduce some overhead to each function call that is larger than the function call overhead of Python itself. Extremely fast functions (like the above one) will be hurt by this.\n",
+ "\n",
+ "(However, if you call one Numba function from another one, there is very little function overhead, sometimes even zero if the compiler inlines the function into the other one.)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How does Numba work?\n",
+ "\n",
+ "The first time we called our Numba-wrapped `hypot` function, the following process was initiated:\n",
+ "\n",
+ "![Numba Flowchart](img/numba_flowchart.png \"The compilation process\")\n",
+ "\n",
+ "We can see the result of type inference by using the `.inspect_types()` method, which prints an annotated version of the source code:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 98,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:50:36.336290Z",
+ "start_time": "2020-10-14T03:50:36.332574Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "hypot (float64, float64)\n",
+ "--------------------------------------------------------------------------------\n",
+ "# File: \n",
+ "# --- LINE 5 --- \n",
+ "\n",
+ "@jit\n",
+ "\n",
+ "# --- LINE 6 --- \n",
+ "\n",
+ "def hypot(x, y):\n",
+ "\n",
+ " # --- LINE 7 --- \n",
+ "\n",
+ " # Implementation from https://en.wikipedia.org/wiki/Hypot\n",
+ "\n",
+ " # --- LINE 8 --- \n",
+ " # label 0\n",
+ " # x = arg(0, name=x) :: float64\n",
+ " # y = arg(1, name=y) :: float64\n",
+ " # $2load_global.0 = global(abs: ) :: Function()\n",
+ " # $6call_function.2 = call $2load_global.0(x, func=$2load_global.0, args=[Var(x, :8)], kws=(), vararg=None) :: (float64,) -> float64\n",
+ " # del x\n",
+ " # del $2load_global.0\n",
+ " # x.1 = $6call_function.2 :: float64\n",
+ " # del $6call_function.2\n",
+ "\n",
+ " x = abs(x)\n",
+ "\n",
+ " # --- LINE 9 --- \n",
+ " # $10load_global.3 = global(abs: ) :: Function()\n",
+ " # $14call_function.5 = call $10load_global.3(y, func=$10load_global.3, args=[Var(y, :8)], kws=(), vararg=None) :: (float64,) -> float64\n",
+ " # del y\n",
+ " # del $10load_global.3\n",
+ " # y.1 = $14call_function.5 :: float64\n",
+ " # del $14call_function.5\n",
+ "\n",
+ " y = abs(y)\n",
+ "\n",
+ " # --- LINE 10 --- \n",
+ " # $18load_global.6 = global(min: ) :: Function()\n",
+ " # $24call_function.9 = call $18load_global.6(x.1, y.1, func=$18load_global.6, args=[Var(x.1, :8), Var(y.1, :9)], kws=(), vararg=None) :: (float64, float64) -> float64\n",
+ " # del $18load_global.6\n",
+ " # t = $24call_function.9 :: float64\n",
+ " # del $24call_function.9\n",
+ "\n",
+ " t = min(x, y)\n",
+ "\n",
+ " # --- LINE 11 --- \n",
+ " # $28load_global.10 = global(max: ) :: Function()\n",
+ " # $34call_function.13 = call $28load_global.10(x.1, y.1, func=$28load_global.10, args=[Var(x.1, :8), Var(y.1, :9)], kws=(), vararg=None) :: (float64, float64) -> float64\n",
+ " # del y.1\n",
+ " # del x.1\n",
+ " # del $28load_global.10\n",
+ " # x.2 = $34call_function.13 :: float64\n",
+ " # del $34call_function.13\n",
+ "\n",
+ " x = max(x, y)\n",
+ "\n",
+ " # --- LINE 12 --- \n",
+ " # $42binary_true_divide.16 = t / x.2 :: float64\n",
+ " # del t\n",
+ " # t.1 = $42binary_true_divide.16 :: float64\n",
+ " # del $42binary_true_divide.16\n",
+ "\n",
+ " t = t / x\n",
+ "\n",
+ " # --- LINE 13 --- \n",
+ " # $48load_global.18 = global(math: ) :: Module()\n",
+ " # $50load_method.19 = getattr(value=$48load_global.18, attr=sqrt) :: Function()\n",
+ " # del $48load_global.18\n",
+ " # $const52.20 = const(int, 1) :: Literal[int](1)\n",
+ " # $58binary_multiply.23 = t.1 * t.1 :: float64\n",
+ " # del t.1\n",
+ " # $60binary_add.24 = $const52.20 + $58binary_multiply.23 :: float64\n",
+ " # del $const52.20\n",
+ " # del $58binary_multiply.23\n",
+ " # $62call_method.25 = call $50load_method.19($60binary_add.24, func=$50load_method.19, args=[Var($60binary_add.24, :13)], kws=(), vararg=None) :: (float64,) -> float64\n",
+ " # del $60binary_add.24\n",
+ " # del $50load_method.19\n",
+ " # $64binary_multiply.26 = x.2 * $62call_method.25 :: float64\n",
+ " # del x.2\n",
+ " # del $62call_method.25\n",
+ " # $66return_value.27 = cast(value=$64binary_multiply.26) :: float64\n",
+ " # del $64binary_multiply.26\n",
+ " # return $66return_value.27\n",
+ "\n",
+ " return x * math.sqrt(1 + t * t)\n",
+ "\n",
+ "\n",
+ "================================================================================\n"
+ ]
+ }
+ ],
+ "source": [
+ "hypot.inspect_types()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that Numba's type names tend to mirror the NumPy type names, so a Python `float` is a `float64` (also called \"double precision\" in other languages). Taking a look at the data types can sometimes be important in GPU code because the performance of `float32` and `float64` computations will be very different on CUDA devices. An accidental upcast can dramatically slow down a function."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### When Things Go Wrong\n",
+ "\n",
+ "Numba cannot compile all Python code. Some functions don't have a Numba-translation, and some kinds of Python types can't be efficiently compiled at all (yet). For example, Numba does not support `FrozenSet` (as of this tutorial):"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 99,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:51:04.544166Z",
+ "start_time": "2020-10-14T03:51:04.459625Z"
+ },
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ ":1: NumbaWarning: \n",
+ "Compilation is falling back to object mode WITH looplifting enabled because Function \"cannot_compile\" failed type inference due to: non-precise type pyobject\n",
+ "During: typing of argument at (3)\n",
+ "\n",
+ "File \"\", line 3:\n",
+ "def cannot_compile(x):\n",
+ " return \"a\" in x\n",
+ " ^\n",
+ "\n",
+ " @jit\n",
+ "/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/numba/core/object_mode_passes.py:178: NumbaWarning: Function \"cannot_compile\" was compiled in object mode without forceobj=True.\n",
+ "\n",
+ "File \"\", line 2:\n",
+ "@jit\n",
+ "def cannot_compile(x):\n",
+ "^\n",
+ "\n",
+ " state.func_ir.loc))\n",
+ "/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/numba/core/object_mode_passes.py:188: NumbaDeprecationWarning: \n",
+ "Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.\n",
+ "\n",
+ "For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit\n",
+ "\n",
+ "File \"\", line 2:\n",
+ "@jit\n",
+ "def cannot_compile(x):\n",
+ "^\n",
+ "\n",
+ " state.func_ir.loc))\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "True"
]
},
- "execution_count": 4,
+ "execution_count": 99,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
- "# But you can use .compute\n",
- "res.compute()"
+ "@jit\n",
+ "def cannot_compile(x):\n",
+ " return \"a\" in x\n",
+ "\n",
+ "\n",
+ "cannot_compile(frozenset((\"a\", \"b\", \"c\")))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Wait, what happened?? By default, Numba will fall back to a mode, called \"object mode,\" which does not do type-specialization. Object mode exists to enable other Numba functionality, but in many cases, you want Numba to tell you if type inference fails. You can force \"nopython mode\" (the other compilation mode) by passing arguments to the decorator:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 101,
"metadata": {
"ExecuteTime": {
- "end_time": "2020-10-13T11:31:32.414803Z",
- "start_time": "2020-10-13T11:31:31.326331Z"
- }
+ "end_time": "2020-10-14T03:55:40.035890Z",
+ "start_time": "2020-10-14T03:55:40.026166Z"
+ },
+ "scrolled": true
},
- "outputs": [],
- "source": []
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "ERROR:root:Failed in nopython mode pipeline (step: nopython frontend)\n",
+ "non-precise type pyobject\n",
+ "During: typing of argument at (3)\n",
+ "\n",
+ "File \"\", line 3:\n",
+ "def cannot_compile(x):\n",
+ " return \"a\" in x\n",
+ " ^\n",
+ "\n",
+ "This error may have been caused by the following argument(s):\n",
+ "- argument 0: cannot determine Numba type of \n",
+ "Traceback (most recent call last):\n",
+ " File \"\", line 6, in \n",
+ " cannot_compile(frozenset((\"a\", \"b\", \"c\")))\n",
+ " File \"/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/numba/core/dispatcher.py\", line 415, in _compile_for_args\n",
+ " error_rewrite(e, 'typing')\n",
+ " File \"/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/numba/core/dispatcher.py\", line 358, in error_rewrite\n",
+ " reraise(type(e), e, None)\n",
+ " File \"/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/numba/core/utils.py\", line 80, in reraise\n",
+ " raise value.with_traceback(tb)\n",
+ "numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)\n",
+ "non-precise type pyobject\n",
+ "During: typing of argument at (3)\n",
+ "\n",
+ "File \"\", line 3:\n",
+ "def cannot_compile(x):\n",
+ " return \"a\" in x\n",
+ " ^\n",
+ "\n",
+ "This error may have been caused by the following argument(s):\n",
+ "- argument 0: cannot determine Numba type of \n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "@jit(nopython=True)\n",
+ "def cannot_compile(x):\n",
+ " return \"a\" in x\n",
+ "\n",
+ "try:\n",
+ " cannot_compile(frozenset((\"a\", \"b\", \"c\")))\n",
+ "except Exception as e:\n",
+ " logging.exception(e)"
+ ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "
\n",
- "
Exercise
\n",
- "\n",
- " 1. Look at the output of `ds.time.dt.season`\n",
- " 2. Try grouping by season and getting the mean\n",
- " 3. Plot each season (use the plotting code from above)\n",
- " \n",
- "\n",
- " \n",
- " → Hints\n",
- "\n",
- " * You do a for loop over groups `for season, ds_season in ds.groupby(ds.time.dt.season):`\n",
- " * You need to remove the time dimension, with `.mean('time')`\n",
- " * Use `mean['Tair'].plot.pcolormesh()` to plot\n",
+ "Now we get an exception when Numba tries to compile the function, with an error that says:\n",
+ "```\n",
+ "- argument 0: cannot determine Numba type of \n",
+ "```\n",
+ "which is the underlying problem. Numba doesn't know about frozenset. There are classes that we use regularly in our code but they might not be defined in Numba. An example of a common class that you cannot use in Numba is pandas data frames. Now the question is: what does Numba support? Some of the types/classes that are supported by Numba are listed below:\n",
+ "* Numbers (integers, floats, etc)\n",
+ "* Numpy arrays\n",
+ "* Strings\n",
+ "* Lists and tuples (note that a list/tuple of numbers or strings is supported but a list of lists is not)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "So, if we want the last example to be compiled successfully by Numba jit, we need to use a tuple or a list."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 102,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-14T03:56:02.269146Z",
+ "start_time": "2020-10-14T03:56:01.765963Z"
+ }
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 102,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "@jit(nopython=True)\n",
+ "def can_compile(x):\n",
+ " return \"a\" in x\n",
"\n",
- " \n",
"\n",
- " \n",
- " \n",
- " \n",
- " \n",
- " → Solution\n",
- " \n",
+ "can_compile((\"a\", \"b\", \"c\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Exercise\n",
+ "Gregory–Leibniz infinite series converges to $\\pi$:\n",
+ "$$\\pi = \\frac{4}{1} - \\frac{4}{3} + \\frac{4}{5} - \\frac{4}{7} + \\frac{4}{9} - \\frac{4}{11} + \\frac{4}{13} - \\cdots$$\n",
"\n",
- " ```python\n",
- " for season, ds_season in ds.groupby(ds.time.dt.season): \n",
- " mean = ds_season.mean('time')\n",
- " mean['Tair'].plot.pcolormesh(\n",
- " vmin=-30, vmax=30)\n",
- " plt.title(season)\n",
- " plt.show()\n",
- " ```\n",
+ "Write a Numba function which calculates the sum of first $n$ terms in this series. Then test its speed agains normal Python function for $ n = 1000000$."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 49,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2020-10-04T07:14:16.556883Z",
+ "start_time": "2020-10-04T07:14:16.553515Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "# Code Here"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Solution\n",
"\n",
- " \n",
+ "```Python\n",
+ " @jit\n",
+ " def gl_pi(n):\n",
+ " pi = 0\n",
+ " for i in range(n):\n",
+ " if i%2 ==0:\n",
+ " pi += 4/(2*i+1)\n",
+ " else:\n",
+ " pi -= 4/(2*i+1)\n",
+ " return pi \n",
+ "```\n",
"\n",
- "
+#
+#
+
+# +
+# With pandas
+# pixels = df1.loc[:, ['pixel' in c for c in df1.columns]]
+pixels = df1.loc[:, 'pixel1':'pixel783']
+pixels
+df1['sum']=pixels.sum(axis=1)
+df1['sum']
+# task = df1[['label','sum']].groupby('label').mean()
+df1[['label','sum']].groupby('label').mean()
+
+gg = df1[['label','sum']].groupby('label')
+list(gg)[0]
+# df1.loc[:, 'pixel1':'pixel783'].csum
+# print(result)
+
+# # With dask
+pixels = df2.loc[:, ['pixel' in c for c in df2.columns]]
+pixels
+# df2['sum']=pixels.sum(axis=1)
+# task = df2[['label','sum']].groupby('label').mean()
+# with ProgressBar():
+# result=task.compute()
+# print(result)
+# -
# ## When to use Dask DataFrame?
#
@@ -271,7 +297,7 @@ def memory_usage():
# d = array.linalg.norm(y,axis=1)
# with ProgressBar():
# result = d.compute()
-# hist(result,bins=100);
+# plt.hist(result,bins=100);
# ```
#
#
@@ -347,6 +373,83 @@ def task2(x, y):
y = task2(x1,x2)
y.compute()
+# # Xarray
+#
+# Xarray is pandas for N-dimensional data. It also has a [dask backend](http://xarray.pydata.org/en/stable/dask.html)
+
+# +
+# %matplotlib inline
+import numpy as np
+import pandas as pd
+import xarray as xr
+import matplotlib.pyplot as plt
+
+ds = xr.tutorial.open_dataset('rasm').load().chunk(dict(time=10))
+ds
+
+# +
+# You can use isel instead of iloc. You always need to specify the dimension
+ds.isel(time=10)['Tair'].plot.pcolormesh(
+ vmin=-30, vmax=30, cmap='Spectral_r',
+ add_colorbar=True, extend='both')
+
+ds.isel(time=10)
+plt.title('Seasonal Surface Air Temperature')
+# -
+
+# You can also resample by date
+res = ds.resample(time='A').mean().isel(x=200, y=200)['Tair']
+# The result is a dask array
+res
+
+# But you can use .compute
+res.compute()
+
+# You can see all the datetime methods
+print(ds.time.dt)
+dir(ds.time.dt)
+
+# These are seasons specified by the months inside
+# JJA = Jun, Jul, Aug.
+ds.time.dt.season
+
+#
+#
Exercise
+#
+# 1. Look at the output of `ds.time.dt.season`
+# 2. Try grouping by season and getting the mean
+# 3. Plot each season (use the plotting code from above)
+#
+#
+#
+# → Hints
+#
+# * You do a for loop over groups `for season, ds_season in ds.groupby(ds.time.dt.season):`
+# * You need to remove the time dimension, with `.mean('time')`
+# * Use `mean['Tair'].plot.pcolormesh()` to plot
+#
+#
+#
+#
+#
+#
+#
+# → Solution
+#
+#
+# ```python
+# for season, ds_season in ds.groupby(ds.time.dt.season):
+# mean = ds_season.mean('time')
+# mean['Tair'].plot.pcolormesh(
+# vmin=-30, vmax=30)
+# plt.title(season)
+# plt.show()
+# ```
+#
+#
+#
+#
+
# # Introduction to Numba
# ## What is Numba?
@@ -363,6 +466,17 @@ def task2(x, y):
#
# Let's write our first Numba function and compile it for the **CPU**. The Numba compiler is typically enabled by applying a *decorator* to a Python function. Decorators are functions that transform Python functions. Here we will use the CPU compilation decorator:
+#
+# The length of the hypotenuse of a triangle is
+#
+# $r = \sqrt{x^2 + y^2}.$
+#
+# However, the squares of very large or small values of x and y may exceed the range of machine precision when calculated on a computer, leading to an inaccurate result caused by arithmetic underflow and/or arithmetic overflow.
+#
+# $ hypot = |x| \sqrt{1 + \left(\tfrac{y}{x}\right)^2}$
+#
+#
+
# +
from numba import jit
import math
@@ -397,10 +511,12 @@ def hypot(x, y):
#
# Let's try out our hypotenuse calculation:
+# %%time
hypot(3.0, 4.0)
# The first time we call `hypot`, the compiler is triggered and compiles a machine code implementation for float inputs. Numba also saves the original Python implementation of the function in the `.py_func` attribute, so we can call the original Python code to make sure we get the same answer:
+# %%time
hypot.py_func(3.0, 4.0)
# ### Benchmarking
@@ -523,76 +639,6 @@ def can_compile(x):
#
#
-# # Xarray
-#
-# Xarray is pandas for N-dimensional data. It also has a [dask backend](http://xarray.pydata.org/en/stable/dask.html)
-
-# +
-# %matplotlib inline
-import numpy as np
-import pandas as pd
-import xarray as xr
-import matplotlib.pyplot as plt
-
-ds = xr.tutorial.open_dataset('rasm').load().chunk(dict(time=10))
-ds
-
-# +
-# You can use isel instead of iloc. You always need to specify the dimension
-ds.isel(time=10)['Tair'].plot.pcolormesh(
- vmin=-30, vmax=30, cmap='Spectral_r',
- add_colorbar=True, extend='both')
-
-ds.isel(time=10)
-# -
-
-# You can also resample by date
-res = ds.resample(time='A').mean().isel(x=200, y=200)['Tair']
-# The result is a dask array
-res
-
-# But you can use .compute
-res.compute()
-
-
-
-#
-#
Exercise
-#
-# 1. Look at the output of `ds.time.dt.season`
-# 2. Try grouping by season and getting the mean
-# 3. Plot each season (use the plotting code from above)
-#
-#
-#
-# → Hints
-#
-# * You do a for loop over groups `for season, ds_season in ds.groupby(ds.time.dt.season):`
-# * You need to remove the time dimension, with `.mean('time')`
-# * Use `mean['Tair'].plot.pcolormesh()` to plot
-#
-#
-#
-#
-#
-#
-#
-# → Solution
-#
-#
-# ```python
-# for season, ds_season in ds.groupby(ds.time.dt.season):
-# mean = ds_season.mean('time')
-# mean['Tair'].plot.pcolormesh(
-# vmin=-30, vmax=30)
-# plt.title(season)
-# plt.show()
-# ```
-#
-#
-#
-#
-
# # References
# The following sources where used for creation of this notebook:
# - https://github.com/NCAR/ncar-python-tutorial