t81_558_class_14_05_new_tech.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 14 Video Material\n",
    "\n",
    "* Part 14.1: What is AutoML [[Video]](https://www.youtube.com/watch?v=TFUysIR5AB0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_01_automl.ipynb)\n",
    "* Part 14.2: Using Denoising AutoEncoders in Keras [[Video]](https://www.youtube.com/watch?v=4bTSu6_fucc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_02_auto_encode.ipynb)\n",
    "* Part 14.3: Training an Intrusion Detection System with KDD99 [[Video]](https://www.youtube.com/watch?v=1ySn6h2A68I&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_03_anomaly.ipynb)\n",
    "* Part 14.4: Anomaly Detection in Keras [[Video]](https://www.youtube.com/watch?v=VgyKQ5MTDFc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_04_ids_kdd99.ipynb)\n",
    "* **Part 14.5: The Deep Learning Technologies I am Excited About** [[Video]]() [[Notebook]](t81_558_class_14_05_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 14.5: New Technologies\n",
    "\n",
    "This course changes often to keep up with the rapidly evolving landscape that is deep learning.  If you would like to continue to monitor this class, I suggest following me on the following:\n",
    "\n",
    "* [GitHub](https://github.com/jeffheaton) - I post all changes to GitHub.\n",
    "* [Jeff Heaton's YouTube Channel](https://www.youtube.com/user/HeatonResearch) - I add new videos for this class at my channel.\n",
    "\n",
    "## New Technology Radar\n",
    "\n",
    "Currently, these new technologies are on my radar for possible future inclusion in this course:\n",
    "\n",
    "* Transformers\n",
    "* More Advanced Transfer Learning\n",
    "* Augmentation\n",
    "* Reinforcement Learning beyond TF-Agents\n",
    "\n",
    "This section seeks only to provide a high-level overview of these emerging technologies. I provide links to supplemental material and code in each subsection. I describe these technologies in the following sections.\n",
    "\n",
    "Transformers are a relatively new technology that I will soon add to this course.  They have resulted in many NLP applications. Projects such as the Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT-1,2,3) received much attention from practitioners. Transformers allow the sequence to sequence machine learning, allowing the model to utilize variable length, potentially textual, input.  The output from the transformer is also a variable-length sequence.  This feature enables the transformer to learn to perform such tasks as translation between human languages or even complicated NLP-based classification.  Considerable compute power is needed to take advantage of transformers; thus, you should be taking advantage of transfer learning to train and fine-tune your transformers.\n",
    "\n",
    "Complex models can require considerable training time.  It is not unusual to see GPU clusters trained for days to achieve state of the art results.  This complexity requires a substantial monetary cost to train a state of the art model. Because of this cost, you must consider transfer learning.  Services, such as Hugging Face and NVIDIA GPU Cloud (NGC), contains many advanced pretrained neural networks for you to implement.\n",
    "\n",
    "Augmentation is a technique where algorithms generate additional training data augmenting the training data with new items that are modified versions of the original training data. This technique has seen many applications to computer vision. In this most basic example, the algorithm can flip images vertically and horizontally to quadruple the training set's size. Projects, such as NVIDIA StyleGAN2 ADA have implemented augmentation to substantially decrease the amount of training data that the algorithm needs.\n",
    "\n",
    "Currently, this course makes use of TF-Agents to implement reinforcement learning.  TF-Agents is convenient because it is based on TensorFlow.  However, TF-Agents has been slow to update compared to other frameworks.  Additionally, when TF-Agents is updated, internal errors are often introduced that can take months for the TF-Agents team to fix.  When I compare simple \"Hello World\" type examples for Atari games on platforms like Stable Baselines, to their TF-Agents equivilants, I am left wanting more from TF-Agents.\n",
    "\n",
    "## Programming Language Radar\n",
    "\n",
    "As a machine learning programming language, Python has an absolute lock on the industry.  Python is not going anywhere, any time soon.  My main issue with Python is end-to-end deployment.  Unless you are dealing with Jupyter notebooks or training/pipeline scripts, Python will be your go-to language.  However, to create edge applications, such as web pages and mobile apps, you will certainly need to utilize other languages.  I do not suggest replacing Python with any of the following languages; however, these are some alternative languages and domains that you might choose to use them.\n",
    "\n",
    "* **IOS Application Development** - Swift\n",
    "* **Android Development** - Kotlin and Java\n",
    "* **Web Development** - NodeJS and JavaScript\n",
    "* **Mac Application Development** - Swift or JavaScript with Electron or React Native\n",
    "* **Windows Application Development** - C# or JavaScript with Electron or React Native\n",
    "* **Linux Application Development**  - C/C++ w with Tcl/Tk or JavaScript with Electron or React Native\n",
    "\n",
    "\n",
    "## What About PyTorch?\n",
    "\n",
    "Technical folks love debates that can reach levels of fervor generally reserved for religion or politics. Python and TensorFlow are approaching this level of spirited competition.  There is no clear winner, at least at this point.  Why did I base this class on Keras/TensorFlow, as opposed to PyTorch?  There are two primary reasons.  The first reason is a fact; the second is my opinion.\n",
    "\n",
    "PyTorch was not available in early 2016 when I introduced/developed this course.\n",
    "PyTorch exposes lower-level details that would be distracting for an applications of deep learning course.\n",
    "I recommend being familiar with core deep learning techniques and being adaptable to switch between these two frameworks.\n",
    "\n",
    "## Where to From Here?\n",
    "\n",
    "\n",
    "So whats next? Here are some some ideas.\n",
    "\n",
    "* [Google CoLab Pro](https://colab.research.google.com/signup) - If you need more GPU power; but are not yet ready to buy a GPU of your own.\n",
    "* [TensorFlow Certification](https://www.tensorflow.org/certificate)\n",
    "* [Coursera](https://www.coursera.org/)\n",
    "\n",
    "I really hope that you have enjoyed this course.  If you have any suggestions for improvement or technology suggestions, please contact me. This course is always evolving, and I invite you to subscribe to my [YouTube channel](https://www.youtube.com/user/HeatonResearch) for my latest updates.  I also frequently post videos beyond the scope of this course, so the channel itself is a good next step. Thank you very much for your interest and focus on this course. Other social media links for me include:\n",
    "\n",
    "* [Jeff Heaton GitHub](https://github.com/jeffheaton)\n",
    "* [Jeff Heaton Twitter](https://twitter.com/jeffheaton)\n",
    "* [Jeff Heaton Medium](https://medium.com/@heatonresearch)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}