05-neural-networks

Training a Neural Network

Slides

Objectives

• Learn steps to construct a "vanilla" neural network and train a classification model with ml5.js.
• Understand Neural Network architecture.
  • What is a perceptron?
  • What is a multi-layered perceptron?
  • What are activation functions?
• Understand the terminology of the training process.
  • Training
  • Learning rate
  • Epochs
  • Batch size
  • Loss
• Understand the difference between training and inference.
• Understand the distinction between different types of layers in a neural network.
  • What is a convolutional layer?
  • What is a pooling layer?
• Learn to train an image classifier with ml5.js, with and without convolutional layers.
• Revisit and examine the concepts of classification and regression as applied to real-time interaction.

Tools

• ml5.neuralNetwork Reference
  • ml5.neuralNetwork Source Code
  • TensorFlow.js Source Code

Artificial Neural Networks

Code Examples

• NeuralNetwork - Color Classifier
• NeuralNetwork - Mouse Gesture
• NeuralNetwork - Train and Save
• NeuralNetwork - Load Model

Supplemental Materials

• Chapter 10: Neural Network, Nature of Code by Daniel Shiffman.
• But what is a Neural Network? by 3Blue1Brown.
• The 7 steps of machine learning from Google Cloud Tech.

Video Tutorials

🚨 Note: ml5.js tutorials below were taught using an older version of ml5.js, refer to the ml5.js Resources Wiki page for more information. 🚨

Neural Network Concepts

• Neural Networks: Perceptron (2 parts) - video tutorial
• Neural Networks: Multilayer Perceptron (2 parts) - video tutorial.
• Neural Networks: Matrix Math by Daniel Shiffman.

ml5.js Neural Network

• ml5.js: Train Your Own Neural Network - video tutorial
• ml5.js: Save Neural Network Training Data - video tutorial
• ml5.js: Save Neural Network Model - video tutorial
• ml5: Neural Network Regression - video tutorial
• ml5.js: Pose Classification with PoseNet and ml5.neuralNetwork() - video tutorial
• ml5.js: Pose Regression with PoseNet and ml5.neuralNetwork() - video tutorial

Convolutional Neural Networks

Video Tutorials

• ml5.js: What is Convolutional Neural Network Part 1 - Filters - video tutorial by Daniel Shiffman.
• ml5.js: What is Convolutional Neural Network Part 2 - Max Pooling - video tutorial by Daniel Shiffman.- ml5.js: Training a Neural Network with Pixels as Input - video tutorial
• ml5.js: Training a Convolutional Neural Network for Image Classification - video tutorial

Datasets

• The Quick, Draw! Dataset from Google Creative Lab.
• The MNIST Dataset by Yann LeCun el al.

Supplemental Materials

• An Intuitive Explanation of Convolutional Neural Networks by Ujjwal Karn.
• What Neural Networks See by Gene Kogan.
• "Gradient-Based Learning Applied to Document Recognition" by Y. LeCun, L. Bottou, Y. Bengio, P. Haffner.
• How computers got shockingly good at recognizing images by Timothy B. Lee.
• A visual and intuitive understanding of deep learning, CNNs (0:00–9:40) by Octavio Good.
• Recognizing Human Facial Expressions With Machine Learning by Angelica Perez.
• Image Kernels Explained Visually by Victor Powell.
• Interactive Node-Link Visualizations of Convolutional Neural Networks by Adam W. Harley.
• Convolution Operation Demo by Deeplizard.

Related Projects

• Letter Collages by Deborah Schmidt.
• Face Tracking Experiment by Neil Mendoza.
• Faces of Humanity by La Boite à Tortue.
• Scribbling Speech by Xinyue Yang.
• How do you draw a circle? by Thu-Huong Ha and Nikhil Sonnad.
• Machine Learning for Visualization by Ian Johnson.

Code Examples

Working with Datasets

• Loading and Displaying Quick, Draw! Dataset
• Loading and Displaying MNIST Dataset

Convolution Neural Network Layers

• Convolutional Neural Network - Filter Demo
• Convolutional Neural Network - Max Pooling Demo

Training Image Classifiers

• Training a Doodle Classifier with Convolutional Layers
• Training a Handwritten Digit Classifier with Convolutional Layers
• Training a Webcam Image Classifier with Convolutional Layers
• Doodle Classification with DoodleNet (model trained by @yining1023)

ml5.js project

Due Thursday, Oct 10

1. Watch Machine Learning for Human Creative Practice by Dr. Rebecca Fiebrink at Eyeo 2018. Reflect on the question: How can machine learning support and expand people’s creative practices?

Write a response to this question and outline an initial idea for your project. Do you plan to build on a previous assignment or start something new? Post a link to your response on the ml5.js Project Step 1 Wiki page.

Due Thursday, Oct 17

2. Develop a p5.js sketch with ml5.js, using one or more of the models we’ve covered in the first half of the semester. Here are some ideas to get you started:

   • Create a simple movement-controlled game using hand, body, or face keypoints. Think about how different gestures might influence gameplay or character actions.
   • Design a musical instrument that responds to hand, body, or face keypoints, allowing gestures to control sound parameters such as pitch, volume, or effects. Explore creating melodies or soundscapes.
   • Build an augmented mirror or photo booth by overlaying p5.js elements on real-time video. Use segmentation or keypoint data to add visuals that respond to movements.
   • Make an interactive painting tool controlled by hand, body, or face keypoints. Experiment with effects like trailing particles or dynamic brushes that change shape and color based on gestures.
   • Chain models by using the output of one model as input to another with ml5.neuralNetwork(). For example, capture body or hand poses to create a custom classifier that drives an interaction. How could regression be applied to modify continuous elements like color gradients or speed?
   • Use transfer learning with Teachable Machine to train a custom image classifier on objects from your surroundings. Experiment with using these classifications to trigger interactions, visuals, or sounds in your sketch.

3. Document your project in a blog post and come prepared to share your work for ~5 minutes. You are not required to have a slide deck, however, you might find slides useful to help you plan and structure your demo. Add a link to your documentation to the ml5.js Project Step 2 Wiki page.

Consider including:

• Title
• Sketch link
• One sentence description
• Summary (250-500 words)
• Visual documentation such as a recorded screen capture / video / GIFs.
• Inspiration: How did you become interested in this idea? Quotes, photographs, products, projects, people, music, political events, social ills, etc.
• Process: How did you make this? What did you struggle with? What were you able to implement easily and what was difficult?
• Code references: What examples, tutorials, references did you use to create the project? (You must cite the source of any code you use in code comments. Please note the following additional expectations and guidelines at the bottom of this page.)
• Next steps: If you had more time to work on this, what would you do next?