Skip to content

germanKoch/nlp_course

This branch is 1 commit ahead of, 40 commits behind yandexdataschool/nlp_course:2024.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

932581e · Jan 7, 2024
Sep 19, 2020
Oct 28, 2021
Sep 29, 2023
Jan 7, 2024
Oct 6, 2023
Oct 9, 2023
Oct 16, 2023
Nov 6, 2023
Nov 23, 2023
Dec 4, 2023
Dec 3, 2023
Dec 21, 2023
Sep 8, 2018
Mar 30, 2019
Sep 8, 2018
Oct 28, 2023

Repository files navigation

YSDA Natural Language Processing course

  • This is the 2023 version. For previous year' course materials, go to this branch
  • Lecture and seminar materials for each week are in ./week* folders, see README.md for materials and instructions
  • Any technical issues, ideas, bugs in course materials, contribution ideas - add an issue
  • Installing libraries and troubleshooting: this thread.

Syllabus

  • week01 Word Embeddings

    • Lecture: Word embeddings. Distributional semantics. Count-based (pre-neural) methods. Word2Vec: learn vectors. GloVe: count, then learn. Evaluation: intrinsic vs extrinsic. Analysis and Interpretability. Interactive lecture materials and more.
    • Seminar: Playing with word and sentence embeddings
    • Homework: Embedding-based machine translation system
  • week02 Text Classification

    • Lecture: Text classification: introduction and datasets. General framework: feature extractor + classifier. Classical approaches: Naive Bayes, MaxEnt (Logistic Regression), SVM. Neural Networks: General View, Convolutional Models, Recurrent Models. Practical Tips: Data Augmentation. Analysis and Interpretability. Interactive lecture materials and more.
    • Seminar: Text classification with convolutional NNs.
    • Homework: Statistical & neural text classification.
  • week03 Language Modeling

    • Lecture: Language Modeling: what does it mean? Left-to-right framework. N-gram language models. Neural Language Models: General View, Recurrent Models, Convolutional Models. Evaluation. Practical Tips: Weight Tying. Analysis and Interpretability. Interactive lecture materials and more.
    • Seminar: Build a N-gram language model from scratch
    • Homework: Neural LMs & smoothing in count-based models.
  • week04 Seq2seq and Attention

    • Lecture: Seq2seq Basics: Encoder-Decoder framework, Training, Simple Models, Inference (e.g., beam search). Attention: general, score functions, models. Transformer: self-attention, masked self-attention, multi-head attention; model architecture. Subword Segmentation (BPE). Analysis and Interpretability: functions of attention heads; probing for linguistic structure. Interactive lecture materials and more.
    • Seminar: Basic sequence to sequence model
    • Homework: Machine translation with attention
  • week05 Transfer Learning

    • Lecture: What is Transfer Learning? Great idea 1: From Words to Words-in-Context (CoVe, ELMo). Great idea 2: From Replacing Embeddings to Replacing Models (GPT, BERT). (A Bit of) Adaptors. Analysis and Interpretability. Interactive lecture materials and more.
    • Homework: fine-tuning a pre-trained BERT model
  • week06 LLMs and Prompting

    • Lecture: Scaling laws. Emergent abilities. Prompting (aka "in-context learning"): techiques that work; questioning whether model "understands" prompts. Hypotheses for why and how in-context learning works. Analysis and Interpretability.
    • Homework: manual prompt engneering and chain-of-thought reasoning
  • [week07] Transformer architecture and training

    • Lecture: training tips for transformers; the evolution of transformer architecture from Vaswani et al (2017) to modern LLMs; parameter-efficient fine-tuning (PEFT)
    • Homework: fine-tuning a large language model with PEFT algorithms

More TBA

Contributors & course staff

Course materials and teaching performed by

About

YSDA course in Natural Language Processing

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Jupyter Notebook 95.3%
  • Python 3.0%
  • HTML 1.5%
  • Dockerfile 0.1%
  • Cuda 0.1%
  • C++ 0.0%