index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Lil&#39;Log</title>
    <link>https://lilianweng.github.io/</link>
    <description>Recent content on Lil&#39;Log</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Thu, 28 Nov 2024 00:00:00 +0000</lastBuildDate><atom:link href="https://lilianweng.github.io/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Reward Hacking in Reinforcement Learning</title>
      <link>https://lilianweng.github.io/posts/2024-11-28-reward-hacking/</link>
      <pubDate>Thu, 28 Nov 2024 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2024-11-28-reward-hacking/</guid>
      <description>&lt;p&gt;Reward hacking occurs when a &lt;a href=&#34;(https://lilianweng.github.io/posts/2018-02-19-rl-overview/)&#34;&gt;reinforcement learning (RL)&lt;/a&gt; agent &lt;a href=&#34;https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/#exploitation-vs-exploration&#34;&gt;exploits&lt;/a&gt; flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.&lt;/p&gt;
&lt;p&gt;With the rise of &lt;a href=&#34;https://lilianweng.github.io/posts/2019-01-31-lm/&#34;&gt;language models&lt;/a&gt; generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user&amp;rsquo;s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Extrinsic Hallucinations in LLMs</title>
      <link>https://lilianweng.github.io/posts/2024-07-07-hallucination/</link>
      <pubDate>Sun, 07 Jul 2024 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2024-07-07-hallucination/</guid>
      <description>&lt;p&gt;Hallucination in large language models usually refers to the model generating unfaithful, fabricated, inconsistent, or nonsensical content. As a term, hallucination has been somewhat generalized to cases when the model makes mistakes. Here, I would like to narrow down the problem of hallucination to cases where the model output is fabricated and &lt;strong&gt;not grounded&lt;/strong&gt; by either the provided context or world knowledge.&lt;/p&gt;
&lt;p&gt;There are two types of hallucination:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;In-context hallucination: The model output should be consistent with the source content in context.&lt;/li&gt;
&lt;li&gt;Extrinsic hallucination: The model output should be grounded by the pre-training dataset. However, given the size of the pre-training dataset, it is too expensive to retrieve and identify conflicts per generation. If we consider the pre-training data corpus as a proxy for world knowledge, we essentially try to ensure the model output is factual and verifiable by external world knowledge. Equally importantly, when the model does not know about a fact, it should say so.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This post focuses on extrinsic hallucination. To avoid hallucination, LLMs need to be (1) factual and (2) acknowledge not knowing the answer when applicable.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Diffusion Models for Video Generation</title>
      <link>https://lilianweng.github.io/posts/2024-04-12-diffusion-video/</link>
      <pubDate>Fri, 12 Apr 2024 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2024-04-12-diffusion-video/</guid>
      <description>&lt;p&gt;&lt;a href=&#34;https://lilianweng.github.io/posts/2021-07-11-diffusion-models/&#34;&gt;Diffusion models&lt;/a&gt; have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task&amp;mdash;using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;It has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model.&lt;/li&gt;
&lt;li&gt;In comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs.&lt;/li&gt;
&lt;/ol&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;br/&gt;&lt;b&gt;
🥑 Required Pre-read: Please make sure you have read the previous blog on &lt;a href=&#34;https://lilianweng.github.io/posts/2021-07-11-diffusion-models/&#34;&gt;&amp;ldquo;What are Diffusion Models?&amp;rdquo;&lt;/a&gt; for image generation before continue here.
&lt;/b&gt;&lt;br/&gt;&lt;br/&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Thinking about High-Quality Human Data</title>
      <link>https://lilianweng.github.io/posts/2024-02-05-human-data-quality/</link>
      <pubDate>Mon, 05 Feb 2024 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2024-02-05-human-data-quality/</guid>
      <description>&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Special thank you to &lt;a href=&#34;https://scholar.google.com/citations?user=FRBObOwAAAAJ&amp;amp;hl=en&#34;&gt;Ian Kivlichan&lt;/a&gt; for many useful pointers (E.g. the 100+ year old Nature paper &amp;ldquo;Vox populi&amp;rdquo;) and nice feedback. 🙏 ]&lt;/span&gt;&lt;br/&gt;&lt;/p&gt;
&lt;p&gt;High-quality data is the fuel for modern data deep learning model training. Most of the task-specific labeled data comes from human annotation, such as classification task or &lt;a href=&#34;https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/#rl-fine-tuning-with-human-preferences&#34;&gt;RLHF&lt;/a&gt; labeling (which can be constructed as classification format) for LLM alignment training. Lots of ML techniques in the post can help with data quality, but fundamentally human data collection involves attention to details and careful execution. The community knows the value of high quality data, but somehow we have this subtle impression that “Everyone wants to do the model work, not the data work” (&lt;a href=&#34;https://dl.acm.org/doi/abs/10.1145/3411764.3445518&#34;&gt;Sambasivan et al. 2021&lt;/a&gt;).&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Adversarial Attacks on LLMs</title>
      <link>https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/</link>
      <pubDate>Wed, 25 Oct 2023 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/</guid>
      <description>&lt;p&gt;The use of large language models in the real world has strongly accelerated by the launch of ChatGPT. We (including my team at OpenAI, shoutout to them) have invested a lot of effort to build default safe behavior into the model during the alignment process (e.g. via &lt;a href=&#34;https://openai.com/research/learning-to-summarize-with-human-feedback&#34;&gt;RLHF&lt;/a&gt;). However, adversarial attacks or jailbreak prompts could potentially trigger the model to output something undesired.&lt;/p&gt;
&lt;p&gt;A large body of ground work on adversarial attacks is on images, and differently it operates in the continuous, high-dimensional space. Attacks for discrete data like text have been considered to be a lot more challenging, due to lack of direct gradient signals. My past post on &lt;a href=&#34;https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/&#34;&gt;Controllable Text Generation&lt;/a&gt; is quite relevant to this topic, as attacking LLMs is essentially to control the model to output a certain type of (unsafe) content.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>LLM Powered Autonomous Agents</title>
      <link>https://lilianweng.github.io/posts/2023-06-23-agent/</link>
      <pubDate>Fri, 23 Jun 2023 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2023-06-23-agent/</guid>
      <description>&lt;p&gt;Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as &lt;a href=&#34;https://github.com/Significant-Gravitas/Auto-GPT&#34;&gt;AutoGPT&lt;/a&gt;, &lt;a href=&#34;https://github.com/AntonOsika/gpt-engineer&#34;&gt;GPT-Engineer&lt;/a&gt; and &lt;a href=&#34;https://github.com/yoheinakajima/babyagi&#34;&gt;BabyAGI&lt;/a&gt;, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.&lt;/p&gt;
&lt;h1 id=&#34;agent-system-overview&#34;&gt;Agent System Overview&lt;/h1&gt;
&lt;p&gt;In a LLM-powered autonomous agent system, LLM functions as the agent&amp;rsquo;s brain, complemented by several key components:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Planning&lt;/strong&gt;
&lt;ul&gt;
&lt;li&gt;Subgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks.&lt;/li&gt;
&lt;li&gt;Reflection and refinement: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Memory&lt;/strong&gt;
&lt;ul&gt;
&lt;li&gt;Short-term memory: I would consider all the in-context learning (See &lt;a href=&#34;https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/&#34;&gt;Prompt Engineering&lt;/a&gt;) as utilizing short-term memory of the model to learn.&lt;/li&gt;
&lt;li&gt;Long-term memory: This provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tool use&lt;/strong&gt;
&lt;ul&gt;
&lt;li&gt;The agent learns to call external APIs for extra information that is missing from the model weights (often hard to change after pre-training), including current information, code execution capability, access to proprietary information sources and more.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;img src=&#34;agent-overview.png&#34; style=&#34;width: 100%;&#34; class=&#34;center&#34; /&gt;
&lt;figcaption&gt;Fig. 1. Overview of a LLM-powered autonomous agent system.&lt;/figcaption&gt;
&lt;h1 id=&#34;component-one-planning&#34;&gt;Component One: Planning&lt;/h1&gt;
&lt;p&gt;A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Prompt Engineering</title>
      <link>https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/</link>
      <pubDate>Wed, 15 Mar 2023 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/</guid>
      <description>&lt;p&gt;&lt;strong&gt;Prompt Engineering&lt;/strong&gt;, also known as &lt;strong&gt;In-Context Prompting&lt;/strong&gt;, refers to methods for how to communicate with LLM to steer its behavior for desired outcomes &lt;em&gt;without&lt;/em&gt; updating the model weights. It is an empirical science and the effect of prompt engineering methods can vary a lot among models, thus requiring heavy experimentation and heuristics.&lt;/p&gt;
&lt;p&gt;This post only focuses on prompt engineering for autoregressive language models, so nothing with Cloze tests, image generation or multimodality models. At its core, the goal of prompt engineering is about alignment and model steerability. Check my &lt;a href=&#34;https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/&#34;&gt;previous post&lt;/a&gt; on controllable text generation.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>The Transformer Family Version 2.0</title>
      <link>https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/</link>
      <pubDate>Fri, 27 Jan 2023 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/</guid>
      <description>&lt;p&gt;Many new Transformer architecture improvements have been proposed since my last post on &lt;a href=&#34;https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/&#34;&gt;&lt;ins&gt;&amp;ldquo;The Transformer Family&amp;rdquo;&lt;/ins&gt;&lt;/a&gt; about three years ago. Here I did a big refactoring and enrichment of that 2020 post &amp;mdash; restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length.&lt;/p&gt;
&lt;h1 id=&#34;notations&#34;&gt;Notations&lt;/h1&gt;
&lt;table&gt;
  &lt;thead&gt;
      &lt;tr&gt;
          &lt;th&gt;Symbol&lt;/th&gt;
          &lt;th&gt;Meaning&lt;/th&gt;
      &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
      &lt;tr&gt;
          &lt;td&gt;$d$&lt;/td&gt;
          &lt;td&gt;The model size / hidden state dimension / positional encoding size.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$h$&lt;/td&gt;
          &lt;td&gt;The number of heads in multi-head attention layer.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$L$&lt;/td&gt;
          &lt;td&gt;The segment length of input sequence.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$N$&lt;/td&gt;
          &lt;td&gt;The total number of attention layers in the model; not considering MoE.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{X} \in \mathbb{R}^{L \times d}$&lt;/td&gt;
          &lt;td&gt;The input sequence where each element has been mapped into an embedding vector of shape $d$, same as the model size.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{W}^k \in \mathbb{R}^{d \times d_k}$&lt;/td&gt;
          &lt;td&gt;The key weight matrix.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{W}^q \in \mathbb{R}^{d \times d_k}$&lt;/td&gt;
          &lt;td&gt;The query weight matrix.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{W}^v \in \mathbb{R}^{d \times d_v}$&lt;/td&gt;
          &lt;td&gt;The value weight matrix. Often we have $d_k = d_v = d$.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{W}^k_i, \mathbf{W}^q_i \in \mathbb{R}^{d \times d_k/h}; \mathbf{W}^v_i \in \mathbb{R}^{d \times d_v/h}$&lt;/td&gt;
          &lt;td&gt;The weight matrices per head.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{W}^o \in \mathbb{R}^{d_v \times d}$&lt;/td&gt;
          &lt;td&gt;The output weight matrix.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{Q} = \mathbf{X}\mathbf{W}^q \in \mathbb{R}^{L \times d_k}$&lt;/td&gt;
          &lt;td&gt;The query embedding inputs.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{K} = \mathbf{X}\mathbf{W}^k \in \mathbb{R}^{L \times d_k}$&lt;/td&gt;
          &lt;td&gt;The key embedding inputs.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{V} = \mathbf{X}\mathbf{W}^v \in \mathbb{R}^{L \times d_v}$&lt;/td&gt;
          &lt;td&gt;The value embedding inputs.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{q}_i, \mathbf{k}_i \in \mathbb{R}^{d_k}, \mathbf{v}_i \in \mathbb{R}^{d_v}$&lt;/td&gt;
          &lt;td&gt;Row vectors in query, key, value matrices, $\mathbf{Q}$, $\mathbf{K}$ and $\mathbf{V}$.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$S_i$&lt;/td&gt;
          &lt;td&gt;A collection of key positions for the $i$-th query $\mathbf{q}_i$ to attend to.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{A} \in \mathbb{R}^{L \times L}$&lt;/td&gt;
          &lt;td&gt;The self-attention matrix between a input sequence of lenght $L$ and itself. $\mathbf{A} = \text{softmax}(\mathbf{Q}\mathbf{K}^\top / \sqrt{d_k})$.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$a_{ij} \in \mathbf{A}$&lt;/td&gt;
          &lt;td&gt;The scalar attention score between query $\mathbf{q}_i$ and key $\mathbf{k}_j$.&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;$\mathbf{P} \in \mathbb{R}^{L \times d}$&lt;/td&gt;
          &lt;td&gt;position encoding matrix, where the $i$-th row $\mathbf{p}_i$ is the positional encoding for input $\mathbf{x}_i$.&lt;/td&gt;
      &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;h1 id=&#34;transformer-basics&#34;&gt;Transformer Basics&lt;/h1&gt;
&lt;p&gt;The &lt;strong&gt;Transformer&lt;/strong&gt; (which will be referred to as &amp;ldquo;vanilla Transformer&amp;rdquo; to distinguish it from other enhanced versions; &lt;a href=&#34;https://arxiv.org/abs/1706.03762&#34;&gt;Vaswani, et al., 2017&lt;/a&gt;) model has an encoder-decoder architecture, as commonly used in many &lt;a href=&#34;https://lilianweng.github.io/posts/2018-06-24-attention/#born-for-translation&#34;&gt;NMT&lt;/a&gt; models. Later simplified Transformer was shown to achieve great performance in language modeling tasks, like in encoder-only &lt;a href=&#34;https://lilianweng.github.io/posts/2019-01-31-lm/#bert&#34;&gt;BERT&lt;/a&gt; or decoder-only &lt;a href=&#34;https://lilianweng.github.io/posts/2019-01-31-lm/#openai-gpt&#34;&gt;GPT&lt;/a&gt;.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Large Transformer Model Inference Optimization</title>
      <link>https://lilianweng.github.io/posts/2023-01-10-inference-optimization/</link>
      <pubDate>Tue, 10 Jan 2023 10:00:00 -0700</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2023-01-10-inference-optimization/</guid>
      <description>&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2023-01-24: add a small section on &lt;a href=&#34;#distillation&#34;&gt;Distillation&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;&lt;/p&gt;
&lt;p&gt;Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Why is it hard to run inference for large transformer models?&lt;/strong&gt; Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (&lt;a href=&#34;https://arxiv.org/abs/2211.05102&#34;&gt;Pope et al. 2022&lt;/a&gt;):&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Some Math behind Neural Tangent Kernel</title>
      <link>https://lilianweng.github.io/posts/2022-09-08-ntk/</link>
      <pubDate>Thu, 08 Sep 2022 10:00:00 -0700</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2022-09-08-ntk/</guid>
      <description>&lt;p&gt;Neural networks are &lt;a href=&#34;https://lilianweng.github.io/posts/2019-03-14-overfit/&#34;&gt;well known&lt;/a&gt; to be over-parameterized and can often easily fit data with near-zero training loss with decent generalization performance on test dataset. Although all these parameters are initialized at random, the optimization process can consistently lead to similarly good outcomes. And this is true even when the number of model parameters exceeds the number of training data points.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Neural tangent kernel (NTK)&lt;/strong&gt; (&lt;a href=&#34;https://arxiv.org/abs/1806.07572&#34;&gt;Jacot et al. 2018&lt;/a&gt;) is a kernel to explain the evolution of neural networks during training via gradient descent. It leads to great insights into why neural networks with enough width can consistently converge to a global minimum when trained to minimize an empirical loss. In the post, we will do a deep dive into the motivation and definition of NTK, as well as the proof of a deterministic convergence at different initializations of neural networks with infinite width by characterizing NTK in such a setting.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Generalized Visual Language Models</title>
      <link>https://lilianweng.github.io/posts/2022-06-09-vlm/</link>
      <pubDate>Thu, 09 Jun 2022 15:10:30 -0700</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2022-06-09-vlm/</guid>
      <description>&lt;p&gt;Processing images to generate text, such as image captioning and visual question-answering, has been studied for years. Traditionally such systems rely on an object detection network as a vision encoder to capture visual features and then produce text via a text decoder. Given a large amount of existing literature, in this post, I would like to only focus on one approach for solving vision language tasks, which is to &lt;em&gt;extend pre-trained &lt;a href=&#34;https://lilianweng.github.io/posts/2019-01-31-lm/&#34;&gt;generalized language models&lt;/a&gt; to be capable of consuming visual signals&lt;/em&gt;.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Learning with not Enough Data Part 3: Data Generation</title>
      <link>https://lilianweng.github.io/posts/2022-04-15-data-gen/</link>
      <pubDate>Fri, 15 Apr 2022 15:10:30 -0700</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2022-04-15-data-gen/</guid>
      <description>&lt;p&gt;Here comes the Part 3 on learning with not enough data (Previous: &lt;a href=&#34;https://lilianweng.github.io/posts/2021-12-05-semi-supervised/&#34;&gt;Part 1&lt;/a&gt; and &lt;a href=&#34;https://lilianweng.github.io/posts/2022-02-20-active-learning/&#34;&gt;Part 2&lt;/a&gt;). Let’s consider two approaches for generating synthetic data for training.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Augmented data&lt;/strong&gt;. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a &lt;a href=&#34;https://lilianweng.github.io/posts/2021-05-31-contrastive/&#34;&gt;previous post&lt;/a&gt; on contrastive learning. For the sake of post completeness, I &lt;em&gt;duplicate&lt;/em&gt; the section on data augmentation here with some edits.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;New data&lt;/strong&gt;. Given few or even no data points, we can rely on powerful pretrained models to generate a number of &lt;em&gt;new&lt;/em&gt; data points. This is especially true in recent years given the fast progress in large pretrained &lt;a href=&#34;https://lilianweng.github.io/posts/2019-01-31-lm/&#34;&gt;language models (LM)&lt;/a&gt;. Few shot prompting is shown to be effective for LM to learn within context without extra training.&lt;/li&gt;
&lt;/ul&gt;
&lt;h1 id=&#34;data-augmentation&#34;&gt;Data Augmentation&lt;/h1&gt;
&lt;p&gt;The goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Learning with not Enough Data Part 2: Active Learning</title>
      <link>https://lilianweng.github.io/posts/2022-02-20-active-learning/</link>
      <pubDate>Sun, 20 Feb 2022 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2022-02-20-active-learning/</guid>
      <description>&lt;!-- The performance of supervised learning tasks improves with more high-quality labels available. However, it is expensive to collect a large number of labeled samples. Active learning is one paradigm to deal with not enough labeled data, when there are resources for labeling more data samples but under a limited budget. --&gt;
&lt;p&gt;This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Learning with not Enough Data Part 1: Semi-Supervised Learning</title>
      <link>https://lilianweng.github.io/posts/2021-12-05-semi-supervised/</link>
      <pubDate>Sun, 05 Dec 2021 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2021-12-05-semi-supervised/</guid>
      <description>&lt;!-- The performance of supervised learning tasks improves with more high-quality labels available. However, it is expensive to collect a large number of labeled samples. There are several paradigms in machine learning to deal with the scenario when the labels are scarce. Semi-supervised learning is one candidate, utilizing a large amount of unlabeled data conjunction with a small amount of labeled data. --&gt;
&lt;p&gt;When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>How to Train Really Large Models on Many GPUs?</title>
      <link>https://lilianweng.github.io/posts/2021-09-25-train-large/</link>
      <pubDate>Fri, 24 Sep 2021 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2021-09-25-train-large/</guid>
      <description>&lt;!-- How to train large and deep neural networks is challenging, as it demands a large amount of GPU memory and a long horizon of training time. This post reviews several popular training parallelism paradigms, as well as a variety of model architecture and memory saving designs to make it possible to train very large neural networks across a large number of GPUs. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2022-03-13: add &lt;a href=&#34;#ec&#34;&gt;expert choice routing&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2022-06-10]: &lt;a href=&#34;https://gregbrockman.com/&#34;&gt;Greg&lt;/a&gt; and I wrote a shorted and upgraded version of this post, published on OpenAI Blog: &lt;a href=&#34;https://openai.com/blog/techniques-for-training-large-neural-networks/&#34;&gt;&amp;ldquo;Techniques for Training Large Neural Networks&amp;rdquo;&lt;/a&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>What are Diffusion Models?</title>
      <link>https://lilianweng.github.io/posts/2021-07-11-diffusion-models/</link>
      <pubDate>Sun, 11 Jul 2021 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2021-07-11-diffusion-models/</guid>
      <description>&lt;!-- Diffusion models are a new type of generative models that are flexible enough to learn any arbitrarily complex data distribution while tractable to analytically evaluate the distribution. It has been shown recently that diffusion models can generate high-quality images and the performance is competitive to SOTA GAN. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2021-09-19: Highly recommend this blog post on &lt;a href=&#34;https://yang-song.github.io/blog/2021/score/&#34;&gt;score-based generative modeling&lt;/a&gt; by Yang Song (author of several key papers in the references)].&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2022-08-27: Added &lt;a href=&#34;#classifier-free-guidance&#34;&gt;classifier-free guidance&lt;/a&gt;, &lt;a href=&#34;#glide&#34;&gt;GLIDE&lt;/a&gt;, &lt;a href=&#34;#unclip&#34;&gt;unCLIP&lt;/a&gt; and &lt;a href=&#34;#imagen&#34;&gt;Imagen&lt;/a&gt;.&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2022-08-31: Added &lt;a href=&#34;#ldm&#34;&gt;latent diffusion model&lt;/a&gt;.&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2024-04-13: Added &lt;a href=&#34;#prog-distll&#34;&gt;progressive distillation&lt;/a&gt;, &lt;a href=&#34;#consistency&#34;&gt;consistency models&lt;/a&gt;, and the &lt;a href=&#34;#model-architecture&#34;&gt;Model Architecture section&lt;/a&gt;.&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Contrastive Representation Learning</title>
      <link>https://lilianweng.github.io/posts/2021-05-31-contrastive/</link>
      <pubDate>Mon, 31 May 2021 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2021-05-31-contrastive/</guid>
      <description>&lt;!-- The main idea of contrastive learning is to learn representations such that similar samples stay close to each other, while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised data and has been shown to achieve good performance on a variety of vision and language tasks. --&gt;
&lt;p&gt;The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in &lt;a href=&#34;https://lilianweng.github.io/posts/2019-11-10-self-supervised/&#34;&gt;self-supervised learning&lt;/a&gt;.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Reducing Toxicity in Language Models</title>
      <link>https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/</link>
      <pubDate>Sun, 21 Mar 2021 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/</guid>
      <description>&lt;!-- Toxicity prevents us from safely deploying powerful pretrained language models for real-world applications. To reduce toxicity in language models, in this post, we will delve into three aspects of the problem: training dataset collection, toxic content detection and model detoxification. --&gt;
&lt;p&gt;Large pretrained &lt;a href=&#34;https://lilianweng.github.io/posts/2019-01-31-lm/&#34;&gt;language models&lt;/a&gt; are trained over a sizable collection of online data. They unavoidably acquire certain toxic behavior and biases from the Internet. Pretrained language models are very powerful and have shown great success in many NLP tasks. However, to safely deploy them for practical real-world applications demands a strong safety control over the model generation process.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Controllable Neural Text Generation</title>
      <link>https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/</link>
      <pubDate>Sat, 02 Jan 2021 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/</guid>
      <description>&lt;!-- The modern language model with SOTA results on many NLP tasks is trained on large scale free text on the Internet. It is challenging to steer such a model to generate content with desired attributes. Although still not perfect, there are several approaches for controllable text generation, such as guided or learned decoding strategy, smart prompt design, or fine-tuning the model with various methods. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.]&lt;/span&gt;
&lt;br /&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the &lt;a href=&#34;#gradient-based-search&#34;&gt;&amp;ldquo;prompt design&amp;rdquo;&lt;/a&gt; section.]&lt;/span&gt;
&lt;br /&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2021-09-19: Add &lt;a href=&#34;##unlikelihood-training&#34;&gt;&amp;ldquo;unlikelihood training&amp;rdquo;&lt;/a&gt;.]&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>How to Build an Open-Domain Question Answering System?</title>
      <link>https://lilianweng.github.io/posts/2020-10-29-odqa/</link>
      <pubDate>Thu, 29 Oct 2020 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2020-10-29-odqa/</guid>
      <description>&lt;!-- A model that is capable of answering any question with regard to factual knowledge can enable many useful applications. This post delves into how we can build an Open-Domain Question Answering (ODQA) system, assuming we have access to a powerful pretrained language model. Both closed-book and open-book approachs are discussed. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2020-11-12: add &lt;a href=&#34;#openai-api-example&#34;&gt;an example&lt;/a&gt; on closed-book factual QA using OpenAI API (beta).&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;A model that can answer any question with regard to factual knowledge can lead to many useful and practical applications, such as working as a chatbot or an AI assistant🤖. In this post, we will review several common approaches for building such an open-domain question answering system.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Neural Architecture Search</title>
      <link>https://lilianweng.github.io/posts/2020-08-06-nas/</link>
      <pubDate>Thu, 06 Aug 2020 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2020-08-06-nas/</guid>
      <description>&lt;!-- Neural Architecture Search (NAS) automates network architecture engineering. It aims to learn a network topology that can achieve best performance on a certain task. By dissecting the methods for NAS into three components: search space, search algorithm and child model evolution strategy, this post reviews many interesting ideas for better, faster and more cost-efficient automatic neural architecture search. --&gt;
&lt;p&gt;Although most popular and successful model architectures are designed by human experts, it doesn&amp;rsquo;t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Exploration Strategies in Deep Reinforcement Learning</title>
      <link>https://lilianweng.github.io/posts/2020-06-07-exploration-drl/</link>
      <pubDate>Sun, 07 Jun 2020 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2020-06-07-exploration-drl/</guid>
      <description>&lt;!-- Exploitation versus exploration is a critical topic in reinforcement learning. This post introduces several common approaches for better exploration in Deep RL. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2020-06-17: Add &lt;a href=&#34;#exploration-via-disagreement&#34;&gt;&amp;ldquo;exploration via disagreement&amp;rdquo;&lt;/a&gt; in the &amp;ldquo;Forward Dynamics&amp;rdquo; &lt;a href=&#34;#forward-dynamics&#34;&gt;section&lt;/a&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/&#34;&gt;Exploitation versus exploration&lt;/a&gt; is a critical topic in Reinforcement Learning. We&amp;rsquo;d like the RL agent to find the best solution as fast as possible. However, in the meantime, committing to solutions too quickly without enough exploration sounds pretty bad, as it could lead to local minima or total failure. Modern &lt;a href=&#34;https://lilianweng.github.io/posts/2018-02-19-rl-overview/&#34;&gt;RL&lt;/a&gt; &lt;a href=&#34;https://lilianweng.github.io/posts/2018-04-08-policy-gradient/&#34;&gt;algorithms&lt;/a&gt; that optimize for the best returns can achieve good exploitation quite efficiently, while exploration remains more like an open topic.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>The Transformer Family</title>
      <link>https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/</link>
      <pubDate>Tue, 07 Apr 2020 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/</guid>
      <description>&lt;!-- Inspired by recent progress on various enhanced versions of Transformer models, this post presents how the vanilla Transformer can be improved for longer-term attention span, less memory and computation consumption, RL task solving, etc. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on &lt;mark&gt;&lt;strong&gt;2023-01-27&lt;/strong&gt;&lt;/mark&gt;: After almost three years, I did a big refactoring update of this post to incorporate a bunch of new Transformer models since 2020. The enhanced version of this post is here: &lt;a href=&#34;https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/&#34;&gt;&lt;mark&gt;&lt;b&gt;The Transformer Family Version 2.0&lt;/b&gt;&lt;/mark&gt;&lt;/a&gt;. Please refer to that post on this topic.]&lt;/span&gt;
&lt;br/&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Curriculum for Reinforcement Learning</title>
      <link>https://lilianweng.github.io/posts/2020-01-29-curriculum-rl/</link>
      <pubDate>Wed, 29 Jan 2020 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2020-01-29-curriculum-rl/</guid>
      <description>&lt;!-- A curriculum is an efficient tool for humans to progressively learn from simple concepts to hard problems. It breaks down complex knowledge by providing a sequence of learning steps of increasing difficulty. In this post, we will examine how the idea of curriculum can help reinforcement learning models learn to solve complicated tasks. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2020-02-03: mentioning &lt;a href=&#34;#pcg&#34;&gt;PCG&lt;/a&gt; in the &amp;ldquo;Task-Specific Curriculum&amp;rdquo; section.&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-02-04: Add a new &lt;a href=&#34;#curriculum-through-distillation&#34;&gt;&amp;ldquo;curriculum through distillation&amp;rdquo;&lt;/a&gt; section.&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Self-Supervised Representation Learning</title>
      <link>https://lilianweng.github.io/posts/2019-11-10-self-supervised/</link>
      <pubDate>Sun, 10 Nov 2019 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2019-11-10-self-supervised/</guid>
      <description>&lt;!-- Self-supervised learning opens up a huge opportunity for better utilizing unlabelled data, while learning in a supervised learning manner. This post covers many interesting ideas of self-supervised learning tasks on images, videos, and control problems. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2020-01-09: add a new section on &lt;a href=&#34;#contrastive-predictive-coding&#34;&gt;Contrastive Predictive Coding&lt;/a&gt;].&lt;/span&gt;
&lt;br/&gt;
&lt;del&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2020-04-13: add a &amp;ldquo;Momentum Contrast&amp;rdquo; section on MoCo, SimCLR and CURL.]&lt;/span&gt;&lt;/del&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-07-08: add a &lt;a href=&#34;#bisimulation&#34;&gt;&amp;ldquo;Bisimulation&amp;rdquo;&lt;/a&gt; section on DeepMDP and DBC.]&lt;/span&gt;
&lt;br/&gt;
&lt;del&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2020-09-12: add &lt;a href=&#34;https://lilianweng.github.io/posts/2021-05-31-contrastive/#moco--moco-v2&#34;&gt;MoCo V2&lt;/a&gt; and &lt;a href=&#34;https://lilianweng.github.io/posts/2021-05-31-contrastive/#byol&#34;&gt;BYOL&lt;/a&gt; in the &amp;ldquo;Momentum Contrast&amp;rdquo; section.]&lt;/span&gt;&lt;/del&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2021-05-31: remove section on &amp;ldquo;Momentum Contrast&amp;rdquo; and add a pointer to a full post on &lt;a href=&#34;https://lilianweng.github.io/posts/2021-05-31-contrastive/&#34;&gt;&amp;ldquo;Contrastive Representation Learning&amp;rdquo;&lt;/a&gt;]&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Evolution Strategies</title>
      <link>https://lilianweng.github.io/posts/2019-09-05-evolution-strategies/</link>
      <pubDate>Thu, 05 Sep 2019 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2019-09-05-evolution-strategies/</guid>
      <description>&lt;!-- Gradient descent is not the only option when learning optimal model parameters. Evolution Strategies (ES)  works out well in the cases where we don&#39;t know the precise analytic form of an objective function or cannot compute the gradients directly. This post dives into several classic ES methods, as well as how ES can be used in deep reinforcement learning. --&gt;
&lt;p&gt;Stochastic gradient descent is a universal choice for optimizing deep learning models. However, it is not the only option. With black-box optimization algorithms, you can evaluate a target function $f(x): \mathbb{R}^n \to \mathbb{R}$, even when you don&amp;rsquo;t know the precise analytic form of $f(x)$ and thus cannot compute gradients or the Hessian matrix. Examples of black-box optimization methods include &lt;a href=&#34;https://en.wikipedia.org/wiki/Simulated_annealing&#34;&gt;Simulated Annealing&lt;/a&gt;, &lt;a href=&#34;https://en.wikipedia.org/wiki/Hill_climbing&#34;&gt;Hill Climbing&lt;/a&gt; and &lt;a href=&#34;https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method&#34;&gt;Nelder-Mead method&lt;/a&gt;.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Meta Reinforcement Learning</title>
      <link>https://lilianweng.github.io/posts/2019-06-23-meta-rl/</link>
      <pubDate>Sun, 23 Jun 2019 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2019-06-23-meta-rl/</guid>
      <description>&lt;!-- Meta-RL is meta-learning on reinforcement learning tasks. After trained over a distribution of tasks, the agent is able to solve a new task by developing a new RL algorithm with its internal activity dynamics. This post starts with the origin of meta-RL and then dives into three key components of meta-RL. --&gt;
&lt;p&gt;In my earlier post on &lt;a href=&#34;https://lilianweng.github.io/posts/2018-11-30-meta-learning/&#34;&gt;meta-learning&lt;/a&gt;, the problem is mainly defined in the context of few-shot classification. Here I would like to explore more into cases when we try to &amp;ldquo;meta-learn&amp;rdquo; &lt;a href=&#34;https://lilianweng.github.io/posts/2018-02-19-rl-overview/&#34;&gt;Reinforcement Learning (RL)&lt;/a&gt; tasks by developing an agent that can solve unseen tasks fast and efficiently.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Domain Randomization for Sim2Real Transfer</title>
      <link>https://lilianweng.github.io/posts/2019-05-05-domain-randomization/</link>
      <pubDate>Sun, 05 May 2019 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2019-05-05-domain-randomization/</guid>
      <description>&lt;!-- If a model or policy is mainly trained in a simulator but expected to work on a real robot, it would surely face the sim2real gap. *Domain Randomization* (DR) is a simple but powerful idea of closing this gap by randomizing properties of the training environment. --&gt;
&lt;p&gt;In Robotics, one of the hardest problems is how to make your model transfer to the real world. Due to the sample inefficiency of deep RL algorithms and the cost of data collection on real robots, we often need to train models in a simulator which theoretically provides an infinite amount of data. However, the reality gap between the simulator and the physical world often leads to failure when working with physical robots. The gap is triggered by an inconsistency between physical parameters (i.e. friction, kp, damping, mass, density) and, more fatally, the incorrect physical modeling (i.e. collision between soft surfaces).&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Are Deep Neural Networks Dramatically Overfitted?</title>
      <link>https://lilianweng.github.io/posts/2019-03-14-overfit/</link>
      <pubDate>Thu, 14 Mar 2019 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2019-03-14-overfit/</guid>
      <description>&lt;!-- If you are, like me, confused by why deep neural networks can generalize to out-of-sample data points without drastic overfitting, keep on reading. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2019-05-27: add the &lt;a href=&#34;#the-lottery-ticket-hypothesis&#34;&gt;section&lt;/a&gt; on Lottery Ticket Hypothesis.]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;If you are like me, entering into the field of deep learning with experience in traditional machine learning, you may often ponder over this question: Since a typical deep neural network has so many parameters and training error can easily be perfect, it should surely suffer from substantial overfitting. How could it be ever generalized to out-of-sample data points?&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Generalized Language Models</title>
      <link>https://lilianweng.github.io/posts/2019-01-31-lm/</link>
      <pubDate>Thu, 31 Jan 2019 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2019-01-31-lm/</guid>
      <description>&lt;!-- As a follow up of word embedding post, we will discuss the models on learning contextualized word vectors, as well as the new trend in large unsupervised pre-trained language models which have achieved amazing SOTA results on a variety of language tasks. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2019-02-14: add &lt;a href=&#34;#ulmfit&#34;&gt;ULMFiT&lt;/a&gt; and &lt;a href=&#34;#gpt-2&#34;&gt;GPT-2&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-02-29: add &lt;a href=&#34;#albert&#34;&gt;ALBERT&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-10-25: add &lt;a href=&#34;#roberta&#34;&gt;RoBERTa&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-12-13: add &lt;a href=&#34;#t5&#34;&gt;T5&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-12-30: add &lt;a href=&#34;#gpt-3&#34;&gt;GPT-3&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2021-11-13: add &lt;a href=&#34;#xlnet&#34;&gt;XLNet&lt;/a&gt;, &lt;a href=&#34;#bart&#34;&gt;BART&lt;/a&gt; and &lt;a href=&#34;#electra&#34;&gt;ELECTRA&lt;/a&gt;; Also updated the &lt;a href=&#34;#summary&#34;&gt;Summary&lt;/a&gt; section.]&lt;/span&gt;&lt;/p&gt;
&lt;br /&gt;
&lt;img src=&#34;elmo-and-bert.png&#34; style=&#34;width: 60%;&#34; class=&#34;center&#34; /&gt;
&lt;figcaption&gt;Fig. 0. I guess they are Elmo &amp; Bert? (Image source: &lt;a href=&#34;https://www.youtube.com/watch?v=l5einDQ-Ttc&#34; target=&#34;_blank&#34;&gt;here&lt;/a&gt;)&lt;/figcaption&gt;
&lt;p&gt;We have seen amazing progress in NLP in 2018. Large-scale pre-trained language modes like &lt;a href=&#34;https://blog.openai.com/language-unsupervised/&#34;&gt;OpenAI GPT&lt;/a&gt; and &lt;a href=&#34;https://arxiv.org/abs/1810.04805&#34;&gt;BERT&lt;/a&gt; have achieved great performance on a variety of language tasks using generic model architectures. The idea is similar to how ImageNet classification pre-training helps many vision tasks (*). Even better than vision classification pre-training, this simple and powerful approach in NLP does not require labeled data for pre-training, allowing us to experiment with increased training scale, up to our very limit.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Object Detection Part 4: Fast Detection Models</title>
      <link>https://lilianweng.github.io/posts/2018-12-27-object-recognition-part-4/</link>
      <pubDate>Thu, 27 Dec 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-12-27-object-recognition-part-4/</guid>
      <description>&lt;!-- Part 4 of the &#34;Object Detection for Dummies&#34; series focuses on one-stage models for fast detection, including SSD, RetinaNet, and models in the YOLO family. These models skip the explicit region proposal stage but apply the detection directly on dense sampled areas. --&gt;
&lt;p&gt;In &lt;a href=&#34;https://lilianweng.github.io/posts/2017-12-31-object-recognition-part-3/&#34;&gt;Part 3&lt;/a&gt;, we have reviewed models in the R-CNN family. All of them are region-based object detection algorithms. They can achieve high accuracy but could be too slow for certain applications such as autonomous driving. In Part 4, we only focus on fast object detection models, including SSD, RetinaNet, and models in the YOLO family.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Meta-Learning: Learning to Learn Fast</title>
      <link>https://lilianweng.github.io/posts/2018-11-30-meta-learning/</link>
      <pubDate>Fri, 30 Nov 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-11-30-meta-learning/</guid>
      <description>&lt;!-- Meta-learning, also known as &#34;learning to learn&#34;, intends to design models that can learn new skills or adapt to new environments rapidly with a few training examples. There are three common approaches: 1) learn an efficient distance metric (metric-based); 2) use (recurrent) network with external or internal memory (model-based); 3) optimize the model parameters explicitly for fast learning (optimization-based). --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2019-10-01: thanks to Tianhao, we have this post translated in &lt;a href=&#34;https://wei-tianhao.github.io/blog/2019/09/17/meta-learning.html&#34;&gt;Chinese&lt;/a&gt;!]&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Flow-based Deep Generative Models</title>
      <link>https://lilianweng.github.io/posts/2018-10-13-flow-models/</link>
      <pubDate>Sat, 13 Oct 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-10-13-flow-models/</guid>
      <description>&lt;!-- In this post, we are looking into the third type of generative models: flow-based generative models. Different from GAN and VAE, they explicitly learn the probability density function of the input data. --&gt;
&lt;p&gt;So far, I&amp;rsquo;ve written about two types of generative models, &lt;a href=&#34;https://lilianweng.github.io/posts/2017-08-20-gan/&#34;&gt;GAN&lt;/a&gt; and &lt;a href=&#34;https://lilianweng.github.io/posts/2018-08-12-vae/&#34;&gt;VAE&lt;/a&gt;. Neither of them explicitly learns the probability density function of real data, $p(\mathbf{x})$ (where $\mathbf{x} \in \mathcal{D}$) &amp;mdash; because it is really hard! Taking the generative model with latent variables as an example, $p(\mathbf{x}) = \int p(\mathbf{x}\vert\mathbf{z})p(\mathbf{z})d\mathbf{z}$ can hardly be calculated as it is intractable to go through all possible values of the latent code $\mathbf{z}$.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>From Autoencoder to Beta-VAE</title>
      <link>https://lilianweng.github.io/posts/2018-08-12-vae/</link>
      <pubDate>Sun, 12 Aug 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-08-12-vae/</guid>
      <description>&lt;!-- Autocoders are a family of neural network models aiming to learn compressed latent variables of high-dimensional data. Starting from the basic autocoder model, this post reviews several variations, including denoising, sparse, and contractive autoencoders, and then Variational Autoencoder (VAE) and its modification beta-VAE. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2019-07-18: add a section on &lt;a href=&#34;#vq-vae-and-vq-vae-2&#34;&gt;VQ-VAE &amp;amp; VQ-VAE-2&lt;/a&gt;.]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2019-07-26: add a section on &lt;a href=&#34;#td-vae&#34;&gt;TD-VAE&lt;/a&gt;.]&lt;/span&gt;
&lt;br/&gt;&lt;/p&gt;
&lt;p&gt;Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer in the middle (oops, this is probably not true for &lt;a href=&#34;#vae-variational-autoencoder&#34;&gt;Variational Autoencoder&lt;/a&gt;, and we will investigate it in details in later sections). A nice byproduct is dimension reduction: the bottleneck layer captures a compressed latent encoding. Such a low-dimensional representation can be used as en embedding vector in various applications (i.e. search), help data compression, or reveal the underlying data generative factors.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Attention? Attention!</title>
      <link>https://lilianweng.github.io/posts/2018-06-24-attention/</link>
      <pubDate>Sun, 24 Jun 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-06-24-attention/</guid>
      <description>&lt;!-- Attention has been a fairly popular concept and a useful tool in the deep learning community in recent years. In this post, we are gonna look into how attention was invented, and various attention mechanisms and models, such as transformer and SNAIL. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2018-10-28: Add &lt;a href=&#34;#pointer-network&#34;&gt;Pointer Network&lt;/a&gt; and the &lt;a href=&#34;https://github.com/lilianweng/transformer-tensorflow&#34;&gt;link&lt;/a&gt; to my implementation of Transformer.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2018-11-06: Add a &lt;a href=&#34;https://github.com/lilianweng/transformer-tensorflow&#34;&gt;link&lt;/a&gt; to the implementation of Transformer model.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2018-11-18: Add &lt;a href=&#34;#neural-turing-machines&#34;&gt;Neural Turing Machines&lt;/a&gt;.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2019-07-18: Correct the mistake on using the term &amp;ldquo;self-attention&amp;rdquo; when introducing the &lt;a href=&#34;https://arxiv.org/abs/1502.03044&#34;&gt;show-attention-tell&lt;/a&gt; paper; moved it to &lt;a href=&#34;#self-attention&#34;&gt;Self-Attention&lt;/a&gt; section.]&lt;/span&gt;&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-04-07: A follow-up post on improved Transformer models is &lt;a href=&#34;https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/&#34;&gt;here&lt;/a&gt;.]&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Implementing Deep Reinforcement Learning Models with Tensorflow &#43; OpenAI Gym</title>
      <link>https://lilianweng.github.io/posts/2018-05-05-drl-implementation/</link>
      <pubDate>Sat, 05 May 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-05-05-drl-implementation/</guid>
      <description>&lt;!-- Let&#39;s see how to implement a number of classic deep reinforcement learning models in code. --&gt;
&lt;p&gt;The full implementation is available in &lt;a href=&#34;https://github.com/lilianweng/deep-reinforcement-learning-gym&#34;&gt;lilianweng/deep-reinforcement-learning-gym&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;In the previous two posts, I have introduced the algorithms of many deep reinforcement learning models. Now it is the time to get our hands dirty and practice how to implement the models in the wild. The implementation is gonna be built in Tensorflow and OpenAI &lt;a href=&#34;https://github.com/openai/gym&#34;&gt;gym&lt;/a&gt; environment. The full version of the code in this tutorial is available in &lt;a href=&#34;https://github.com/lilianweng/deep-reinforcement-learning-gym&#34;&gt;[lilian/deep-reinforcement-learning-gym]&lt;/a&gt;.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Policy Gradient Algorithms</title>
      <link>https://lilianweng.github.io/posts/2018-04-08-policy-gradient/</link>
      <pubDate>Sun, 08 Apr 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-04-08-policy-gradient/</guid>
      <description>&lt;!-- Abstract: In this post, we are going to look deep into policy gradient, why it works, and many new policy gradient algorithms proposed in recent years: vanilla policy gradient, actor-critic, off-policy actor-critic, A3C, A2C, DPG, DDPG, D4PG, MADDPG, TRPO, PPO, ACER, ACTKR, SAC, TD3 &amp; SVPG. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2018-06-30: add two new policy gradient methods, &lt;a href=&#34;#sac&#34;&gt;SAC&lt;/a&gt; and &lt;a href=&#34;#d4pg&#34;&gt;D4PG&lt;/a&gt;.]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2018-09-30: add a new policy gradient method, &lt;a href=&#34;#td3&#34;&gt;TD3&lt;/a&gt;.]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2019-02-09: add &lt;a href=&#34;#sac-with-automatically-adjusted-temperature&#34;&gt;SAC with automatically adjusted temperature&lt;/a&gt;].&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2019-06-26: Thanks to Chanseok, we have a version of this post in &lt;a href=&#34;https://talkingaboutme.tistory.com/entry/RL-Policy-Gradient-Algorithms&#34;&gt;Korean&lt;/a&gt;].&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2019-09-12: add a new policy gradient method &lt;a href=&#34;#svpg&#34;&gt;SVPG&lt;/a&gt;.]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2019-12-22: add a new policy gradient method &lt;a href=&#34;#impala&#34;&gt;IMPALA&lt;/a&gt;.]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2020-10-15: add a new policy gradient method &lt;a href=&#34;#ppg&#34;&gt;PPG&lt;/a&gt; &amp;amp; some new discussion in &lt;a href=&#34;#ppo&#34;&gt;PPO&lt;/a&gt;.]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2021-09-19: Thanks to Wenhao &amp;amp; 爱吃猫的鱼, we have this post in &lt;a href=&#34;https://tomaxent.com/2019/04/14/%E7%AD%96%E7%95%A5%E6%A2%AF%E5%BA%A6%E6%96%B9%E6%B3%95/&#34;&gt;Chinese1&lt;/a&gt; &amp;amp; &lt;a href=&#34;https://paperexplained.cn/articles/article/detail/31/&#34;&gt;Chinese2&lt;/a&gt;].&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>A (Long) Peek into Reinforcement Learning</title>
      <link>https://lilianweng.github.io/posts/2018-02-19-rl-overview/</link>
      <pubDate>Mon, 19 Feb 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-02-19-rl-overview/</guid>
      <description>&lt;!-- In this post, we are gonna briefly go over the field of Reinforcement Learning (RL), from fundamental concepts to classic algorithms. Hopefully, this review is helpful enough so that newbies would not get lost in specialized terms and jargons while starting. [WARNING] This is a long read. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2020-09-03: Updated the algorithm of &lt;a href=&#34;#sarsa-on-policy-td-control&#34;&gt;SARSA&lt;/a&gt; and &lt;a href=&#34;#q-learning-off-policy-td-control&#34;&gt;Q-learning&lt;/a&gt; so that the difference is more pronounced.&lt;/span&gt;
&lt;br /&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2021-09-19: Thanks to 爱吃猫的鱼, we have this post in &lt;a href=&#34;https://paperexplained.cn/articles/article/detail/33/&#34;&gt;Chinese&lt;/a&gt;].&lt;/span&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>The Multi-Armed Bandit Problem and Its Solutions</title>
      <link>https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/</link>
      <pubDate>Tue, 23 Jan 2018 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/</guid>
      <description>&lt;!-- The multi-armed bandit problem is a class example to demonstrate the exploration versus exploitation dilemma. This post introduces the bandit problem and how to solve it using different exploration strategies. --&gt;
&lt;p&gt;The algorithms are implemented for Bernoulli bandit in &lt;a href=&#34;http://github.com/lilianweng/multi-armed-bandit&#34;&gt;lilianweng/multi-armed-bandit&lt;/a&gt;.&lt;/p&gt;
&lt;h1 id=&#34;exploitation-vs-exploration&#34;&gt;Exploitation vs Exploration&lt;/h1&gt;
&lt;p&gt;The exploration vs exploitation dilemma exists in many aspects of our life. Say, your favorite restaurant is right around the corner. If you go there every day, you would be confident of what you will get, but miss the chances of discovering an even better option. If you try new places all the time, very likely you are gonna have to eat unpleasant food from time to time. Similarly, online advisors try to balance between the known most attractive ads and the new ads that might be even more successful.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Object Detection for Dummies Part 3: R-CNN Family</title>
      <link>https://lilianweng.github.io/posts/2017-12-31-object-recognition-part-3/</link>
      <pubDate>Sun, 31 Dec 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-12-31-object-recognition-part-3/</guid>
      <description>&lt;!-- In Part 3, we would examine four object detection models: R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN. These models are highly related and the new versions show great speed improvement compared to the older ones. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2018-12-20: Remove YOLO here. Part 4 will cover multiple fast object detection algorithms, including YOLO.]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2018-12-27: Add &lt;a href=&#34;#bounding-box-regression&#34;&gt;bbox regression&lt;/a&gt; and &lt;a href=&#34;#common-tricks&#34;&gt;tricks&lt;/a&gt; sections for R-CNN.]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;In the series of &amp;ldquo;Object Detection for Dummies&amp;rdquo;, we started with basic concepts in image processing, such as gradient vectors and HOG, in &lt;a href=&#34;https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/&#34;&gt;Part 1&lt;/a&gt;. Then we introduced classic convolutional neural network architecture designs for classification and pioneer models for object recognition, Overfeat and DPM, in &lt;a href=&#34;https://lilianweng.github.io/posts/2017-12-15-object-recognition-part-2/&#34;&gt;Part 2&lt;/a&gt;. In the third post of this series, we are about to review a set of models in the R-CNN (&amp;ldquo;Region-based CNN&amp;rdquo;) family.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Object Detection for Dummies Part 2: CNN, DPM and Overfeat</title>
      <link>https://lilianweng.github.io/posts/2017-12-15-object-recognition-part-2/</link>
      <pubDate>Fri, 15 Dec 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-12-15-object-recognition-part-2/</guid>
      <description>&lt;!-- Part 2 introduces several classic convolutional neural work architecture designs for image classification (AlexNet, VGG, ResNet), as well as DPM (Deformable Parts Model) and Overfeat models for object recognition. --&gt;
&lt;p&gt;&lt;a href=&#34;https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/&#34;&gt;Part 1&lt;/a&gt; of the &amp;ldquo;Object Detection for Dummies&amp;rdquo; series introduced: (1) the concept of image gradient vector and how HOG algorithm summarizes the information across all the gradient vectors in one image; (2) how the image segmentation algorithm works to detect regions that potentially contain objects; (3) how the Selective Search algorithm refines the outcomes of image segmentation for better region proposal.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Object Detection for Dummies Part 1: Gradient Vector, HOG, and SS</title>
      <link>https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/</link>
      <pubDate>Sun, 29 Oct 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/</guid>
      <description>&lt;!-- In this series of posts on &#34;Object Detection for Dummies&#34;, we will go through several basic concepts, algorithms, and popular deep learning models for image processing and objection detection. Hopefully, it would be a good read for people with no experience in this field but want to learn more. The Part 1 introduces the concept of Gradient Vectors, the HOG (Histogram of Oriented Gradients) algorithm, and Selective Search for image segmentation. --&gt;
&lt;p&gt;I&amp;rsquo;ve never worked in the field of computer vision and has no idea how the magic could work when an autonomous car is configured to tell apart a stop sign from a pedestrian in a red hat. To motivate myself to look into the maths behind object recognition and detection algorithms, I&amp;rsquo;m writing a few posts on this topic &amp;ldquo;Object Detection for Dummies&amp;rdquo;. This post, part 1, starts with super rudimentary concepts in image processing and a few methods for image segmentation. Nothing related to deep neural networks yet. Deep learning models for object detection and recognition will be discussed in &lt;a href=&#34;https://lilianweng.github.io/posts/2017-12-15-object-recognition-part-2/&#34;&gt;Part 2&lt;/a&gt; and &lt;a href=&#34;https://lilianweng.github.io/posts/2017-12-31-object-recognition-part-3/&#34;&gt;Part 3&lt;/a&gt;.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Learning Word Embedding</title>
      <link>https://lilianweng.github.io/posts/2017-10-15-word-embedding/</link>
      <pubDate>Sun, 15 Oct 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-10-15-word-embedding/</guid>
      <description>&lt;!-- Word embedding is a dense representation of words in the form of numeric vectors. It can be learned using a variety of language models. The word embedding representation is able to reveal many hidden relationships between words. For example, vector(&#34;cat&#34;) - vector(&#34;kitten&#34;) is similar to vector(&#34;dog&#34;) - vector(&#34;puppy&#34;). This post introduces several models for learning word embedding and how their loss functions are designed for the purpose. --&gt;
&lt;p&gt;Human vocabulary comes in free text. In order to make a machine learning model understand and process the natural language, we need to transform the free-text words into numeric values. One of the simplest transformation approaches is to do a one-hot encoding in which each distinct word stands for one dimension of the resulting vector and a binary value indicates whether the word presents (1) or not (0).&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Anatomize Deep Learning with Information Theory</title>
      <link>https://lilianweng.github.io/posts/2017-09-28-information-bottleneck/</link>
      <pubDate>Thu, 28 Sep 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-09-28-information-bottleneck/</guid>
      <description>&lt;!-- This post is a summary of Prof Naftali Tishby&#39;s recent talk on &#34;Information Theory in Deep Learning&#34;. It presented how to apply the information theory to study the growth and transformation of deep neural networks during training. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;Professor Naftali Tishby passed away in 2021. Hope the post can introduce his cool idea of information bottleneck to more people.&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;Recently I watched the talk &lt;a href=&#34;https://youtu.be/bLqJHjXihK8&#34;&gt;&amp;ldquo;Information Theory in Deep Learning&amp;rdquo;&lt;/a&gt; by Prof Naftali Tishby and found it very interesting. He presented how to apply the information theory to study the growth and transformation of deep neural networks during training. Using the &lt;a href=&#34;https://arxiv.org/pdf/physics/0004057.pdf&#34;&gt;Information Bottleneck (IB)&lt;/a&gt; method, he proposed a new learning bound for deep neural networks (DNN), as the traditional learning theory fails due to the exponentially large number of parameters. Another keen observation is that DNN training involves two distinct phases: First, the network is trained to fully represent the input data and minimize the generalization error; then, it learns to forget the irrelevant details by compressing the representation of the input.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>From GAN to WGAN</title>
      <link>https://lilianweng.github.io/posts/2017-08-20-gan/</link>
      <pubDate>Sun, 20 Aug 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-08-20-gan/</guid>
      <description>&lt;!-- This post explains the maths behind a generative adversarial network (GAN) model and why it is hard to be trained. Wasserstein GAN is intended to improve GANs&#39; training by adopting a smooth metric for measuring the distance between two probability distributions. --&gt;
&lt;p&gt;&lt;span class=&#34;update&#34;&gt;[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in &lt;a href=&#34;https://github.com/yjucho1/articles/blob/master/fromGANtoWGAN/readme.md&#34;&gt;Korean&lt;/a&gt;!]&lt;/span&gt;
&lt;br/&gt;
&lt;span class=&#34;update&#34;&gt;[Updated on 2019-04-18: this post is also available on &lt;a href=&#34;https://arxiv.org/abs/1904.08994&#34;&gt;arXiv&lt;/a&gt;.]&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1406.2661.pdf&#34;&gt;Generative adversarial network&lt;/a&gt; (GAN) has shown great results in many generative tasks to replicate the real-world rich content such as images, human language, and music. It is inspired by game theory: two models, a generator and a critic, are competing with each other while making each other stronger at the same time. However, it is rather challenging to train a GAN model, as people are facing issues like training instability or failure to converge.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>How to Explain the Prediction of a Machine Learning Model?</title>
      <link>https://lilianweng.github.io/posts/2017-08-01-interpretation/</link>
      <pubDate>Tue, 01 Aug 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-08-01-interpretation/</guid>
      <description>&lt;!-- This post reviews some research in model interpretability, covering two aspects: (i) interpretable models with model-specific interpretation methods and (ii) approaches of explaining black-box models. I included an open discussion on explainable artificial intelligence at the end. --&gt;
&lt;p&gt;The machine learning models have started penetrating into critical areas like health care, justice systems, and financial industry. Thus to figure out how the models make the decisions and make sure the decisioning process is aligned with the ethnic requirements or legal regulations becomes a necessity.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Predict Stock Prices Using RNN: Part 2</title>
      <link>https://lilianweng.github.io/posts/2017-07-22-stock-rnn-part-2/</link>
      <pubDate>Sat, 22 Jul 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-07-22-stock-rnn-part-2/</guid>
      <description>&lt;!-- This post is a continued tutorial for how to build a recurrent neural network using Tensorflow to predict stock market prices. Part 2 attempts to predict prices of multiple stocks using embeddings. The full working code is available in [lilianweng/stock-rnn](https://github.com/lilianweng/stock-rnn). --&gt;
&lt;p&gt;In the Part 2 tutorial, I would like to continue the topic on stock price prediction and to endow the recurrent neural network that I have built in &lt;a href=&#34;https://lilianweng.github.io/posts/2017-07-08-stock-rnn-part-1/&#34;&gt;Part 1&lt;/a&gt; with the capability of responding to multiple stocks. In order to distinguish the patterns associated with different price sequences, I use the stock symbol embedding vectors as part of the input.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>Predict Stock Prices Using RNN: Part 1</title>
      <link>https://lilianweng.github.io/posts/2017-07-08-stock-rnn-part-1/</link>
      <pubDate>Sat, 08 Jul 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-07-08-stock-rnn-part-1/</guid>
      <description>&lt;!-- This post is a tutorial for how to build a recurrent neural network using Tensorflow to predict stock market prices. Part 1 focuses on the prediction of S&amp;P 500 index. The full working code is available in [lilianweng/stock-rnn](https://github.com/lilianweng/stock-rnn). --&gt;
&lt;p&gt;This is a tutorial for how to build a recurrent neural network using Tensorflow to predict stock market prices. The full working code is available in &lt;a href=&#34;https://github.com/lilianweng/stock-rnn&#34;&gt;github.com/lilianweng/stock-rnn&lt;/a&gt;. If you don&amp;rsquo;t know what is recurrent neural network or LSTM cell, feel free to check &lt;a href=&#34;https://lilianweng.github.io/posts/2017-06-21-overview/#recurrent-neural-network&#34;&gt;my previous post&lt;/a&gt;.&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>An Overview of Deep Learning for Curious People</title>
      <link>https://lilianweng.github.io/posts/2017-06-21-overview/</link>
      <pubDate>Wed, 21 Jun 2017 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/posts/2017-06-21-overview/</guid>
      <description>&lt;!-- Starting earlier this year, I grew a strong curiosity of deep learning and spent some time reading about this field. To document what I’ve learned and to provide some interesting pointers to people with similar interests, I wrote this overview of deep learning models and their applications. --&gt;
&lt;p&gt;&lt;span style=&#34;color: #aaaaaa;&#34;&gt;(The post was originated from my talk for &lt;a href=&#34;http://wimlds.org/chapters/about-bay-area/&#34;&gt;WiMLDS x Fintech meetup&lt;/a&gt; hosted by &lt;a href=&#34;www.affirm.com&#34;&gt;Affirm&lt;/a&gt;.)&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;I believe many of you have watched or heard of the &lt;a href=&#34;https://youtu.be/vFr3K2DORc8&#34;&gt;games&lt;/a&gt; between AlphaGo and professional Go player &lt;a href=&#34;https://en.wikipedia.org/wiki/Lee_Sedol&#34;&gt;Lee Sedol&lt;/a&gt; in 2016. Lee has the highest rank of nine dan and many world championships. No doubt, he is one of the best Go players in the world, but he &lt;a href=&#34;https://www.scientificamerican.com/article/how-the-computer-beat-the-go-master/&#34;&gt;lost by 1-4&lt;/a&gt; in this series versus AlphaGo. Before this, Go was considered to be an intractable game for computers to master, as its simple rules lay out an exponential number of variations in the board positions, many more than what in Chess. This event surely highlighted 2016 as a big year for AI. Because of AlphaGo, much attention has been attracted to the progress of AI.&lt;/p&gt;</description>
    </item>
    
    
    <item>
      <title>FAQ</title>
      <link>https://lilianweng.github.io/faq/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://lilianweng.github.io/faq/</guid>
      <description></description>
    </item>
    
    
  </channel>
</rss>