Code

Explore moe#26

Code

Open

xzyaoi wants to merge 13 commits intomaineth-easl/deltazip:mainfrom explore-moeeth-easl/deltazip:explore-moeCopy head branch name to clipboard

+4,373-34

Commits

Commits on Mar 1, 2024

• add notebook with MoE delta experiment

  

  BoykoBorisov

  committed
  45e2338

• cleanup

  

  BoykoBorisov

  committed
  7b68ff1

• Add plotting for both layers

  

  BoykoBorisov

  committed
  d2e5753

Commits on Mar 14, 2024

• Start implementation

  

  BoykoBorisov

  committed
  348dc90

Commits on Mar 15, 2024

• Skeleton moe

  

  Boyko Borisov

  committed
  bc20339

• Add stubs

  

  BoykoBorisov

  committed
  c7239ad

Commits on Mar 19, 2024

• Add utility functions for weight extraction; start preparation of lossy compression function to support moe

  

  BoykoBorisov

  committed
  d32c6ec

Commits on Mar 24, 2024

• Start working on CLI moe compression; bugfixes and minor changes

  

  BoykoBorisov

  committed
  4b583b6

• Fix base model weight lookup

  

  BoykoBorisov

  committed
  fb071d5

Commits on Apr 1, 2024

• compression pipeline

  

  BoykoBorisov

  committed
  5c6a0c7

Commits on Apr 15, 2024

• Adds infra for benchmarking

  

  BoykoBorisov

  committed
  61c931b

Commits on May 24, 2024

• Bugfixes, changes for GPTNeoX Experiments

  

  BoykoBorisov

  committed
  74cabfa

Commits on Oct 25, 2024

• Add llama-moe support

  

  BoykoBorisov

  committed
  807748d