Code

Implement DeepSeekV2#1010

Code

Merged

EricLBuehler merged 19 commits intomasterEricLBuehler/mistral.rs:masterfrom deepseek2EricLBuehler/mistral.rs:deepseek2Copy head branch name to clipboardJan 5, 2025

+1,962-205

Commits

Commits on Dec 27, 2024

• Initial steps toward supporting deepseekv2

  EricLBuehler

  committed
  b50ef1a

  Copy full SHA for b50ef1a

• Implement the attention forward

  EricLBuehler

  committed
  bad05e1

  Copy full SHA for bad05e1

• Add the mlp

  EricLBuehler

  committed
  aca6f73

  Copy full SHA for aca6f73

• Implement the moe gate and forward

  EricLBuehler

  committed
  7ef8801

  Copy full SHA for 7ef8801

Commits on Dec 28, 2024

• Fixes

  EricLBuehler

  committed
  cf5eb9f

  Copy full SHA for cf5eb9f

• Forward pass runs

  EricLBuehler

  committed
  9b73d23

  Copy full SHA for 9b73d23

• Clippy

  EricLBuehler

  committed
  3daa76e

  Copy full SHA for 3daa76e

Commits on Dec 30, 2024

• Merge branch 'master' into deepseek2

  EricLBuehler

  committed
  ead1fa4

  Copy full SHA for ead1fa4

Commits on Jan 2, 2025

• Update

  EricLBuehler

  committed
  b62ccbd

  Copy full SHA for b62ccbd

Commits on Jan 4, 2025

• It works

  EricLBuehler

  committed
  51111c4

  Copy full SHA for 51111c4

• Use faster rope

  EricLBuehler

  committed
  9e7d178

  Copy full SHA for 9e7d178

• Use normal cache

  EricLBuehler

  committed
  b1780d8

  Copy full SHA for b1780d8

• Add framework for paged attn

  EricLBuehler

  committed
  2e7cbf9

  Copy full SHA for 2e7cbf9

• Fixes

  EricLBuehler

  committed
  2799bf5

  Copy full SHA for 2799bf5

• Support isq

  EricLBuehler

  committed
  25dfd3c

  Copy full SHA for 25dfd3c

Commits on Jan 5, 2025

• Add moqe support, residual tensors

  EricLBuehler

  committed
  7343214

  Copy full SHA for 7343214

• Add examples, python API, docs

  EricLBuehler

  committed
  da79734

  Copy full SHA for da79734

• Fix tests

  EricLBuehler

  committed
  317f5ff

  Copy full SHA for 317f5ff

• Update deps

  EricLBuehler

  committed
  4a5a616

  Copy full SHA for 4a5a616