Code

llama : support RWKV v6 models#8980

Code

Merged

ggerganov merged 53 commits intoggerganov:masterggerganov/llama.cpp:masterfrom MollySophia:for-upstreamMollySophia/llama.cpp:for-upstreamCopy head branch name to clipboardSep 1, 2024

+1,266-103

Commits

Commits on Aug 28, 2024

• convert_hf_to_gguf: Add support for RWKV v6

  Show description for 8d2eca3
  MollySophia

  committed
  8d2eca3

• Add RWKV tokenization

  

  LaylBongers

  authored and

  MollySophia

  committed
  dc0767f

• Fix build

  Show description for 865167d
  MollySophia

  committed
  865167d

• Do not use special tokens when matching in RWKV tokenizer

  

  LaylBongers

  authored and

  MollySophia

  committed
  7cac72a

• Fix model loading

  

  LaylBongers

  authored and

  MollySophia

  committed
  e92c74f

• Add (broken) placeholder graph builder for RWKV

  

  LaylBongers

  authored and

  MollySophia

  committed
  a0aae8d

• Add workaround for kv cache

  

  LaylBongers

  authored and

  MollySophia

  committed
  a866789

• Add logits conversion to rwkv5

  

  LaylBongers

  authored and

  MollySophia

  committed
  4e23d97

• Add rwkv5 layer norms

  

  LaylBongers

  authored and

  MollySophia

  committed
  5479588

• Add time mix KVRG & correct merge mistake

  

  LaylBongers

  authored and

  MollySophia

  committed
  dd3aa3d

• Add remaining time mix parameters

  

  LaylBongers

  authored and

  MollySophia

  committed
  b409fd8

• Add time mix output loading

  

  LaylBongers

  authored and

  MollySophia

  committed
  3cbeffc

• Add placeholder llm_build_time_mix

  

  LaylBongers

  authored and

  MollySophia

  committed
  b3b17e0

• Fix build

  Show description for 700dad1
  MollySophia

  committed
  700dad1

• Load more tensors for rwkv v6

  Show description for a180b63
  MollySophia

  committed
  a180b63

• Fix rwkv tokenizer

  Show description for 0e5ac34
  MollySophia

  committed
  0e5ac34

• ggml: Add unary operator Exp

  Show description for 5732de8
  MollySophia

  committed
  5732de8

• RWKV v6 graph building

  Show description for 0784a0c
  MollySophia

  committed
  0784a0c

• Add rescale_every_n_layers parameter

  Show description for 8d498c7
  MollySophia

  committed
  8d498c7

• Add wkv.head_size key for RWKV

  Show description for 903089b
  MollySophia

  committed
  903089b

• Fix offloading layers to CUDA

  Show description for 98ce5f4
  MollySophia

  committed
  98ce5f4

• Fix parallel inferencing for RWKV

  Show description for 01dcf4b
  MollySophia

  committed
  01dcf4b

• Remove trailing whitespaces

  Show description for 6ae2f48
  MollySophia

  committed
  6ae2f48

• build_rwkv: Avoid using inplace operations

  Show description for 8bc1f9a
  MollySophia

  committed
  8bc1f9a

• convert_hf_to_gguf: rwkv: Avoid using eval

  Show description for 18decea
  MollySophia

  committed
  18decea

• convert_hf_to_gguf: rwkv tokenizer: Don't escape sequences manually

  Show description for 7f2e370
  MollySophia

  committed
  7f2e370

• Update convert_hf_to_gguf.py

  Show description for c695552
  

  MollySophia

  and

  compilade

  committed
  c695552

• ggml: Add backward computation for unary op exp

  Show description for 8aa711a
  MollySophia

  committed
  8aa711a

• Update convert_hf_to_gguf.py

  Show description for ae9936a
  

  MollySophia

  and

  compilade

  committed
  ae9936a

• Update convert_hf_to_gguf.py

  Show description for 5afa3ef
  

  MollySophia

  and

  compilade

  committed
  5afa3ef

• Use MODEL_ARCH.RWKV6 instead of MODEL_ARCH.RWKV

  Show description for 12fbe1a
  MollySophia

  committed
  12fbe1a

• build_rwkv6: Simplify graph

  Show description for 276d53b
  MollySophia

  committed
  276d53b

• llama: rwkv6: Detect model.type

  Show description for b0f4fe5
  MollySophia

  committed
  b0f4fe5

• llama: rwkv6: Fix tensor loading for 7B/14B models

  Show description for 683d70c
  MollySophia

  committed
  683d70c

• llama: rwkv6: Fix group_norm assertion failure with Metal

  Show description for ee1b78c
  MollySophia

  committed
  ee1b78c

• llama: rwkv6: Clean up

  Show description for c165e34
  MollySophia

  committed
  c165e34

• llama: rwkv6: Add quantization tensor exclusion

  Show description for 6da6aa4
  MollySophia

  committed
  6da6aa4

• llama: rwkv6: Use the new advanced batch splits

  Show description for f5d955d
  MollySophia

  committed
  f5d955d

• Update src/llama.cpp

  Show description for 57decb4
  

  MollySophia

  and

  compilade

  committed
  57decb4

• llama: rwkv6: Use ggml_norm instead of ggml_group_norm

  Show description for e94778a
  

  MollySophia

  and

  compilade

  committed
  e94778a

• llama: rwkv6: Apply code style and misc changes

  Show description for 7756afd
  MollySophia

  committed
  7756afd

• converter: Use class name Rwkv6Model

  Show description for 87a2901
  MollySophia

  committed
  87a2901

• llama: rwkv6: Make use of key feed_forward_length

  Show description for c414a24
  MollySophia

  committed
  c414a24

• llama: rwkv6: Add kv time_mix_extra_dim and time_decay_extra_dim

  Show description for 6d69fd7
  MollySophia

  committed
  6d69fd7

• converter: Match new_name instead of name for float32 explicit tensors

  Show description for 601b592
  MollySophia

  committed
  601b592

• llama: rwkv6: Keep time_mix_w1/w2 as F32

  Show description for e0ea511
  MollySophia

  committed
  e0ea511

• llama: rwkv6: Remove unused nodes

  Show description for 5f00c52
  MollySophia

  committed
  5f00c52

• llama: rwkv6: Apply code format changes

  Show description for 7444046
  MollySophia

  committed
  7444046

Commits on Aug 30, 2024

• llama: rwkv6: Add lora for some supported tensors

  Show description for 7f2ef56
  MollySophia

  committed
  7f2ef56

• rwkv : speed-up tokenization using trie

  ggerganov

  committed
  7004323

• minor : style + indentation

  ggerganov

  committed
  59dc2e7

Commits on Aug 31, 2024

• llama: rwkv6: Avoid division by zero

  Show description for 5175375
  

  MollySophia

  and

  compilade

  authored
  5175375

• ggml: rwkv_wkv: Avoid copying the state

  Show description for 846358d
  MollySophia

  committed
  846358d