gguf-dump.py: add --markdown dump output #7853
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mofosyne
added
Review Complexity : Low
|
FYI the context of this, was I was trying to understand the GGUF file standard a bit more. I also attempted to get the dump to print as a graphviz dotfile... but it was quite a pain to get working. However markdown is quite easy so this seems like a good compromise. The main benefit I can see however is in automatically grouping all the blocks together. |
mofosyne
force-pushed
the
gguf-dump-markdown
branch
from
aff4168 to
f15ce9f
Compare
mofosyne
force-pushed
the
gguf-dump-markdown
branch
from
c0439a4 to
e38b649
Compare
compilade
reviewed
Jun 13, 2024
|
@compilade updated the script with your suggestion also updated the example output with example output of a 'phi-2.Q6_K.gguf - GGUF Internal File Dump' dump so you can see how it looks now. |
compilade
reviewed
Jun 13, 2024
|
@compilade looks good thanks |
compilade
approved these changes
Jun 14, 2024
|
Adjusted ### <a name="blk_1">Block 1 Tensor Group : ~79M Elements</a>
| T_ID | Tensor Layer Name | Human Friendly Tensor Layer Name | Elements | Shape | Type |
|-----:|:-------------------------|:----------------------------------------|:-----------------|:----------------------|:-----|
| 11 | blk.1.attn_norm.bias | Block 1 Attention Normalization (B) | ( ~3K) 2560 | 2560 x 1 x 1 x 1 | F32 |
| 12 | blk.1.attn_norm.weight | Block 1 Attention Normalization (W) | ( ~3K) 2560 | 2560 x 1 x 1 x 1 | F32 |
| 13 | blk.1.attn_qkv.bias | Block 1 Attention Query-Key-Value (B) | ( ~8K) 7680 | 7680 x 1 x 1 x 1 | F32 |
| 14 | blk.1.attn_qkv.weight | Block 1 Attention Query-Key-Value (W) | ( ~20M) 19660800 | 2560 x 7680 x 1 x 1 | Q6_K |
| 15 | blk.1.attn_output.bias | Block 1 Attention Output (B) | ( ~3K) 2560 | 2560 x 1 x 1 x 1 | F32 |
| 16 | blk.1.attn_output.weight | Block 1 Attention Output (W) | ( ~7M) 6553600 | 2560 x 2560 x 1 x 1 | Q6_K |
| 17 | blk.1.ffn_up.bias | Block 1 Feed-Forward Network "Up" (B) | ( ~10K) 10240 | 10240 x 1 x 1 x 1 | F32 |
| 18 | blk.1.ffn_up.weight | Block 1 Feed-Forward Network "Up" (W) | ( ~26M) 26214400 | 2560 x 10240 x 1 x 1 | Q6_K |
| 19 | blk.1.ffn_down.bias | Block 1 Feed-Forward Network "Down" (B) | ( ~3K) 2560 | 2560 x 1 x 1 x 1 | F32 |
| 20 | blk.1.ffn_down.weight | Block 1 Feed-Forward Network "Down" (W) | ( ~26M) 26214400 | 10240 x 2560 x 1 x 1 | Q6_K |
- Total elements in blk.1: (~79M) 78671360
- Percentage of total elements: 2.83%If you have no other thoughts @compilade feel free to press merge. |
compilade
reviewed
Jun 15, 2024
|
@compilade included your recommendation. How is it now? |
Co-authored-by: compilade <[email protected]>
mofosyne
force-pushed
the
gguf-dump-markdown
branch
from
7dc405b to
51032b1
Compare
|
rebased with no modification to sync against latest master to deal with ci issue |
|
Will merge soon as I've addressed all of @compilade outstanding points and ci has passed. |
mofosyne
added
the
merge ready
compilade
reviewed
Jun 17, 2024
Co-authored-by: compilade <[email protected]>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will allow
$gguf-dump.py Tinyllama-5M-v0.2-Q8_0.gguf --markdownto output a markdown formatted dump that is designed to be as easy as possible to read as a markdown file.It's sent to stdout so you can do stuff like
$gguf-dump.py Tinyllama-5M-v0.2-Q8_0.gguf --markdown | mdlessto render the markdown dump directly. Alternatively it might be part of your workflow to render this file when creating a new gguf.Why do this when you can still manually dump it whenever? Well in a github / huggingface repo, it still might be good courtesy to have an easy to read technical dump of the layers. Note that I also added a function that reads the tensor name and converts it into a human friendly name as devs coming across it may not necessarily know what ffn etc.. means.
Below is how it currently renders. Feel free to suggest changes to it to make it as useful as possible for you.
Example Output