Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llama-bench : add JSONL (NDJSON) output mode #9288

Merged
merged 3 commits into from
Sep 3, 2024

Conversation

akx
Copy link
Contributor

@akx akx commented Sep 3, 2024

This PR adds a JSONL/NDJSON output mode to llama-bench. JSONL/NDJSON can be processed line by line even if the generating process fails to e.g. print out the final ] for the regular JSON output.

$ ./llama-bench -n 16,16,16,16 -ngl 99 -o jsonl > foo.jsonl
$ duckdb -c 'select n_gen, mean(avg_ts) from "out.jsonl" group by n_gen;'
┌───────┬──────────────┐
│ n_gen │ mean(avg_ts) │
│ int64 │    double    │
├───────┼──────────────┤
│     0 │   746.241384 │
│    16 │     22.08429 │
└───────┴──────────────┘

@akx akx marked this pull request as draft September 3, 2024 07:10
@akx akx marked this pull request as ready for review September 3, 2024 07:23
examples/llama-bench/llama-bench.cpp Outdated Show resolved Hide resolved
examples/llama-bench/llama-bench.cpp Show resolved Hide resolved
Co-authored-by: slaren <slarengh@gmail.com>
@akx akx requested a review from slaren September 3, 2024 17:33
@akx akx mentioned this pull request Sep 3, 2024
4 tasks
@slaren slaren merged commit 8962422 into ggerganov:master Sep 3, 2024
51 checks passed
@akx akx mentioned this pull request Sep 4, 2024
4 tasks
dsx1986 pushed a commit to dsx1986/llama.cpp that referenced this pull request Oct 29, 2024
* llama-bench : add JSONL (NDJSON) output mode

* llama-bench : update usage docs
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 15, 2024
* llama-bench : add JSONL (NDJSON) output mode

* llama-bench : update usage docs
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 18, 2024
* llama-bench : add JSONL (NDJSON) output mode

* llama-bench : update usage docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants