instructions

allenai · Oct 24, 2024 · 476d6d0 · 476d6d0
1 parent 42be092
commit 476d6d0
Show file tree

Hide file tree

Showing 2 changed files with 9 additions and 3 deletions.
diff --git a/classifiers/README.md b/classifiers/README.md
@@ -19,6 +19,12 @@ python -m dolma_classifiers.inference \
     -m HuggingFaceFW/fineweb-edu-classifier
 ```
 
+Run [NVIDIA's Deberta quality classifier](https://huggingface.co/nvidia/quality-classifier-deberta) on S3 data with model compilation:
 
-<!-- Run [NVIDIA's Deberta quality classifier](https://huggingface.co/nvidia/quality-classifier-deberta) on S3 data:
- -->
+```bash
+python -m dolma_classifiers.inference \
+    -s 's3://ai2-llm/pretraining-data/sources/dclm/v0/documents/40b-split/*/*zstd' \
+    -m nvidia/quality-classifier-deberta \
+    --model-compile \
+    --max-length 1024
+```
diff --git a/classifiers/scripts/nvidia-deberta-100b.sh b/classifiers/scripts/nvidia-deberta-100b.sh
@@ -5,7 +5,7 @@ DOCUMENTS='s3://ai2-llm/pretraining-data/sources/dclm/v0/documents/100*/*.jsonl.
 NUM_NODES=2
 MODEL_NAME="nvidia/quality-classifier-deberta"
 CLUSTER="ai2/jupiter*"
-BATCH_SIZE=1024
+BATCH_SIZE=512
 PRIORITY="high"
 
 # Generate a hash for the run name by combining model name and documents