Add data format to readme (#90)

arcee-ai · Jan 19, 2024 · 699aedd · 699aedd
1 parent 5196a07
commit 699aedd
Showing 1 changed file with 20 additions and 0 deletions.
diff --git a/dalm/pipelines/README.md b/dalm/pipelines/README.md
@@ -13,4 +13,24 @@ python dalm/pipelines/reading_comprehension_pipeline.py --model_name HuggingFace
 	--llm_synth_model_name meta-llama/Llama-2-13b-chat-hf \
     --llm_synth_model_context_length 4096
 
+```
+
+### Data format
+
+```
+{"messages":[
+    [{"role":"...", "content": "..."}, {"role":"...", "content": "..."}, ...],
+    [{"role":"...", "content": "..."}, {"role":"...", "content": "..."}, ...],
+    [{"role":"...", "content": "..."}, {"role":"...", "content": "..."}, ...],
+    ....
+    ]
+}
+```
+
+take from this snippet:
+
+```
+import datasets
+a = datasets.load_dataset("arcee-ai/azure-reading-comprehension-dataset")
+print(a["train"]["messages"][0])
 ```