pipelines: Add missing drop_duplicates for context in grounded skills

The full grounded skills pipeline begins by generating context. This block had "drop_duplicates: context" in its config, but it was accidentally dropped in the conversion to yaml. Signed-off-by: Russell Bryant <[email protected]>
instructlab · Jul 12, 2024 · 7c5c1c3 · 7c5c1c3
1 parent 5a0b7a6
commit 7c5c1c3
Showing 1 changed file with 2 additions and 0 deletions.
diff --git a/src/instructlab/sdg/pipelines/full/grounded_skills.yaml b/src/instructlab/sdg/pipelines/full/grounded_skills.yaml
@@ -9,6 +9,8 @@ blocks:
     gen_kwargs:
       temperature: 0.7
       max_tokens: 2048
+    drop_duplicates:
+      - context
   - name: gen_grounded_questions
     type: LLMBlock
     config: