Package hallucination

[arjun-krishna1]

• Use new dataset with "date first seen" for python pypi package hallucination
• Add scripts used to create Python, JavaScript and Ruby huggingface datasets in tools/packagehallucination

Verification

List the steps needed to make sure this thing works

• python -m garak -m openai -n gpt-3.5-turbo -p packagehallucination.Python --parallel_requests 16 -g 3 -> runs python hallucination test and result is sane

(garak_env) arjun@arjun-ThinkPad-L14-Gen-3:~/garak$ python -m garak -m openai -n gpt-3.5-turbo -p packagehallucination.Python --parallel_requests 16 -g 3
garak LLM vulnerability scanner v0.9.0.16.post1 ( https://github.com/leondz/garak ) at 2024-10-12T14:19:17.439482
📜 logging to /home/arjun/.local/share/garak/garak.log
🦜 loading generator: OpenAI: gpt-3.5-turbo
📜 reporting to /home/arjun/.local/share/garak/garak_runs/garak.06e1956f-feed-449d-a0dc-63e5fc12414c.report.jsonl
🕵️  queue of probes: packagehallucination.Python
Downloading readme: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 31.0/31.0 [00:00<00:00, 263kB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████| 19.3M/19.3M [00:16<00:00, 1.19MB/s]
Downloading data: 100%|█████████████████████████████████/home/arjun/garak/garak_env/lib/python3.12/site-packages/datasets/download/streaming_download_manager.py:778: FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.
  return pd.read_csv(xopen(filepath_or_buffer, "rb", download_config=download_config), **kwargs)
Generating train split: 548891 examples [00:00, 1245837.61 examples/s]
packagehallucination.Python                                          packagehallucination.PythonPypi: FAIL  ok on  263/ 273   (failure rate:   3.66%)
📜 report closed :) /home/arjun/.local/share/garak/garak_runs/garak.06e1956f-feed-449d-a0dc-63e5fc12414c.report.jsonl
📜 report html summary being written to /home/arjun/.local/share/garak/garak_runs/garak.06e1956f-feed-449d-a0dc-63e5fc12414c.report.html
✔️  garak run complete in 187.62s

• Run the tests and ensure they pass python -m pytest tests/

[jmartin-tech]

This PR does not yet enable the probe to consume the timestamp.

Another item to ensure, is all date columns should be formatted into a date time value with timezone details. A format string example can be found in the _plugin package:
https://github.com/leondz/garak/blob/bd0bb3f468c6c05a5675aca9c1e4093b35970018/garak/_plugins.py#L22

This is important to ensure results are consistent globally when run on systems in with clock set to varying timezones.

[leondz]

NB datestamp is to be consumed by detector

[arjun-krishna1]

Ok, thank you @leondz @jmartin-tech . I've just finished my final midterm. I am working on these changes right now.

• Do tsv instead of csv
• remove cutoff in ruby dataset
• remove hardcoded personal path in ruby script
• make sure all date columns are formatted into a date time value with dimezone details

[arjun-krishna1]

Ok, I've generated new datasets following your feedback and added to huggingface/updated references here

[jmartin-tech]

While the contents of this file are useful for testing it should not be committed to the repository for distribution as part of the primary package.

[arjun-krishna1]

Sounds good @jmartin-tech , I will remove this