This repository contains code and resources to quantitatively certify the knowledge comprehension abilities of Large Language Models (LLMs). It hosts two main projects:
- wikidata_quacer: Experiments and utilities based on Wikidata that demonstrate the certification framework using a Wikidata knowledge graph.
- prime_quacer: Experiments and utilities using PrimeKG, a specialized knowledge graph for precision medicine, combined with clinical reasoning patterns.
The repository is organized into two primary directories:
-
wikidata_quacer/: Contains scripts, utilities, and experiment pipelines for Wikidata-based experiments. Inside this folder, you'll find detailed README files and tool documentation explaining how to run and customize the experiment scripts.
-
prime_quacer/: Focuses on precision medicine applications using the PrimeKG knowledge graph. It includes comprehensive documentation with examples on how to execute experiments, utilize subgraph and discovery utilities, and adapt custom experiments using provided templates.
To begin, please navigate to one of the two main project directories based on interest:
- For Wikidata related experiments, see wikidata_quacer/.
- For precision medicine experiments, check out prime_quacer/.
Each folder contains its own README and utility documentation (such as utilsREADME.md) that guide you through installation, setup, and experiment execution.
- Python 3.7 or higher
- Required packages: Install all dependencies with:
pip install -r requirements.txtThis project leverages advanced graph-based reasoning techniques to certify the comprehension of knowledge within Large Language Models. Contributions, suggestions, and feedback are greatly appreciated.
Happy experimenting!