Add GPT2 tokenizer setup and refactor code #56

Merged

aliciafmachado merged 8 commits intoPAIR-code:mainfrom aliciafmachado:gpt-tokenizer

Feb 5, 2025

+485-398

Commits on Feb 1, 2025

Add gpt-tokenizer dependency.
aliciafmachado
committed
Add function for using a standalone tokenizer with GPT2 instead of using the local task token rep.
aliciafmachado
committed
Refactor code so that common functions among transformers are in a single file. Refactor token gemb file so that embedBatch is adapted to a use case where a tokenizer is available.
aliciafmachado
committed
Refactored computePrediction and computeDecoder so that it's compatible with next N tokens prediction.
aliciafmachado
committed
Remove unused test function and clean-up comments.
aliciafmachado
committed
Clarify some TODOs and comments on the code.
aliciafmachado
committed
Add node_modules/gpt-tokenizer to package-lock.json.
aliciafmachado
committed
Clarify testing function and simplify mapToIdx and tokenizeAndMapToIdx.
aliciafmachado
committed