notebook/dataset-config/dataset-config-local-files-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using datasets with local text file\n",
    "\n",
    "This will be covering, specifically on how to load local dataset files.\n",
    "\n",
    "In general you should review through `./dataset-config-huggingface-example.ipynb` first, because a large percentage of settings can be used together with the settings covered here.\n",
    "\n",
    "> Important note: These example focuses only on how to configure your dataset, and does not properly perform checkmarking - for trainer configurations refer to the training notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intial setup\n",
    "\n",
    "Before we go into the dataset setup, lets perform an initial setup for all the folders we need, and a small toy model which we would use throughout the various examples within this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the folders we will need\n",
    "!mkdir -p ../../model/\n",
    "!mkdir -p ../../datapath/\n",
    "!mkdir -p ../../checkpoint/\n",
    "\n",
    "# Initialized a simple L6-D512 model, for both the v4 neox (50277) tokenizer\n",
    "!cd ../../RWKV-v5/ && python3 ./init_model.py --n_layer 6 --n_embd 512 --vocab_size neox --skip-if-exists ../model/L6-D512-neox-init.pth\n",
    "\n",
    "# and rwkv world (65529) tokenizers\n",
    "!cd ../../RWKV-v5/ && python3 ./init_model.py --n_layer 6 --n_embd 512 --vocab_size world --skip-if-exists ../model/L6-D512-world-init.pth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with collection of text files\n",
    "\n",
    "### **Download the dataset file**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the dataset dir\n",
    "!mkdir -p ../../dataset/dataset-config/text/\n",
    "!mkdir -p ../../dataset/dataset-config/zip/\n",
    "\n",
    "# Download the files\n",
    "!cd ../../dataset/dataset-config/zip/ && wget -nc https://data.deepai.org/enwik8.zip\n",
    "!cd ../../dataset/dataset-config/text/ && rm -rf ./*\n",
    "!cd ../../dataset/dataset-config/text/ && unzip ../zip/enwik8.zip\n",
    "!cd ../../dataset/dataset-config/text/ && mv enwik8 enwik8.txt\n",
    "!cd ../../dataset/dataset-config/text/ && ls -lh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Parse the dataset**\n",
    "\n",
    "The following is the `example-local-text.yaml` settings, for using local textual data\n",
    "\n",
    "---\n",
    "```yaml\n",
    "trainer:\n",
    "  max_steps: 10\n",
    "  target_batch_size: 32\n",
    "model:\n",
    "  load_model: ../model/L6-D512-neox-init.pth\n",
    "  ctx_len: 1024\n",
    "  lr_init: 3e-4\n",
    "  bptt_learning: true\n",
    "  bptt_learning_range: -1\n",
    "data:\n",
    "  data_path: ../datapath/enwiki8_neox_1024/\n",
    "\n",
    "  source: \"text\"\n",
    "  source_data_dir: \"../dataset/dataset-config/text/\"\n",
    "  tokenizer: neox\n",
    "  \n",
    "  text_rechunk_size: 2048\n",
    "  \n",
    "  test_split: 0.01\n",
    "  test_split_shuffle: false\n",
    "```\n",
    "---\n",
    "\n",
    "### Understanding the `data` config, for textual datasets\n",
    "\n",
    "**data.data_path** \n",
    "\n",
    "This is where the HF datapath is saved, when used against existing HF data sources. This is a requried parameter\n",
    "\n",
    "**data.source** \n",
    "\n",
    "This can be configured as `text / json / csv / pandas` for local files\n",
    "\n",
    "**data.source_data_dir** \n",
    "\n",
    "Folder / Directory which contains the respective `text / json (or jsonl) / csv / pandas` files\n",
    "\n",
    "**data.tokenizer**\n",
    "\n",
    "The tokenizer to use for the dataset, use either `neox` or `world` for the respective RWKV models. For custom HF tokenizer refer to `./dataset-config-huggingface-examples.ipynb`\n",
    "\n",
    "**data.text_rechunk_size** \n",
    "\n",
    "Number of tokens each datasample should have after rechunking. Recommended sizes is the context size you intend the model to support (ie. 2048, 4096, etc). This is enabled, for text based dataset.\n",
    "\n",
    "**data.test_split**\n",
    "\n",
    "If configured as a floating number between 0.0 and 1.0, it will be the percentage (0.1 is 10%) of the test data that is used for test validation.\n",
    "\n",
    "If configured as an int number, it will be the number of samples.\n",
    "\n",
    "Due to some limitations in the current trainer code, even if its set as 0, we will use a single data sample for the test split.\n",
    "\n",
    "This defaults to 0.01 or 1%\n",
    "\n",
    "**data.test_split_shuffle**\n",
    "\n",
    "Perform a dataset shuffle before test split, this defaults to False.\n",
    "\n",
    "Note, this is not a truely random shuffle, but a detriministic shuffle. To ensure a consistent result.\n",
    "\n",
    "---\n",
    "\n",
    "### Parse the dataset, and run the training process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets preload the requried dataset\n",
    "!cd ../../RWKV-v5 && python3 preload_datapath.py ../notebook/dataset-config/example-local-text.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate the dataset is working, by doing a quick training run\n",
    "!cd ../../RWKV-v5 && python3 lightning_trainer.py fit -c ../notebook/dataset-config/example-local-text.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with JSON / JSONL / CSV files\n",
    "\n",
    "### **Download the dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the dataset dir\n",
    "!mkdir -p ../../dataset/dataset-config/jsonl/\n",
    "\n",
    "# Download the files\n",
    "!cd ../../dataset/dataset-config/jsonl/ && wget -nc https://huggingface.co/datasets/picocreator/RWKV-notebook-assets/raw/main/sample-memory-train-10-word-count.jsonl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Parse the dataset**\n",
    "\n",
    "The following is the `example-local-json.yaml` settings, for using local textual data\n",
    "\n",
    "---\n",
    "```yaml\n",
    "trainer:\n",
    "  max_steps: 10\n",
    "  target_batch_size: 32\n",
    "model:\n",
    "  load_model: ../model/L6-D512-neox-world.pth\n",
    "  ctx_len: 1024\n",
    "  lr_init: 3e-4\n",
    "  bptt_learning: true\n",
    "  bptt_learning_range: -1\n",
    "data:\n",
    "  data_path: ../datapath/enwiki8_neox_1024/\n",
    "\n",
    "  # Note that json work for both \".json\" and \".jsonl\"\n",
    "  source: \"json\"\n",
    "  source_data_dir: \"../dataset/dataset-config/jsonl/\"\n",
    "  tokenizer: world\n",
    "  \n",
    "  test_split: 0.01\n",
    "  test_split_shuffle: false\n",
    "```\n",
    "---\n",
    "\n",
    "### Understanding the `data` config, for textual datasets\n",
    "\n",
    "**data.source** \n",
    "\n",
    "This can be configured as `text / json / csv / pandas` for local files. For most part, since json/csv/pandas deal with structured data formatting, they should work in a similar fashion.\n",
    "\n",
    "**data.source_data_dir** \n",
    "\n",
    "Folder / Directory which contains the respective `text / json (or jsonl) / csv / pandas` files\n",
    "\n",
    "**data.tokenizer**\n",
    "\n",
    "The tokenizer to use for the dataset, use either `neox` or `world` for the respective RWKV models. For custom HF tokenizer refer to `./dataset-config-huggingface-examples.ipynb`\n",
    "\n",
    "> All the advance settings for collumn datasets handling in `./dataset-config-huggingface-examples.ipynb` works with `JSON / CSV / pandas` based local data files formats. This includes force text rechunking, multi-column formatting, collumn masking, etc.\n",
    "\n",
    "---\n",
    "\n",
    "### Parse the dataset, and run the training process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets preload the requried dataset\n",
    "!cd ../../RWKV-v5 && python3 preload_datapath.py ../notebook/dataset-config/example-local-jsonl.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate the dataset is working, by doing a quick training run\n",
    "!cd ../../RWKV-v5 && python3 lightning_trainer.py fit -c ../notebook/dataset-config/example-local-jsonl.yaml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rwkv-infctx",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}