llama_index_quickstart.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s6AXwQKpGcgH"
      },
      "source": [
        "# 📓 Llama-Index Quickstart\n",
        "\n",
        "In this quickstart you will create a simple Llama Index app and learn how to log it and get feedback on an LLM response.\n",
        "\n",
        "For evaluation, we will leverage the \"hallucination triad\" of groundedness, context relevance and answer relevance.\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/quickstart/llama_index_quickstart.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yqn1ylTyGcgK"
      },
      "source": [
        "## Setup\n",
        "\n",
        "### Install dependencies\n",
        "Let's install some of the dependencies for this notebook if we don't have them already"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "gk38ONvbGcgL"
      },
      "outputs": [],
      "source": [
        "# pip install trulens_eval llama_index openai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "luAt0pUVGcgM"
      },
      "source": [
        "### Add API keys\n",
        "For this quickstart, you will need Open AI and Huggingface keys. The OpenAI key is used for embeddings and GPT, and the Huggingface key is used for evaluation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "0jznuTR2GcgM"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"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\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kwIRFTarGcgM"
      },
      "source": [
        "### Import from TruLens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "A2aeJvHDGcgM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
            "🛑 Secret keys may be written to the database. See the `database_redact_keys` option of Tru` to prevent this.\n"
          ]
        }
      ],
      "source": [
        "from trulens_eval import Tru\n",
        "tru = Tru()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ynXp4XVGcgN"
      },
      "source": [
        "### Download data\n",
        "\n",
        "This example uses the text of Paul Graham’s essay, [“What I Worked On”](https://paulgraham.com/worked.html), and is the canonical llama-index example.\n",
        "\n",
        "The easiest way to get it is to [download it via this link](https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt) and save it in a folder called data. You can do so with the following command:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api'"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import os\n",
        "os.getcwd()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e03zR8LXGcgN"
      },
      "source": [
        "### Create Simple LLM Application\n",
        "\n",
        "This example uses LlamaIndex which internally uses an OpenAI LLM."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "RSZWQiVGGcgN"
      },
      "outputs": [],
      "source": [
        "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n",
        "from llama_index.core import VectorStoreIndex,SimpleDirectoryReader,ServiceContext\n",
        "documents = SimpleDirectoryReader(\"rag_api/uploads\").load_data()\n",
        "index = VectorStoreIndex.from_documents(documents)\n",
        "query_engine = index.as_query_engine()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'llama_index.vector_stores'",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "Cell \u001b[1;32mIn[16], line 6\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mre\u001b[39;00m\n\u001b[0;32m      5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mllama_index\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcore\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m VectorStoreIndex,SimpleDirectoryReader,ServiceContext,StorageContext\n\u001b[1;32m----> 6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mllama_index\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mvector_stores\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mqdrant\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m QdrantVectorStore\n\u001b[0;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01membeddings\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mhuggingface\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m HuggingFaceEmbeddings\n\u001b[0;32m      8\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mllama_index\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01membeddings\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlangchain\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m LangchainEmbedding\n",
            "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'llama_index.vector_stores'"
          ]
        },
        {
          "ename": "",
          "evalue": "",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31mThe Kernel crashed while executing code in the current cell or a previous cell. \n",
            "\u001b[1;31mPlease review the code in the cell(s) to identify a possible cause of the failure. \n",
            "\u001b[1;31mClick <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. \n",
            "\u001b[1;31mView Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
          ]
        }
      ],
      "source": [
        "\n",
        "from fastapi import UploadFile, HTTPException, File\n",
        "import aiofiles\n",
        "import  os\n",
        "import re\n",
        "from llama_index.core import VectorStoreIndex,SimpleDirectoryReader,ServiceContext,StorageContext\n",
        "from llama_index.vector_stores.qdrant import QdrantVectorStore\n",
        "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
        "from llama_index.embeddings.langchain import LangchainEmbedding\n",
        "from tqdm import tqdm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vcZYB5jLGcgO"
      },
      "source": [
        "### Send your first request"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "6RONB7T4GcgO"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The author attended government schools for 9 out of 12 years before spending 2.5 years at a particular school.\n"
          ]
        }
      ],
      "source": [
        "response = query_engine.query(\"What did the author do growing up?\")\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OGcfR06GGcgO"
      },
      "source": [
        "## Initialize Feedback Function(s)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "xN4iRDsyGcgO"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "✅ In groundedness_measure_with_cot_reasons, input source will be set to __record__.app.query.rets.source_nodes[:].node.text.collect() .\n",
            "✅ In groundedness_measure_with_cot_reasons, input statement will be set to __record__.main_output or `Select.RecordOutput` .\n",
            "✅ In relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
            "✅ In relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .\n",
            "✅ In qs_relevance, input question will be set to __record__.main_input or `Select.RecordInput` .\n",
            "✅ In qs_relevance, input statement will be set to __record__.app.query.rets.source_nodes[:].node.text .\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "\n",
        "# Initialize provider class\n",
        "from trulens_eval.feedback.provider.openai import OpenAI\n",
        "openai = OpenAI()\n",
        "\n",
        "# select context to be used in feedback. the location of context is app specific.\n",
        "from trulens_eval.app import App\n",
        "context = App.select_context(query_engine)\n",
        "\n",
        "# imports for feedback\n",
        "from trulens_eval import Feedback\n",
        "\n",
        "# Define a groundedness feedback function\n",
        "from trulens_eval.feedback import Groundedness\n",
        "grounded = Groundedness(groundedness_provider=OpenAI())\n",
        "f_groundedness = (\n",
        "    Feedback(grounded.groundedness_measure_with_cot_reasons)\n",
        "    .on(context.collect()) # collect context chunks into a list\n",
        "    .on_output()\n",
        "    .aggregate(grounded.grounded_statements_aggregator)\n",
        ")\n",
        "\n",
        "# Question/answer relevance between overall question and answer.\n",
        "f_qa_relevance = Feedback(openai.relevance).on_input_output()\n",
        "\n",
        "# Question/statement relevance between question and each context chunk.\n",
        "f_qs_relevance = (\n",
        "    Feedback(openai.qs_relevance)\n",
        "    .on_input()\n",
        "    .on(context)\n",
        "    .aggregate(np.mean)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fauuR4seGcgO"
      },
      "source": [
        "## Instrument app for logging with TruLens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "n8INoAvUGcgO"
      },
      "outputs": [],
      "source": [
        "from trulens_eval import TruLlama\n",
        "tru_query_engine_recorder = TruLlama(query_engine,\n",
        "    app_id='LlamaIndex_App1',\n",
        "    feedbacks=[f_groundedness, f_qa_relevance, f_qs_relevance])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "UK6CH-A2GcgO"
      },
      "outputs": [],
      "source": [
        "# or as context manager\n",
        "with tru_query_engine_recorder as recording:\n",
        "    query_engine.query(\"What did the author do growing up?\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6bMy9LPeGcgO"
      },
      "source": [
        "## Retrieve records and feedback"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "Zo-YrUd1GcgP"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Record(record_id='record_hash_56e53381af40be526ab632ae206ada68', app_id='LlamaIndex_App1', cost=Cost(n_requests=2, n_successful_requests=2, n_classes=0, n_tokens=2056, n_stream_chunks=0, n_prompt_tokens=2031, n_completion_tokens=25, cost=0.0030845000000000004), perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 11, 59, 342173), end_time=datetime.datetime(2024, 2, 29, 16, 12, 3, 630191)), ts=datetime.datetime(2024, 2, 29, 16, 12, 3, 631727), tags='-', meta=None, main_input='What did the author do growing up?', main_output='The author attended government schools for 9 out of 12 years before spending 2.5 years at a particular school.', main_error=None, calls=[RecordAppCall(stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='retrieve')), RecordAppCallMethod(path=Lens().app._retriever, method=Method(obj=Obj(cls=llama_index.core.indices.vector_store.retrievers.retriever.VectorIndexRetriever, id=2416092148304, init_bindings=None), name='retrieve')), RecordAppCallMethod(path=Lens().app._retriever, method=Method(obj=Obj(cls=llama_index.core.indices.vector_store.retrievers.retriever.VectorIndexRetriever, id=2416092148304, init_bindings=None), name='_retrieve'))], args={'query_bundle': {'query_str': 'What did the author do growing up?', 'image_path': None, 'custom_embedding_strs': None, 'embedding': [0.012144206091761589, -0.015698930248618126, 0.007664461154490709, -0.010062908753752708, -0.021275486797094345, 0.02139441855251789, -0.006689885165542364, -0.01778683438897133, -0.027882780879735947, -0.03047283925116062, 0.024301627650856972, 0.0009704462718218565, 0.0025801481679081917, 0.01097471546381712, 0.003690173616632819, 0.01857971027493477, 0.03898303583264351, -0.009613612666726112, -0.0009795313235372305, -0.019055435433983803, -0.0023604556918144226, 0.010723638348281384, 0.0035844568628817797, -0.008569660596549511, -0.003151679178699851, 0.006812119856476784, 0.025041643530130386, -0.028543509542942047, 0.023363390937447548, -0.0033779789227992296, 0.011450440622866154, -0.013901746831834316, -0.008450728841125965, -0.005563341546803713, -0.024883069097995758, -0.009395572356879711, -0.016082152724266052, -0.014469973742961884, 0.0017724066274240613, 0.00755874440073967, 0.016095368191599846, -0.0030955171678215265, -0.013888532295823097, -0.0018087467178702354, 0.02201550267636776, 0.007327489089220762, -0.010360237210988998, -0.0012966814683750272, -0.016068939119577408, -0.0013982686214148998, 0.03425221145153046, 0.013406199403107166, -0.004350902978330851, -0.024103408679366112, -0.019491517916321754, 0.0012314344057813287, 0.0026974277570843697, 0.014099964872002602, 0.0016691676573827863, -0.021275486797094345, -0.007023553363978863, 0.022174078971147537, -0.00299640791490674, 0.007413383573293686, -0.024750923737883568, 0.0007007861277088523, -0.013194765895605087, -0.005199940409511328, -0.008424299769103527, -0.0153157077729702, 0.035996537655591965, 0.00011222076136618853, -0.009904333390295506, -0.0027948853094130754, 0.019002577289938927, -0.010941678658127785, -0.00964004173874855, -0.007367132697254419, -0.009957191534340382, 0.010611314326524734, -0.00785607285797596, -0.0219230018556118, -0.0064322007820010185, 0.009560754522681236, 0.01099453680217266, -0.005642629228532314, 0.016808954998850822, 0.029627105221152306, -0.020257962867617607, -0.01844756491482258, -0.0026214439421892166, 0.017601830884814262, 0.01657109335064888, 0.015130703337490559, 0.014773909002542496, 0.0032887805718928576, -0.01564607210457325, 0.01129186525940895, -0.008840559050440788, -0.003425881965085864, -0.005804507527500391, 0.0041592917405068874, -0.007195343263447285, -0.012401890940964222, -0.01877792924642563, 0.013036190532147884, -0.0062967510893940926, -0.021909786388278008, 0.028305647894740105, -0.03242859989404678, -0.014086750335991383, 0.0202711783349514, 0.02759205922484398, -0.05277906358242035, -0.008543231524527073, -0.025226648896932602, 0.0007338225841522217, 0.000338417332386598, 0.008952883072197437, 0.0009630130953155458, 0.007968396879732609, 0.001654301187954843, 0.021962644532322884, -0.010399880819022655, 0.00944182276725769, -0.0012454749085009098, -0.015897149220108986, -0.00881412997841835, -0.011126683093607426, -0.007102841045707464, 0.017469685524702072, -0.0016848599771037698, 0.012481178157031536, -0.013478879816830158, -0.017667904496192932, 0.016610736027359962, -0.019029006361961365, 0.015342136844992638, -0.021473705768585205, -0.02525307796895504, 0.020482610911130905, 0.022914094850420952, -0.008179829455912113, 0.002928683068603277, 0.006983909755945206, 0.018473993986845016, 0.030552126467227936, 0.039062321186065674, 0.01008273009210825, -0.018698642030358315, 0.017469685524702072, -0.0025884073693305254, 0.007089626509696245, -0.008457336574792862, 0.01437747199088335, 0.035071514546871185, -0.015381780453026295, 0.012540644034743309, -0.016002865508198738, -0.01994081400334835, 0.0209319069981575, 0.025689158588647842, 0.0221476498991251, -0.008212866261601448, -0.01170151773840189, 0.03816372901201248, 0.023416249081492424, 0.014958913438022137, 0.0028097517788410187, -0.012679397128522396, -0.024050550535321236, 0.012593502178788185, -0.03351219370961189, 0.009435215964913368, 0.007281237747520208, 0.024235554039478302, -0.0026924721896648407, 0.015540354885160923, -0.01706003211438656, -0.003908214159309864, -0.012666182592511177, 0.001992099219933152, 0.02501521445810795, 0.04424244165420532, -0.008622518740594387, -0.00881412997841835, 0.00861591100692749, 0.008080720901489258, -0.004327777307480574, 0.0031896710861474276, -0.012124384753406048, 0.024473417550325394, -0.012501000426709652, -0.02449984662234783, -0.66982102394104, -0.012752077542245388, -0.0032127967569977045, -0.01967652142047882, 0.01149008423089981, 0.022755520418286324, -0.002528941724449396, 0.0365779809653759, -0.0306314155459404, 0.004301348235458136, -0.007320881821215153, 0.03858659788966179, 0.026719896122813225, 0.0004080004000570625, 0.011866699904203415, -0.011476869694888592, 0.010109160095453262, -0.02218729257583618, 0.00853001605719328, -0.015342136844992638, 0.00508431252092123, 0.02073368802666664, -0.007624817080795765, -0.0062571074813604355, 0.008576267398893833, 0.010743459686636925, 0.008992526680231094, -0.03311575576663017, 0.013835673220455647, 0.02571558766067028, -0.017839694395661354, 0.028807802125811577, 0.03647226095199585, 0.025794874876737595, 0.05513126030564308, -0.012322602793574333, -0.008378048427402973, 0.03536223620176315, 0.005566644947975874, 0.04387243092060089, -0.02062797173857689, -0.016676809638738632, 0.015619643032550812, 0.014205682091414928, 0.007710712030529976, -0.01119275577366352, 0.039564475417137146, 0.0032887805718928576, 0.035071514546871185, 0.019266869872808456, -0.005854062270373106, -0.001008438179269433, -0.001815353985875845, -0.007922145538032055, -0.0007602517725899816, -0.006504880730062723, 0.0012182198697701097, 0.015421424061059952, 0.007294452283531427, 0.017800049856305122, -0.006389253307133913, 0.03715942054986954, -0.023033026605844498, 0.004704393446445465, -0.0033829344902187586, 0.02241194061934948, -0.016544664278626442, 0.026204528287053108, -0.005606289021670818, -0.022636588662862778, 0.009778794832527637, 0.007585173472762108, -0.015289277769625187, -0.009474859572947025, 0.01335334125906229, 0.010538633912801743, 0.03274574875831604, 0.005295746028423309, -0.002448002342134714, -0.006716314237564802, -0.027023831382393837, -0.0037000845186412334, -0.04067450016736984, -0.010849176906049252, 0.02205514721572399, 0.006448718719184399, -0.03023497760295868, 0.007922145538032055, -0.003838837845250964, 0.007677675690501928, -0.01026773452758789, 0.03277217969298363, -0.007690890226513147, -0.026019522920250893, -0.012540644034743309, 0.009811831638216972, -0.013445843011140823, -0.001188487047329545, 0.0007317577837966383, 0.002581800101324916, 0.010611314326524734, -0.002264650072902441, 0.010221484117209911, 0.011338116601109505, 0.00550387566909194, 0.000641320482827723, 0.012434926815330982, 0.025530584156513214, 0.03927375376224518, -0.018923290073871613, 0.005708701908588409, 0.0009473207755945623, 0.0013354993425309658, -0.002353848423808813, -0.007869287393987179, -0.023151958361268044, 0.014033892191946507, 0.008292153477668762, 0.011001144535839558, -0.013459057547152042, -0.0014023981057107449, 0.010413095355033875, -0.00911145843565464, -0.018209701403975487, -0.005014935974031687, 0.023455893620848656, -0.009197353385388851, 0.010703816078603268, -0.011357937939465046, 0.0007561221718788147, -0.0007342355675064027, -0.010023265145719051, 0.013300483115017414, -0.01910829357802868, 0.013769600540399551, 0.0031285537406802177, -0.003805801272392273, -0.0061943382024765015, 0.00477707339450717, -0.00725480867549777, -0.020509039983153343, 0.004998418036848307, -0.005209851078689098, -0.003214448457583785, -0.017152534797787666, -0.02046939730644226, 0.01117293443530798, -0.002784974407404661, -0.012765292078256607, 0.02152656391263008, 0.003366416320204735, -0.013518523424863815, -0.022517656907439232, -0.015883933752775192, -0.01828899048268795, -0.016201084479689598, 0.01265957485884428, -0.005351908039301634, -0.01431139837950468, 0.009085029363632202, -0.0023109009489417076, 0.03628725931048393, -0.010928464122116566, 0.010155410505831242, -0.014218896627426147, 0.01036684401333332, 0.013181551359593868, 0.012296173721551895, 0.010868998244404793, -0.008701805956661701, 0.011272042989730835, -0.016148226335644722, 0.0034490074031054974, -0.005170207470655441, -0.007446420378983021, 0.012652968056499958, 0.0008837255882099271, -0.002722205128520727, -0.008477157913148403, 0.004420279525220394, -0.004479745402932167, -0.007902323268353939, -0.007882501929998398, -0.011575979180634022, 0.023706970736384392, -0.009382357820868492, 0.0012628190452232957, 0.03245502710342407, 0.016201084479689598, 0.03227002173662186, 7.371262472588569e-05, 0.014536046423017979, 0.006762565113604069, 0.009058600291609764, -0.0006429722998291254, -0.009461645036935806, 0.030049972236156464, 0.007049982436001301, 0.009930762462317944, 0.00785607285797596, -0.026191312819719315, 0.026191312819719315, 0.01399424858391285, 0.0019755808170884848, 0.017245037481188774, -0.037317994982004166, -0.004004020243883133, -0.02023153379559517, -0.008219473995268345, 0.0025223344564437866, 0.003528294852003455, -0.008153400383889675, 0.006025852169841528, -0.018989363685250282, -0.01437747199088335, 0.006141479592770338, -0.018077556043863297, 0.017152534797787666, -0.01047916803508997, 0.001551062217913568, 0.017932195216417313, -0.011873307637870312, -0.004704393446445465, -0.001744325621984899, -0.015223205089569092, 0.019861524924635887, 0.001973929116502404, -0.019385799765586853, 0.006937658414244652, -0.022068360820412636, -0.004836539272218943, -0.004796895198523998, 0.01090864185243845, 0.01802469789981842, -0.005242887884378433, 0.021143341436982155, 0.02218729257583618, -0.01544785313308239, 0.01835506223142147, -0.011675088666379452, -0.00881412997841835, 0.016861815005540848, 0.024737708270549774, -0.005655843764543533, 0.030552126467227936, -0.011959201656281948, 0.017773620784282684, -0.021169770509004593, -0.011893128976225853, -0.0057483455166220665, 0.003977591171860695, -0.012084740214049816, 0.0026743023190647364, -0.023680541664361954, 0.006250500213354826, 0.007895716466009617, 0.005355211906135082, -0.004192328080534935, 0.00868198461830616, 0.02957424707710743, -0.0042352755554020405, 0.009626827202737331, -0.003957768902182579, 0.01706003211438656, 0.0014511268818750978, 0.013703527860343456, -0.009659864008426666, -0.005081009119749069, -0.03219073638319969, -0.0105782775208354, -0.022755520418286324, -0.010644350200891495, 0.006716314237564802, -0.02126227132976055, 0.010743459686636925, 0.007307667285203934, -0.00924360379576683, -0.004988506902009249, 0.00911145843565464, -0.0010546892881393433, -0.02261015959084034, -0.0025702372658997774, 0.022821594029664993, 0.01716575026512146, 0.0043608141131699085, -0.007003731559962034, -0.007076411973685026, 0.0016361312009394169, -0.014033892191946507, -0.0004860490735154599, 0.005262709688395262, -0.002244828036054969, 0.0026528285816311836, -0.010849176906049252, 0.0041592917405068874, 0.016465377062559128, 0.03020854853093624, -0.023588038980960846, 0.023521967232227325, -0.02099798060953617, -0.03345933556556702, -0.009177531115710735, -0.022359082475304604, 0.0017509328899905086, 0.02838493511080742, 0.005150385666638613, -0.014641763642430305, -0.013095656409859657, -0.007003731559962034, -0.007023553363978863, -0.012302781455218792, 0.0009704462718218565, 0.0061447834596037865, 0.01338637713342905, 0.010360237210988998, -0.011278650723397732, -0.014694621786475182, -0.006039066705852747, 0.015936793759465218, 0.0032028856221586466, -0.026415960863232613, -0.021275486797094345, -0.029124950990080833, 0.012104562483727932, 0.1068795844912529, 0.014403901062905788, -0.01564607210457325, 0.015077845193445683, 0.009785402566194534, -0.011424011550843716, -0.016161441802978516, -0.021804070100188255, 0.01004969421774149, -0.0018682123627513647, -0.00121904572006315, -0.011873307637870312, 0.01411317940801382, -0.013716742396354675, -0.007076411973685026, -0.0016295238165184855, -3.1307215976994485e-05, -0.025755232200026512, -0.01523641962558031, -0.021315129473805428, 0.003528294852003455, 0.026230957359075546, 0.010095945559442043, 0.022636588662862778, 0.007809821516275406, -0.006267018150538206, 0.004080004058778286, 0.0010505596874281764, 0.0286492258310318, -0.01076328195631504, 0.00599281582981348, 0.0006190208368934691, 0.009461645036935806, -0.015513925813138485, 0.006415682379156351, 0.04109736904501915, 0.014086750335991383, 0.0287549439817667, 0.02525307796895504, -0.0040106275118887424, 0.031635724008083344, -0.009428608231246471, 0.014205682091414928, -0.0352565199136734, 0.014060321263968945, -0.026363102719187737, 0.0026974277570843697, 0.028173500671982765, 0.0040866113267838955, 0.0007334096007980406, 0.0024314841721206903, -0.0022894274443387985, -0.0057483455166220665, -0.007684282958507538, 0.014998557046055794, -0.00549726840108633, 0.013848887756466866, -0.0032161003910005093, -0.0018748196307569742, 0.02627060003578663, -0.010849176906049252, -0.031609293073415756, 0.009283248335123062, -0.026257386431097984, 0.007301060017198324, -0.02719562128186226, -0.013207980431616306, 0.0042716157622635365, -0.009501288644969463, -0.002657783916220069, -0.009236996993422508, -0.00550387566909194, -0.038110870867967606, -0.015831075608730316, 0.024024121463298798, 0.01835506223142147, 0.039564475417137146, -0.004829932004213333, -0.008424299769103527, 0.00785607285797596, -0.0031615900807082653, -0.02994425594806671, 0.018500423058867455, -0.018764715641736984, 0.00019966416584793478, 0.024830210953950882, -0.0059201354160904884, -0.01450961735099554, -0.020614756271243095, 0.008741449564695358, 0.0012396934907883406, 0.00745963491499424, -0.004582158289849758, -0.04244525730609894, -0.012798327952623367, -0.023218030110001564, -0.010670779272913933, 0.01257367990911007, 0.01641251891851425, -0.014443544670939445, 0.0061943382024765015, -0.023257674649357796, -0.028702083975076675, -0.006554435472935438, -0.0042484900914132595, -0.0007796607096679509, -0.005758256651461124, 0.019293298944830894, -0.022914094850420952, -0.02851708047091961, 0.03766157478094101, -0.006554435472935438, 0.0002335265453439206, 0.017588617280125618, 0.015791432932019234, 0.0345957912504673, -0.0055104829370975494, 0.008113756775856018, 0.0038586596492677927, -0.011324902065098286, -0.002576844533905387, -0.02644238993525505, 0.020680829882621765, 0.018460778519511223, -0.037873007357120514, -0.009085029363632202, -0.008060898631811142, -0.02225336618721485, -0.018870431929826736, -0.005137171130627394, 0.0004455793823581189, 0.02745991386473179, -0.031239286065101624, -0.013769600540399551, -0.012157420627772808, 0.00023311359109357, -0.007935360074043274, -5.884621259610867e-06, 0.002801492577418685, 0.0002467411395628005, -0.023482322692871094, -0.01690145768225193, -0.005612896289676428, -0.018619354814291, 0.002233265433460474, -0.02706347592175007, -0.003321816911920905, -0.006931051146239042, -0.012712433934211731, 0.03874517232179642, -0.008477157913148403, 0.0021110305096954107, 0.010003442876040936, 0.017918981611728668, -0.026521677151322365, -0.02571558766067028, -0.008721628226339817, 0.020125817507505417, 0.04067450016736984, -0.0031467238441109657, 0.019332941621541977, -0.00912467297166586, 0.03462221845984459, 0.008259117603302002, -0.00028266830486245453, 0.01584429107606411, 0.0076578538864851, 0.0031037763692438602, -0.005315567832440138, 0.03895660489797592, 0.014932484365999699, 0.0211565550416708, -0.0019425443606451154, 0.0017311109695583582, 0.007135877385735512, 0.024843424558639526, 0.0022084880620241165, -0.004228668287396431, -9.100515308091417e-05, -0.03557366877794266, 0.005692183505743742, 0.014218896627426147, -0.007109448313713074, 0.011252221651375294, 0.008622518740594387, 0.002487646182999015, 0.03425221145153046, 0.005391551647335291, 0.010545240715146065, -0.02267623320221901, 0.02231943979859352, -0.015302492305636406, 0.03382934629917145, -0.018170058727264404, -0.005325478967279196, -0.014575690031051636, -0.017311109229922295, -0.02119619958102703, 0.012421712279319763, 0.02161906659603119, 0.007836250588297844, 0.008358227089047432, -0.00020957511151209474, -0.013558167032897472, -0.00010520051000639796, -0.016082152724266052, 0.008371441625058651, 9.972884436137974e-05, 0.015209990553557873, 0.005536912474781275, 0.018196487799286842, -0.04149380698800087, -0.007505885791033506, -0.01712610572576523, 0.0013462360948324203, -0.01857971027493477, -0.018170058727264404, 0.023786257952451706, 0.0035745459608733654, -0.00955414678901434, -0.022914094850420952, -0.010750067420303822, 0.030974993482232094, -0.005814418662339449, 0.002353848423808813, -0.01875150017440319, -0.02194943092763424, -0.005705398507416248, 0.010717030614614487, 0.002978237811475992, -0.01326744630932808, 0.012692611664533615, 0.011794019490480423, 0.01437747199088335, -0.006904622074216604, 0.005127259995788336, 0.018341848626732826, -0.026455605402588844, -0.01613501086831093, -0.00467796390876174, 0.006508184596896172, -0.025966664776206017, 0.012395283207297325, -0.008219473995268345, -0.021605851128697395, 0.02666703797876835, -0.005768167786300182, -0.008543231524527073, -0.008800915442407131, -0.017945410683751106, -0.016068939119577408, 0.0101355891674757, 0.01584429107606411, 0.00632978742942214, -0.01809077151119709, 0.01089542731642723, -0.005470839329063892, -0.029257098212838173, 0.015355351381003857, 0.014258540235459805, -0.013571381568908691, 0.0068319421261549, 0.0018533458933234215, 0.02013903111219406, 0.0037661574315279722, -0.0012809891486540437, 0.0025355489924550056, -0.017733976244926453, -0.024209124967455864, 0.026402747258543968, 0.00548075046390295, -0.005021543242037296, -0.02036367915570736, 0.01242832001298666, 0.0077701774425804615, 0.026521677151322365, -0.018923290073871613, -0.009884512051939964, 0.016690025106072426, -0.009303069673478603, 0.010888820514082909, 0.004321170039474964, -0.03562653064727783, 0.009098243899643421, -0.009382357820868492, -0.018302204087376595, -0.014892840757966042, -0.011457047425210476, 0.012884222902357578, -0.031609293073415756, -0.04247168451547623, -0.0033251207787543535, 0.011014359071850777, 0.012553858570754528, -0.03647226095199585, -0.003501865779981017, 0.015738574787974358, 0.0273277685046196, -0.029759252443909645, 0.012606716714799404, -0.007228379603475332, -0.013412807136774063, -0.007677675690501928, 0.005880491808056831, -0.012699219398200512, -0.008444122038781643, 0.039035893976688385, -0.0013478879118338227, -0.025160575285553932, -0.009316284209489822, -0.0010183491976931691, -0.013201373629271984, -0.022570516914129257, -0.009580575861036777, -0.023521967232227325, 0.008933061733841896, -0.008767878636717796, -0.02849065139889717, -0.008193044923245907, 0.010188447311520576, 0.004208846017718315, -0.004070092923939228, 0.004790287930518389, -0.004780377261340618, -0.004453316330909729, 0.0062934476882219315, -0.003538205986842513, -0.0028543509542942047, -0.024975571781396866, 0.004783680662512779, 0.0001777775032678619, 0.014575690031051636, -0.022861236706376076, 0.00725480867549777, -0.02069404534995556, 0.002936942270025611, -0.0015444549499079585, 0.03364434093236923, 0.021975859999656677, 0.0025124235544353724, -0.003954465501010418, 0.028702083975076675, 0.005563341546803713, 0.012454749085009098, 0.01544785313308239, -0.009858082979917526, -0.0027106422930955887, 0.000394372851587832, -0.04323813319206238, -0.01038005854934454, 0.016346445307135582, 0.036392975598573685, 0.009904333390295506, -0.01277189888060093, -0.01957080513238907, -0.012296173721551895, -0.03821658715605736, -0.00755213713273406, 0.00736052542924881, 0.0006103487685322762, 0.017271466553211212, -0.008510194718837738, -0.007043375167995691, 0.007915537804365158, -0.017271466553211212, -0.0021870143245905638, -0.0005079357069917023, -0.010802925564348698, -0.020416539162397385, 0.01014880370348692, 0.01877792924642563, -0.001067077973857522, -0.0005087616154924035, -0.004076700191944838, 0.0007639683899469674, -0.011675088666379452, -0.002487646182999015, -0.02304624207317829, -0.01034702267497778, -0.002573540899902582, -0.009633434936404228, 0.03575867414474487, 0.013009761460125446, 0.00035803273203782737, 0.004268311895430088, -0.0286492258310318, -0.00962022040039301, -0.0018748196307569742, -0.020654400810599327, -0.0026511766482144594, -0.010062908753752708, 0.016623951494693756, -0.004195631481707096, 0.008675376884639263, 0.008232688531279564, -0.014932484365999699, 0.0153157077729702, 0.014284969307482243, 0.027116334065794945, -0.020773332566022873, -0.01881757378578186, 0.025134146213531494, -0.00631326949223876, -0.01242832001298666, -0.022663017734885216, 0.021711567416787148, -0.00233072298578918, -0.0020135727245360613, -0.014456759206950665, -0.03597010672092438, 0.014747479930520058, 0.00012626126408576965, -0.004995114170014858, -0.006785690784454346, 0.004674660507589579, -0.0038322305772453547, -0.0015279367798939347, 0.011113468557596207, 0.012593502178788185, -0.0002702796191442758, -0.019266869872808456, 0.006729528773576021, 0.013901746831834316, -0.006138176191598177, -0.00716230645775795, -0.0254512969404459, 0.02538522332906723, -0.024090193212032318, -0.021037623286247253, 0.005877187941223383, -0.013042798265814781, -0.0022217025980353355, -0.017522543668746948, 0.012481178157031536, -0.0006355390651151538, 0.004017234779894352, 0.24145695567131042, 0.006864978466182947, -0.0040866113267838955, 0.011364545673131943, -0.023984476923942566, 0.007188735995441675, 0.03734442591667175, 0.00950789637863636, -0.028702083975076675, 0.015672501176595688, -0.01670323871076107, -0.021605851128697395, 0.01712610572576523, 0.0035844568628817797, 0.007340703625231981, -0.011556156910955906, -0.029653534293174744, -0.0013875317526981235, -0.002456261543557048, -0.025808090344071388, 0.01828899048268795, -0.011853485368192196, -0.029098521918058395, -0.015461067669093609, 0.01877792924642563, 0.036260828375816345, 0.001434608711861074, 0.0008007214055396616, 0.010604706592857838, 0.005728523712605238, -0.0016047465614974499, -0.009871297515928745, 0.0009522762265987694, -0.011945987120270729, -0.027644917368888855, 0.0025983182713389397, 0.018738284707069397, -0.003311906009912491, -0.003343290649354458, -0.007049982436001301, 0.01822291687130928, 0.0036868699826300144, 0.011483476497232914, -0.007915537804365158, -0.021235842257738113, 0.05407409369945526, -0.01617465540766716, 0.0019029006361961365, -0.004714304115623236, 0.02056189812719822, -0.004892701283097267, -0.002264650072902441, 0.01961044780910015, 0.01904222182929516, -0.024962356314063072, 0.0018219612538814545, 0.011615622788667679, 0.007829642854630947, -0.007466242182999849, 0.01398103404790163, 0.00755213713273406, 0.02009938843548298, -0.02432805672287941, 0.01046595349907875, 0.00507440185174346, 0.012481178157031536, -0.011311687529087067, -0.004631713032722473, -0.0025933629367500544, -0.04506174474954605, -0.0015923578757792711, -0.008549838326871395, -0.021737996488809586, 0.00784285832196474, -0.026812398806214333, -0.014853197149932384, 0.004948863293975592, 0.03102785162627697, 0.018526852130889893, 0.026852043345570564, -0.0036241007037460804, -0.006055585108697414, 0.006878193002194166, -0.0007602517725899816, -0.0319528728723526, -0.04545818269252777, 0.01411317940801382, 0.0045359074138104916, -0.006713010836392641, -0.02756563015282154, 0.0008073287317529321, -0.0030294442549347878, -0.006765868980437517, 0.003452311037108302, 0.002398447599261999, -0.012071525678038597, -0.015209990553557873, 0.03385577350854874, -0.00861591100692749, -0.005451017525047064, -0.014562475495040417, -0.009177531115710735, 0.0221476498991251, 0.009375750087201595, -0.010043086484074593, 0.013505308888852596, 0.0018450868083164096, 0.01795862428843975, 0.0042848302982747555, -0.028464222326874733, -0.0039610727690160275, -0.032957181334495544, -0.0020251355599611998, 0.0030459624249488115, 0.008153400383889675, -0.0034853476099669933, -0.0031731529161334038, -0.021381203085184097, 0.006422289647161961, -0.005599681753665209, -0.03340647742152214, -0.001721200067549944, 0.002978237811475992, 0.018923290073871613, -0.003947858233004808, -0.003125250106677413, -0.019993672147393227, -0.002821314614266157, 0.005527001339942217, -0.046594638377428055, 0.01637287437915802, -0.015989651903510094, 0.003059177193790674, 0.00932289194315672, -0.012606716714799404, 0.01842113584280014, -0.010043086484074593, 0.001954107079654932, 0.0023241157177835703, 0.017324324697256088, 0.012375461868941784, -0.022068360820412636, 0.01736396923661232, 0.00045342554221861064, 0.007849465124309063, -0.03541509434580803, 0.012983332388103008, 0.02185692824423313, 0.004020538181066513, -0.029600676149129868, -0.008252509869635105, -0.004611891228705645, -0.012976725585758686, 0.01213099155575037, 0.010360237210988998, -0.04413672164082527, -0.04458601772785187, -0.027644917368888855, -0.011258828453719616, -0.009058600291609764, -0.007466242182999849, 0.012487785890698433, 0.018632568418979645, 0.010413095355033875, -0.0025173788890242577, -0.00808732770383358, -0.1692524403333664, 0.02666703797876835, 0.023429464548826218, -0.020376894623041153, 0.005873884540051222, -0.0025751928333193064, 0.033195044845342636, -0.01036684401333332, -0.01672966778278351, 0.024750923737883568, 0.019319728016853333, 0.006128265056759119, -0.02013903111219406, 0.02284802310168743, -0.007512493059039116, 0.013009761460125446, -0.014588904567062855, 0.03290432319045067, 0.019399015232920647, 0.0063694315031170845, 0.03182072937488556, 0.008536623790860176, 0.012798327952623367, 0.002411662368103862, 0.01339959166944027, 0.01505141519010067, 0.003465525573119521, 0.0018913379171863198, 0.0018748196307569742, -0.005150385666638613, 0.008734842762351036, -8.58432031236589e-05, -0.006950873415917158, -0.022623375058174133, 0.0300764013081789, 0.008820737712085247, -0.00755874440073967, -0.002223354298621416, 0.001600616960786283, 0.015897149220108986, 0.02970639429986477, 0.02089226432144642, 0.007789999712258577, 0.008093935437500477, -0.011424011550843716, 0.026230957359075546, 0.014086750335991383, -0.002994755981490016, 0.01379602961242199, -0.02013903111219406, 0.01437747199088335, -0.023099100217223167, -0.0015428031329065561, 0.00021948604262433946, 0.014879626221954823, -0.005137171130627394, 0.015817862004041672, 0.025279507040977478, -0.007611602544784546, -0.014549260959029198, 0.009435215964913368, -0.02227979525923729, 0.029098521918058395, 0.010307378135621548, 0.008503586985170841, -0.013056012801826, -0.014813552610576153, -0.015804646536707878, -0.03205858916044235, 0.021182984113693237, -0.02023153379559517, -0.008100542239844799, 0.01297011785209179, -0.0009754017810337245, 0.0031235981732606888, 0.004443405196070671, -0.004707696847617626, -0.027777064591646194, 0.0005570774665102363, 0.03594367951154709, -0.017086463049054146, 0.013320304453372955, -0.008021255023777485, 0.004902611952275038, -0.010703816078603268, 0.025279507040977478, -0.013505308888852596, 0.015461067669093609, -0.00882734451442957, -0.013379770331084728, 0.023482322692871094, -0.018936503678560257, -0.029600676149129868, -0.012421712279319763, 0.026164883747696877, 0.009910941123962402, 0.004843146540224552, 0.0033019951079040766, 0.0022117916960269213, -0.022517656907439232, 0.007796606980264187, -0.026759540662169456, 0.01180062722414732, -0.020218320190906525, 0.032296452671289444, 0.003075695363804698, 0.011298472993075848, 0.00964004173874855, 0.033353619277477264, -0.010300771333277225, -0.001560147269628942, -0.004195631481707096, 0.004839842673391104, 0.007268023211508989, -0.016544664278626442, 0.0378994382917881, -0.006105139385908842, -0.007083019241690636, 0.0019227225566282868, -0.005424588453024626, 0.058778487145900726, 0.014443544670939445, -0.027222050353884697, 0.001443693763576448, 0.0004645753651857376, 0.0009390616323798895, -0.12326567620038986, -0.004929041489958763, 0.0027453305665403605, -0.0024446987081319094, 0.01943865790963173, 0.00436411751434207, -0.0273277685046196, -0.014496402814984322, -0.009283248335123062, 0.009692899882793427, -0.009362535551190376, -0.013511915691196918, 0.0024678243789821863, -0.019782237708568573, -0.0015386735321953893, -0.010545240715146065, 0.01891007460653782, -0.02412983775138855, -0.01692788675427437, 0.040330920368433, -0.006898014806210995, -0.005543519742786884, -0.010902035050094128, -0.012633145786821842, 0.004862968344241381, -0.01894971914589405, -0.03456936031579971, 0.0010034827282652259, 0.006818727124482393, 0.005305657163262367, -0.0012760336976498365, -0.025557013228535652, 0.0017740584444254637, -0.013102264143526554, 0.0159764364361763, -0.00862912554293871, -0.01150329876691103, 0.011919558048248291, 0.012395283207297325, -0.01868542656302452, -0.0021407632157206535, 0.01168830320239067, 0.01954437606036663, -0.05092902109026909, -0.003052569692954421, -0.00014876735804136842, -0.020614756271243095, 0.017773620784282684, 0.008992526680231094, -0.014760694466531277, -0.017733976244926453, 0.016280371695756912, -0.025160575285553932, -0.013928175903856754, -0.007717319298535585, 0.0004459923366084695, 0.01024791318923235, 0.009527717716991901, 0.008444122038781643, 0.021909786388278008, -0.01577821746468544, -0.0107963178306818, -0.005705398507416248, -0.019055435433983803, 0.015394994989037514, -0.007968396879732609, -0.004935648757964373, -0.0021143341436982155, -0.014060321263968945, -0.05248834192752838, -0.030393552035093307, -0.004734125919640064, -0.028702083975076675, 0.0042484900914132595, 0.0007197820814326406, -0.000820130342617631, -0.013967819511890411, 0.0022745609749108553, 0.011714732274413109, -0.02597988024353981, -0.010175232775509357, -0.004843146540224552, 0.009732544422149658, -0.028147071599960327, 0.015791432932019234, 0.0016509975539520383, 0.0004214214568492025, 0.009646649472415447, -0.0008011343888938427, 0.00010509727144381031, -0.03205858916044235, 0.028332076966762543, -0.00036567242932505906, -0.013743171468377113, -0.007102841045707464, 0.007426598574966192, -0.001273555913940072, -0.0031103836372494698, 0.009091636165976524, -0.0016782527090981603, -0.0084837656468153, 0.0016485198866575956, -0.03895660489797592, 0.021909786388278008, -0.009712722152471542, -0.01256046537309885, 0.0153157077729702, -0.004810110200196505, 0.004344295710325241, -0.025504155084490776, -0.03523009270429611, 0.006055585108697414, -0.009877904318273067, 0.02743348479270935, 0.021235842257738113, -0.016848599538207054, -0.0143246129155159, -0.021473705768585205, 0.005804507527500391, 0.025755232200026512, 0.007003731559962034, 0.02238551154732704, 0.0013412806438282132, -0.009712722152471542, 0.026323458179831505, 0.009930762462317944, -0.010320592671632767, 0.0041758096776902676, -0.022398727014660835, 0.015751788392663002, -0.014020677655935287, -0.0041758096776902676, 0.0028411364182829857, -0.02397126331925392, 0.009085029363632202, 0.020852619782090187, 0.019399015232920647, 0.008008040487766266, 0.004093218594789505, 0.026112025603652, 0.016597522422671318, 0.06099853664636612, -0.009065207093954086, -0.01683538407087326, -0.005424588453024626, -0.03721227869391441, 0.023033026605844498, -0.010915249586105347, -0.02954781800508499, 0.019055435433983803, 0.025557013228535652, 0.004823324736207724, 0.02142084762454033, 0.011397582478821278, -0.006898014806210995, -0.007122662849724293, -0.0038751778192818165, -0.037238709628582, 0.0024446987081319094, 0.008120364509522915, -0.01150329876691103, -0.015024986118078232, 0.03179429844021797, -0.010948286391794682, 0.001663386239670217, -0.02373339980840683, 0.015461067669093609, -0.016399303451180458, 0.01044613216072321, 0.012818150222301483, -0.006871585734188557, -0.033750057220458984, -0.010571670718491077, -0.002182058757171035, 0.011542942374944687, 0.03208502009510994, 0.01110686082392931, 0.0022084880620241165, -0.001183531479910016, 0.03803158551454544, -0.005236280616372824, 0.02944210171699524, -0.009137887507677078, 0.01119275577366352, -0.0027189014945179224, 0.021777641028165817, 0.023521967232227325, 0.0235748253762722, 0.010320592671632767, -0.0016832081601023674, -0.002233265433460474, 0.003980894573032856, -0.003282173303887248, -0.01047916803508997, 0.004853057209402323, -0.01173455361276865, -0.002753589767962694, 0.02716919220983982, -0.007499278523027897, 0.008245903067290783, 0.01924044080078602, 0.01505141519010067, 0.015950007364153862, -0.01472105085849762, 0.009078421629965305, -0.015091059729456902, -0.022174078971147537, 0.04191667214035988, -0.0032408777624368668, -0.028305647894740105, 0.003954465501010418, 0.021103696897625923, 0.005384944379329681, -0.0009572317358106375, -0.010637743398547173, 0.0107963178306818, -0.015077845193445683, 0.018923290073871613, -0.0033862381242215633, -0.013307089917361736, -0.010789711028337479, 0.034410785883665085, -0.00445661973208189, 0.004189024213701487, 0.009547539986670017, -0.022028718143701553, -0.012732255272567272, 0.0244602020829916, 0.012210278771817684, 0.0038091049063950777, 0.006210856139659882, -0.018989363685250282, -0.009098243899643421, -0.01604251004755497, -0.010056301020085812, -0.008126971311867237, 0.011774198152124882, -0.011457047425210476, -0.0014808597043156624, 0.019887953996658325, -0.036234401166439056, 0.02089226432144642, -0.002771759871393442, -0.006891407538205385, -0.011014359071850777, 0.013089049607515335, 0.01961044780910015, 0.014126394875347614, -0.007968396879732609, -0.012203671969473362, -0.0037496392615139484, 0.015011771582067013, -0.0013957908377051353, 0.029785681515932083, -0.028939947485923767, -0.033591482788324356, -0.012012060731649399, 0.009018956683576107, -0.006191034335643053, 0.015157132409512997, 0.023482322692871094, 0.018315419554710388, 0.008642340078949928, 0.02327089011669159, 0.011001144535839558, -0.021011194214224815, 0.0073539181612432, 0.037978727370500565, -0.017271466553211212, -0.006696492433547974, -0.038507308810949326, -0.004694482311606407, 0.005943261086940765, -0.0559769943356514, -0.024869853630661964, 0.01431139837950468, -0.03377648815512657, -0.009805223904550076, -0.027089904993772507, -0.0004559032677207142, 0.013128693215548992, -0.012534036301076412, 0.003640618873760104, -0.02175121195614338, 0.0084837656468153, 0.004426886793226004, -0.003792586736381054, -0.004998418036848307, -0.0017971839988604188, -0.02838493511080742]}}, rets=[{'node': {'id_': 'bcf8cce5-6f77-4391-8eae-1bd1a3749094', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': 'f3ad7036-414b-4f27-adcd-52244cbb27eb', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '18b69e4c8c3b6423e75e5c31805f7ee1ac2e3d8b1a20ed7fabef56d3845272c7'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'c3d00853-0daa-4514-9f1c-1a8b67b5d58e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '0990d9b6dd8f781bc862b7c85f53eb0613d198313f968d3b4743544aa84ffe58'}}, 'text': '@Teslarati @KlenderJoey Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet, 391, 6778, 2022-09-21 15:59:15, Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich I was instrumental in creating OpenAI, which made DALL-E, 153, 2442, 2022-09-21 15:17:17, I was instrumental in creating OpenAI, which made DALL-E\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist), 81, 1416, 2022-09-21 15:09:59, Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist)\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America., 64, 1285, 2022-09-21 15:00:26, Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America.\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich For sure, 58, 1171, 2022-09-21 14:55:11, For sure\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(, 154, 2286, 2022-09-21 14:36:49, Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich This is true, 60, 1356, 2022-09-21 14:26:17, This is true\\n@Teslarati @ResidentSponge With some improvements, it will achieve 300 Mps, 651, 14338, 2022-09-21 14:02:11, With some improvements, it will achieve 300 Mps\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.\\n\\nAlso, I was there for 2.5 years, previously attending govt schools for 9 of 12 years., 89, 1694, 2022-09-21 13:55:04, Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.Also, I was there for 2.5 years, previously attending govt schools for 9 of 12 years.', 'start_char_idx': 82792, 'end_char_idx': 85782, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7469842252500036}, {'node': {'id_': 'de6f0f78-221e-4fe4-b85d-6a992f6a6f6d', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': '3e563428-76cc-4aa0-83ee-3e8049217a83', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '9d14ec9cda85a9c8cc3c34b1cdf75fb105795e73d699b35cfc8234703dc37635'}, <NodeRelationship.NEXT: '3'>: {'node_id': '473c5d14-ac7c-45f4-a7eb-fb6ecc383b07', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'fb859d528497fbefaebc580603c2b0b92cca496d280392da3942cd40754a5cfc'}}, 'text': 'Walter Isaacson is writing a book., 215, 4004, 2022-04-21 08:49:51, I should, but that would take a lot of time away from Tesla, SpaceX &amp; other work. Walter Isaacson is writing a book.\\n@PPathole @teslaownersSV @woonomic @VaibhavSisinty Exactly, 137, 3587, 2022-04-21 08:42:11, Exactly\\n@billhuang688 @woonomic @VaibhavSisinty I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. \\n\\nIf it was really his baby, he would never have stood by while it starved to death.\\n\\nhttps://t.co/ZDfYLD5qkL, 355, 4152, 2022-04-21 08:40:07, I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. If it was really his baby, he would never have stood by while it starved to death.\\n@teslaownersSV @woonomic @VaibhavSisinty But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.\\n\\nWe would’ve lost at least a few talented people if he was the real deal, but we lost no one., 395, 5547, 2022-04-21 08:33:10, But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.We would’ve lost at least a few talented people if he was the real deal, but we lost no one.\\n@teslaownersSV @woonomic @VaibhavSisinty Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years., 452, 6083, 2022-04-21 08:27:57, Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years.\\n@woonomic @VaibhavSisinty They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!, 1071, 10676, 2022-04-21 08:21:40, They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!\\n@woonomic @VaibhavSisinty When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all., 425, 8706, 2022-04-21 08:15:37, When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all.\\n@woonomic @VaibhavSisinty If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for https://t.co/bOUOek5Cvy (later renamed PayPal), but that’s not what founding means., 506, 8631, 2022-04-21 08:12:03, If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for  (later renamed PayPal), but that’s not what founding means.\\n@woonomic @VaibhavSisinty Not even close to that. It was a shell corp with no employees, no IP, no designs, no prototype, literally nothing but a biz plan to commercialize AC Propulsion’s Tzero car, which was introduced to me by JB Straubel, *not* Eberhard. \\n\\nEven name “Tesla Motors” was owned by others!, 1324, 17242, 2022-04-21 08:09:07, Not even close to that.', 'start_char_idx': 380380, 'end_char_idx': 383649, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7451874390981541}], error=None, perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 12, 0, 582986), end_time=datetime.datetime(2024, 2, 29, 16, 12, 1, 257848)), pid=32024, tid=20596), RecordAppCall(stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='retrieve')), RecordAppCallMethod(path=Lens().app._retriever, method=Method(obj=Obj(cls=llama_index.core.indices.vector_store.retrievers.retriever.VectorIndexRetriever, id=2416092148304, init_bindings=None), name='retrieve'))], args={'str_or_query_bundle': {'query_str': 'What did the author do growing up?', 'image_path': None, 'custom_embedding_strs': None, 'embedding': [0.012144206091761589, -0.015698930248618126, 0.007664461154490709, -0.010062908753752708, -0.021275486797094345, 0.02139441855251789, -0.006689885165542364, -0.01778683438897133, -0.027882780879735947, -0.03047283925116062, 0.024301627650856972, 0.0009704462718218565, 0.0025801481679081917, 0.01097471546381712, 0.003690173616632819, 0.01857971027493477, 0.03898303583264351, -0.009613612666726112, -0.0009795313235372305, -0.019055435433983803, -0.0023604556918144226, 0.010723638348281384, 0.0035844568628817797, -0.008569660596549511, -0.003151679178699851, 0.006812119856476784, 0.025041643530130386, -0.028543509542942047, 0.023363390937447548, -0.0033779789227992296, 0.011450440622866154, -0.013901746831834316, -0.008450728841125965, -0.005563341546803713, -0.024883069097995758, -0.009395572356879711, -0.016082152724266052, -0.014469973742961884, 0.0017724066274240613, 0.00755874440073967, 0.016095368191599846, -0.0030955171678215265, -0.013888532295823097, -0.0018087467178702354, 0.02201550267636776, 0.007327489089220762, -0.010360237210988998, -0.0012966814683750272, -0.016068939119577408, -0.0013982686214148998, 0.03425221145153046, 0.013406199403107166, -0.004350902978330851, -0.024103408679366112, -0.019491517916321754, 0.0012314344057813287, 0.0026974277570843697, 0.014099964872002602, 0.0016691676573827863, -0.021275486797094345, -0.007023553363978863, 0.022174078971147537, -0.00299640791490674, 0.007413383573293686, -0.024750923737883568, 0.0007007861277088523, -0.013194765895605087, -0.005199940409511328, -0.008424299769103527, -0.0153157077729702, 0.035996537655591965, 0.00011222076136618853, -0.009904333390295506, -0.0027948853094130754, 0.019002577289938927, -0.010941678658127785, -0.00964004173874855, -0.007367132697254419, -0.009957191534340382, 0.010611314326524734, -0.00785607285797596, -0.0219230018556118, -0.0064322007820010185, 0.009560754522681236, 0.01099453680217266, -0.005642629228532314, 0.016808954998850822, 0.029627105221152306, -0.020257962867617607, -0.01844756491482258, -0.0026214439421892166, 0.017601830884814262, 0.01657109335064888, 0.015130703337490559, 0.014773909002542496, 0.0032887805718928576, -0.01564607210457325, 0.01129186525940895, -0.008840559050440788, -0.003425881965085864, -0.005804507527500391, 0.0041592917405068874, -0.007195343263447285, -0.012401890940964222, -0.01877792924642563, 0.013036190532147884, -0.0062967510893940926, -0.021909786388278008, 0.028305647894740105, -0.03242859989404678, -0.014086750335991383, 0.0202711783349514, 0.02759205922484398, -0.05277906358242035, -0.008543231524527073, -0.025226648896932602, 0.0007338225841522217, 0.000338417332386598, 0.008952883072197437, 0.0009630130953155458, 0.007968396879732609, 0.001654301187954843, 0.021962644532322884, -0.010399880819022655, 0.00944182276725769, -0.0012454749085009098, -0.015897149220108986, -0.00881412997841835, -0.011126683093607426, -0.007102841045707464, 0.017469685524702072, -0.0016848599771037698, 0.012481178157031536, -0.013478879816830158, -0.017667904496192932, 0.016610736027359962, -0.019029006361961365, 0.015342136844992638, -0.021473705768585205, -0.02525307796895504, 0.020482610911130905, 0.022914094850420952, -0.008179829455912113, 0.002928683068603277, 0.006983909755945206, 0.018473993986845016, 0.030552126467227936, 0.039062321186065674, 0.01008273009210825, -0.018698642030358315, 0.017469685524702072, -0.0025884073693305254, 0.007089626509696245, -0.008457336574792862, 0.01437747199088335, 0.035071514546871185, -0.015381780453026295, 0.012540644034743309, -0.016002865508198738, -0.01994081400334835, 0.0209319069981575, 0.025689158588647842, 0.0221476498991251, -0.008212866261601448, -0.01170151773840189, 0.03816372901201248, 0.023416249081492424, 0.014958913438022137, 0.0028097517788410187, -0.012679397128522396, -0.024050550535321236, 0.012593502178788185, -0.03351219370961189, 0.009435215964913368, 0.007281237747520208, 0.024235554039478302, -0.0026924721896648407, 0.015540354885160923, -0.01706003211438656, -0.003908214159309864, -0.012666182592511177, 0.001992099219933152, 0.02501521445810795, 0.04424244165420532, -0.008622518740594387, -0.00881412997841835, 0.00861591100692749, 0.008080720901489258, -0.004327777307480574, 0.0031896710861474276, -0.012124384753406048, 0.024473417550325394, -0.012501000426709652, -0.02449984662234783, -0.66982102394104, -0.012752077542245388, -0.0032127967569977045, -0.01967652142047882, 0.01149008423089981, 0.022755520418286324, -0.002528941724449396, 0.0365779809653759, -0.0306314155459404, 0.004301348235458136, -0.007320881821215153, 0.03858659788966179, 0.026719896122813225, 0.0004080004000570625, 0.011866699904203415, -0.011476869694888592, 0.010109160095453262, -0.02218729257583618, 0.00853001605719328, -0.015342136844992638, 0.00508431252092123, 0.02073368802666664, -0.007624817080795765, -0.0062571074813604355, 0.008576267398893833, 0.010743459686636925, 0.008992526680231094, -0.03311575576663017, 0.013835673220455647, 0.02571558766067028, -0.017839694395661354, 0.028807802125811577, 0.03647226095199585, 0.025794874876737595, 0.05513126030564308, -0.012322602793574333, -0.008378048427402973, 0.03536223620176315, 0.005566644947975874, 0.04387243092060089, -0.02062797173857689, -0.016676809638738632, 0.015619643032550812, 0.014205682091414928, 0.007710712030529976, -0.01119275577366352, 0.039564475417137146, 0.0032887805718928576, 0.035071514546871185, 0.019266869872808456, -0.005854062270373106, -0.001008438179269433, -0.001815353985875845, -0.007922145538032055, -0.0007602517725899816, -0.006504880730062723, 0.0012182198697701097, 0.015421424061059952, 0.007294452283531427, 0.017800049856305122, -0.006389253307133913, 0.03715942054986954, -0.023033026605844498, 0.004704393446445465, -0.0033829344902187586, 0.02241194061934948, -0.016544664278626442, 0.026204528287053108, -0.005606289021670818, -0.022636588662862778, 0.009778794832527637, 0.007585173472762108, -0.015289277769625187, -0.009474859572947025, 0.01335334125906229, 0.010538633912801743, 0.03274574875831604, 0.005295746028423309, -0.002448002342134714, -0.006716314237564802, -0.027023831382393837, -0.0037000845186412334, -0.04067450016736984, -0.010849176906049252, 0.02205514721572399, 0.006448718719184399, -0.03023497760295868, 0.007922145538032055, -0.003838837845250964, 0.007677675690501928, -0.01026773452758789, 0.03277217969298363, -0.007690890226513147, -0.026019522920250893, -0.012540644034743309, 0.009811831638216972, -0.013445843011140823, -0.001188487047329545, 0.0007317577837966383, 0.002581800101324916, 0.010611314326524734, -0.002264650072902441, 0.010221484117209911, 0.011338116601109505, 0.00550387566909194, 0.000641320482827723, 0.012434926815330982, 0.025530584156513214, 0.03927375376224518, -0.018923290073871613, 0.005708701908588409, 0.0009473207755945623, 0.0013354993425309658, -0.002353848423808813, -0.007869287393987179, -0.023151958361268044, 0.014033892191946507, 0.008292153477668762, 0.011001144535839558, -0.013459057547152042, -0.0014023981057107449, 0.010413095355033875, -0.00911145843565464, -0.018209701403975487, -0.005014935974031687, 0.023455893620848656, -0.009197353385388851, 0.010703816078603268, -0.011357937939465046, 0.0007561221718788147, -0.0007342355675064027, -0.010023265145719051, 0.013300483115017414, -0.01910829357802868, 0.013769600540399551, 0.0031285537406802177, -0.003805801272392273, -0.0061943382024765015, 0.00477707339450717, -0.00725480867549777, -0.020509039983153343, 0.004998418036848307, -0.005209851078689098, -0.003214448457583785, -0.017152534797787666, -0.02046939730644226, 0.01117293443530798, -0.002784974407404661, -0.012765292078256607, 0.02152656391263008, 0.003366416320204735, -0.013518523424863815, -0.022517656907439232, -0.015883933752775192, -0.01828899048268795, -0.016201084479689598, 0.01265957485884428, -0.005351908039301634, -0.01431139837950468, 0.009085029363632202, -0.0023109009489417076, 0.03628725931048393, -0.010928464122116566, 0.010155410505831242, -0.014218896627426147, 0.01036684401333332, 0.013181551359593868, 0.012296173721551895, 0.010868998244404793, -0.008701805956661701, 0.011272042989730835, -0.016148226335644722, 0.0034490074031054974, -0.005170207470655441, -0.007446420378983021, 0.012652968056499958, 0.0008837255882099271, -0.002722205128520727, -0.008477157913148403, 0.004420279525220394, -0.004479745402932167, -0.007902323268353939, -0.007882501929998398, -0.011575979180634022, 0.023706970736384392, -0.009382357820868492, 0.0012628190452232957, 0.03245502710342407, 0.016201084479689598, 0.03227002173662186, 7.371262472588569e-05, 0.014536046423017979, 0.006762565113604069, 0.009058600291609764, -0.0006429722998291254, -0.009461645036935806, 0.030049972236156464, 0.007049982436001301, 0.009930762462317944, 0.00785607285797596, -0.026191312819719315, 0.026191312819719315, 0.01399424858391285, 0.0019755808170884848, 0.017245037481188774, -0.037317994982004166, -0.004004020243883133, -0.02023153379559517, -0.008219473995268345, 0.0025223344564437866, 0.003528294852003455, -0.008153400383889675, 0.006025852169841528, -0.018989363685250282, -0.01437747199088335, 0.006141479592770338, -0.018077556043863297, 0.017152534797787666, -0.01047916803508997, 0.001551062217913568, 0.017932195216417313, -0.011873307637870312, -0.004704393446445465, -0.001744325621984899, -0.015223205089569092, 0.019861524924635887, 0.001973929116502404, -0.019385799765586853, 0.006937658414244652, -0.022068360820412636, -0.004836539272218943, -0.004796895198523998, 0.01090864185243845, 0.01802469789981842, -0.005242887884378433, 0.021143341436982155, 0.02218729257583618, -0.01544785313308239, 0.01835506223142147, -0.011675088666379452, -0.00881412997841835, 0.016861815005540848, 0.024737708270549774, -0.005655843764543533, 0.030552126467227936, -0.011959201656281948, 0.017773620784282684, -0.021169770509004593, -0.011893128976225853, -0.0057483455166220665, 0.003977591171860695, -0.012084740214049816, 0.0026743023190647364, -0.023680541664361954, 0.006250500213354826, 0.007895716466009617, 0.005355211906135082, -0.004192328080534935, 0.00868198461830616, 0.02957424707710743, -0.0042352755554020405, 0.009626827202737331, -0.003957768902182579, 0.01706003211438656, 0.0014511268818750978, 0.013703527860343456, -0.009659864008426666, -0.005081009119749069, -0.03219073638319969, -0.0105782775208354, -0.022755520418286324, -0.010644350200891495, 0.006716314237564802, -0.02126227132976055, 0.010743459686636925, 0.007307667285203934, -0.00924360379576683, -0.004988506902009249, 0.00911145843565464, -0.0010546892881393433, -0.02261015959084034, -0.0025702372658997774, 0.022821594029664993, 0.01716575026512146, 0.0043608141131699085, -0.007003731559962034, -0.007076411973685026, 0.0016361312009394169, -0.014033892191946507, -0.0004860490735154599, 0.005262709688395262, -0.002244828036054969, 0.0026528285816311836, -0.010849176906049252, 0.0041592917405068874, 0.016465377062559128, 0.03020854853093624, -0.023588038980960846, 0.023521967232227325, -0.02099798060953617, -0.03345933556556702, -0.009177531115710735, -0.022359082475304604, 0.0017509328899905086, 0.02838493511080742, 0.005150385666638613, -0.014641763642430305, -0.013095656409859657, -0.007003731559962034, -0.007023553363978863, -0.012302781455218792, 0.0009704462718218565, 0.0061447834596037865, 0.01338637713342905, 0.010360237210988998, -0.011278650723397732, -0.014694621786475182, -0.006039066705852747, 0.015936793759465218, 0.0032028856221586466, -0.026415960863232613, -0.021275486797094345, -0.029124950990080833, 0.012104562483727932, 0.1068795844912529, 0.014403901062905788, -0.01564607210457325, 0.015077845193445683, 0.009785402566194534, -0.011424011550843716, -0.016161441802978516, -0.021804070100188255, 0.01004969421774149, -0.0018682123627513647, -0.00121904572006315, -0.011873307637870312, 0.01411317940801382, -0.013716742396354675, -0.007076411973685026, -0.0016295238165184855, -3.1307215976994485e-05, -0.025755232200026512, -0.01523641962558031, -0.021315129473805428, 0.003528294852003455, 0.026230957359075546, 0.010095945559442043, 0.022636588662862778, 0.007809821516275406, -0.006267018150538206, 0.004080004058778286, 0.0010505596874281764, 0.0286492258310318, -0.01076328195631504, 0.00599281582981348, 0.0006190208368934691, 0.009461645036935806, -0.015513925813138485, 0.006415682379156351, 0.04109736904501915, 0.014086750335991383, 0.0287549439817667, 0.02525307796895504, -0.0040106275118887424, 0.031635724008083344, -0.009428608231246471, 0.014205682091414928, -0.0352565199136734, 0.014060321263968945, -0.026363102719187737, 0.0026974277570843697, 0.028173500671982765, 0.0040866113267838955, 0.0007334096007980406, 0.0024314841721206903, -0.0022894274443387985, -0.0057483455166220665, -0.007684282958507538, 0.014998557046055794, -0.00549726840108633, 0.013848887756466866, -0.0032161003910005093, -0.0018748196307569742, 0.02627060003578663, -0.010849176906049252, -0.031609293073415756, 0.009283248335123062, -0.026257386431097984, 0.007301060017198324, -0.02719562128186226, -0.013207980431616306, 0.0042716157622635365, -0.009501288644969463, -0.002657783916220069, -0.009236996993422508, -0.00550387566909194, -0.038110870867967606, -0.015831075608730316, 0.024024121463298798, 0.01835506223142147, 0.039564475417137146, -0.004829932004213333, -0.008424299769103527, 0.00785607285797596, -0.0031615900807082653, -0.02994425594806671, 0.018500423058867455, -0.018764715641736984, 0.00019966416584793478, 0.024830210953950882, -0.0059201354160904884, -0.01450961735099554, -0.020614756271243095, 0.008741449564695358, 0.0012396934907883406, 0.00745963491499424, -0.004582158289849758, -0.04244525730609894, -0.012798327952623367, -0.023218030110001564, -0.010670779272913933, 0.01257367990911007, 0.01641251891851425, -0.014443544670939445, 0.0061943382024765015, -0.023257674649357796, -0.028702083975076675, -0.006554435472935438, -0.0042484900914132595, -0.0007796607096679509, -0.005758256651461124, 0.019293298944830894, -0.022914094850420952, -0.02851708047091961, 0.03766157478094101, -0.006554435472935438, 0.0002335265453439206, 0.017588617280125618, 0.015791432932019234, 0.0345957912504673, -0.0055104829370975494, 0.008113756775856018, 0.0038586596492677927, -0.011324902065098286, -0.002576844533905387, -0.02644238993525505, 0.020680829882621765, 0.018460778519511223, -0.037873007357120514, -0.009085029363632202, -0.008060898631811142, -0.02225336618721485, -0.018870431929826736, -0.005137171130627394, 0.0004455793823581189, 0.02745991386473179, -0.031239286065101624, -0.013769600540399551, -0.012157420627772808, 0.00023311359109357, -0.007935360074043274, -5.884621259610867e-06, 0.002801492577418685, 0.0002467411395628005, -0.023482322692871094, -0.01690145768225193, -0.005612896289676428, -0.018619354814291, 0.002233265433460474, -0.02706347592175007, -0.003321816911920905, -0.006931051146239042, -0.012712433934211731, 0.03874517232179642, -0.008477157913148403, 0.0021110305096954107, 0.010003442876040936, 0.017918981611728668, -0.026521677151322365, -0.02571558766067028, -0.008721628226339817, 0.020125817507505417, 0.04067450016736984, -0.0031467238441109657, 0.019332941621541977, -0.00912467297166586, 0.03462221845984459, 0.008259117603302002, -0.00028266830486245453, 0.01584429107606411, 0.0076578538864851, 0.0031037763692438602, -0.005315567832440138, 0.03895660489797592, 0.014932484365999699, 0.0211565550416708, -0.0019425443606451154, 0.0017311109695583582, 0.007135877385735512, 0.024843424558639526, 0.0022084880620241165, -0.004228668287396431, -9.100515308091417e-05, -0.03557366877794266, 0.005692183505743742, 0.014218896627426147, -0.007109448313713074, 0.011252221651375294, 0.008622518740594387, 0.002487646182999015, 0.03425221145153046, 0.005391551647335291, 0.010545240715146065, -0.02267623320221901, 0.02231943979859352, -0.015302492305636406, 0.03382934629917145, -0.018170058727264404, -0.005325478967279196, -0.014575690031051636, -0.017311109229922295, -0.02119619958102703, 0.012421712279319763, 0.02161906659603119, 0.007836250588297844, 0.008358227089047432, -0.00020957511151209474, -0.013558167032897472, -0.00010520051000639796, -0.016082152724266052, 0.008371441625058651, 9.972884436137974e-05, 0.015209990553557873, 0.005536912474781275, 0.018196487799286842, -0.04149380698800087, -0.007505885791033506, -0.01712610572576523, 0.0013462360948324203, -0.01857971027493477, -0.018170058727264404, 0.023786257952451706, 0.0035745459608733654, -0.00955414678901434, -0.022914094850420952, -0.010750067420303822, 0.030974993482232094, -0.005814418662339449, 0.002353848423808813, -0.01875150017440319, -0.02194943092763424, -0.005705398507416248, 0.010717030614614487, 0.002978237811475992, -0.01326744630932808, 0.012692611664533615, 0.011794019490480423, 0.01437747199088335, -0.006904622074216604, 0.005127259995788336, 0.018341848626732826, -0.026455605402588844, -0.01613501086831093, -0.00467796390876174, 0.006508184596896172, -0.025966664776206017, 0.012395283207297325, -0.008219473995268345, -0.021605851128697395, 0.02666703797876835, -0.005768167786300182, -0.008543231524527073, -0.008800915442407131, -0.017945410683751106, -0.016068939119577408, 0.0101355891674757, 0.01584429107606411, 0.00632978742942214, -0.01809077151119709, 0.01089542731642723, -0.005470839329063892, -0.029257098212838173, 0.015355351381003857, 0.014258540235459805, -0.013571381568908691, 0.0068319421261549, 0.0018533458933234215, 0.02013903111219406, 0.0037661574315279722, -0.0012809891486540437, 0.0025355489924550056, -0.017733976244926453, -0.024209124967455864, 0.026402747258543968, 0.00548075046390295, -0.005021543242037296, -0.02036367915570736, 0.01242832001298666, 0.0077701774425804615, 0.026521677151322365, -0.018923290073871613, -0.009884512051939964, 0.016690025106072426, -0.009303069673478603, 0.010888820514082909, 0.004321170039474964, -0.03562653064727783, 0.009098243899643421, -0.009382357820868492, -0.018302204087376595, -0.014892840757966042, -0.011457047425210476, 0.012884222902357578, -0.031609293073415756, -0.04247168451547623, -0.0033251207787543535, 0.011014359071850777, 0.012553858570754528, -0.03647226095199585, -0.003501865779981017, 0.015738574787974358, 0.0273277685046196, -0.029759252443909645, 0.012606716714799404, -0.007228379603475332, -0.013412807136774063, -0.007677675690501928, 0.005880491808056831, -0.012699219398200512, -0.008444122038781643, 0.039035893976688385, -0.0013478879118338227, -0.025160575285553932, -0.009316284209489822, -0.0010183491976931691, -0.013201373629271984, -0.022570516914129257, -0.009580575861036777, -0.023521967232227325, 0.008933061733841896, -0.008767878636717796, -0.02849065139889717, -0.008193044923245907, 0.010188447311520576, 0.004208846017718315, -0.004070092923939228, 0.004790287930518389, -0.004780377261340618, -0.004453316330909729, 0.0062934476882219315, -0.003538205986842513, -0.0028543509542942047, -0.024975571781396866, 0.004783680662512779, 0.0001777775032678619, 0.014575690031051636, -0.022861236706376076, 0.00725480867549777, -0.02069404534995556, 0.002936942270025611, -0.0015444549499079585, 0.03364434093236923, 0.021975859999656677, 0.0025124235544353724, -0.003954465501010418, 0.028702083975076675, 0.005563341546803713, 0.012454749085009098, 0.01544785313308239, -0.009858082979917526, -0.0027106422930955887, 0.000394372851587832, -0.04323813319206238, -0.01038005854934454, 0.016346445307135582, 0.036392975598573685, 0.009904333390295506, -0.01277189888060093, -0.01957080513238907, -0.012296173721551895, -0.03821658715605736, -0.00755213713273406, 0.00736052542924881, 0.0006103487685322762, 0.017271466553211212, -0.008510194718837738, -0.007043375167995691, 0.007915537804365158, -0.017271466553211212, -0.0021870143245905638, -0.0005079357069917023, -0.010802925564348698, -0.020416539162397385, 0.01014880370348692, 0.01877792924642563, -0.001067077973857522, -0.0005087616154924035, -0.004076700191944838, 0.0007639683899469674, -0.011675088666379452, -0.002487646182999015, -0.02304624207317829, -0.01034702267497778, -0.002573540899902582, -0.009633434936404228, 0.03575867414474487, 0.013009761460125446, 0.00035803273203782737, 0.004268311895430088, -0.0286492258310318, -0.00962022040039301, -0.0018748196307569742, -0.020654400810599327, -0.0026511766482144594, -0.010062908753752708, 0.016623951494693756, -0.004195631481707096, 0.008675376884639263, 0.008232688531279564, -0.014932484365999699, 0.0153157077729702, 0.014284969307482243, 0.027116334065794945, -0.020773332566022873, -0.01881757378578186, 0.025134146213531494, -0.00631326949223876, -0.01242832001298666, -0.022663017734885216, 0.021711567416787148, -0.00233072298578918, -0.0020135727245360613, -0.014456759206950665, -0.03597010672092438, 0.014747479930520058, 0.00012626126408576965, -0.004995114170014858, -0.006785690784454346, 0.004674660507589579, -0.0038322305772453547, -0.0015279367798939347, 0.011113468557596207, 0.012593502178788185, -0.0002702796191442758, -0.019266869872808456, 0.006729528773576021, 0.013901746831834316, -0.006138176191598177, -0.00716230645775795, -0.0254512969404459, 0.02538522332906723, -0.024090193212032318, -0.021037623286247253, 0.005877187941223383, -0.013042798265814781, -0.0022217025980353355, -0.017522543668746948, 0.012481178157031536, -0.0006355390651151538, 0.004017234779894352, 0.24145695567131042, 0.006864978466182947, -0.0040866113267838955, 0.011364545673131943, -0.023984476923942566, 0.007188735995441675, 0.03734442591667175, 0.00950789637863636, -0.028702083975076675, 0.015672501176595688, -0.01670323871076107, -0.021605851128697395, 0.01712610572576523, 0.0035844568628817797, 0.007340703625231981, -0.011556156910955906, -0.029653534293174744, -0.0013875317526981235, -0.002456261543557048, -0.025808090344071388, 0.01828899048268795, -0.011853485368192196, -0.029098521918058395, -0.015461067669093609, 0.01877792924642563, 0.036260828375816345, 0.001434608711861074, 0.0008007214055396616, 0.010604706592857838, 0.005728523712605238, -0.0016047465614974499, -0.009871297515928745, 0.0009522762265987694, -0.011945987120270729, -0.027644917368888855, 0.0025983182713389397, 0.018738284707069397, -0.003311906009912491, -0.003343290649354458, -0.007049982436001301, 0.01822291687130928, 0.0036868699826300144, 0.011483476497232914, -0.007915537804365158, -0.021235842257738113, 0.05407409369945526, -0.01617465540766716, 0.0019029006361961365, -0.004714304115623236, 0.02056189812719822, -0.004892701283097267, -0.002264650072902441, 0.01961044780910015, 0.01904222182929516, -0.024962356314063072, 0.0018219612538814545, 0.011615622788667679, 0.007829642854630947, -0.007466242182999849, 0.01398103404790163, 0.00755213713273406, 0.02009938843548298, -0.02432805672287941, 0.01046595349907875, 0.00507440185174346, 0.012481178157031536, -0.011311687529087067, -0.004631713032722473, -0.0025933629367500544, -0.04506174474954605, -0.0015923578757792711, -0.008549838326871395, -0.021737996488809586, 0.00784285832196474, -0.026812398806214333, -0.014853197149932384, 0.004948863293975592, 0.03102785162627697, 0.018526852130889893, 0.026852043345570564, -0.0036241007037460804, -0.006055585108697414, 0.006878193002194166, -0.0007602517725899816, -0.0319528728723526, -0.04545818269252777, 0.01411317940801382, 0.0045359074138104916, -0.006713010836392641, -0.02756563015282154, 0.0008073287317529321, -0.0030294442549347878, -0.006765868980437517, 0.003452311037108302, 0.002398447599261999, -0.012071525678038597, -0.015209990553557873, 0.03385577350854874, -0.00861591100692749, -0.005451017525047064, -0.014562475495040417, -0.009177531115710735, 0.0221476498991251, 0.009375750087201595, -0.010043086484074593, 0.013505308888852596, 0.0018450868083164096, 0.01795862428843975, 0.0042848302982747555, -0.028464222326874733, -0.0039610727690160275, -0.032957181334495544, -0.0020251355599611998, 0.0030459624249488115, 0.008153400383889675, -0.0034853476099669933, -0.0031731529161334038, -0.021381203085184097, 0.006422289647161961, -0.005599681753665209, -0.03340647742152214, -0.001721200067549944, 0.002978237811475992, 0.018923290073871613, -0.003947858233004808, -0.003125250106677413, -0.019993672147393227, -0.002821314614266157, 0.005527001339942217, -0.046594638377428055, 0.01637287437915802, -0.015989651903510094, 0.003059177193790674, 0.00932289194315672, -0.012606716714799404, 0.01842113584280014, -0.010043086484074593, 0.001954107079654932, 0.0023241157177835703, 0.017324324697256088, 0.012375461868941784, -0.022068360820412636, 0.01736396923661232, 0.00045342554221861064, 0.007849465124309063, -0.03541509434580803, 0.012983332388103008, 0.02185692824423313, 0.004020538181066513, -0.029600676149129868, -0.008252509869635105, -0.004611891228705645, -0.012976725585758686, 0.01213099155575037, 0.010360237210988998, -0.04413672164082527, -0.04458601772785187, -0.027644917368888855, -0.011258828453719616, -0.009058600291609764, -0.007466242182999849, 0.012487785890698433, 0.018632568418979645, 0.010413095355033875, -0.0025173788890242577, -0.00808732770383358, -0.1692524403333664, 0.02666703797876835, 0.023429464548826218, -0.020376894623041153, 0.005873884540051222, -0.0025751928333193064, 0.033195044845342636, -0.01036684401333332, -0.01672966778278351, 0.024750923737883568, 0.019319728016853333, 0.006128265056759119, -0.02013903111219406, 0.02284802310168743, -0.007512493059039116, 0.013009761460125446, -0.014588904567062855, 0.03290432319045067, 0.019399015232920647, 0.0063694315031170845, 0.03182072937488556, 0.008536623790860176, 0.012798327952623367, 0.002411662368103862, 0.01339959166944027, 0.01505141519010067, 0.003465525573119521, 0.0018913379171863198, 0.0018748196307569742, -0.005150385666638613, 0.008734842762351036, -8.58432031236589e-05, -0.006950873415917158, -0.022623375058174133, 0.0300764013081789, 0.008820737712085247, -0.00755874440073967, -0.002223354298621416, 0.001600616960786283, 0.015897149220108986, 0.02970639429986477, 0.02089226432144642, 0.007789999712258577, 0.008093935437500477, -0.011424011550843716, 0.026230957359075546, 0.014086750335991383, -0.002994755981490016, 0.01379602961242199, -0.02013903111219406, 0.01437747199088335, -0.023099100217223167, -0.0015428031329065561, 0.00021948604262433946, 0.014879626221954823, -0.005137171130627394, 0.015817862004041672, 0.025279507040977478, -0.007611602544784546, -0.014549260959029198, 0.009435215964913368, -0.02227979525923729, 0.029098521918058395, 0.010307378135621548, 0.008503586985170841, -0.013056012801826, -0.014813552610576153, -0.015804646536707878, -0.03205858916044235, 0.021182984113693237, -0.02023153379559517, -0.008100542239844799, 0.01297011785209179, -0.0009754017810337245, 0.0031235981732606888, 0.004443405196070671, -0.004707696847617626, -0.027777064591646194, 0.0005570774665102363, 0.03594367951154709, -0.017086463049054146, 0.013320304453372955, -0.008021255023777485, 0.004902611952275038, -0.010703816078603268, 0.025279507040977478, -0.013505308888852596, 0.015461067669093609, -0.00882734451442957, -0.013379770331084728, 0.023482322692871094, -0.018936503678560257, -0.029600676149129868, -0.012421712279319763, 0.026164883747696877, 0.009910941123962402, 0.004843146540224552, 0.0033019951079040766, 0.0022117916960269213, -0.022517656907439232, 0.007796606980264187, -0.026759540662169456, 0.01180062722414732, -0.020218320190906525, 0.032296452671289444, 0.003075695363804698, 0.011298472993075848, 0.00964004173874855, 0.033353619277477264, -0.010300771333277225, -0.001560147269628942, -0.004195631481707096, 0.004839842673391104, 0.007268023211508989, -0.016544664278626442, 0.0378994382917881, -0.006105139385908842, -0.007083019241690636, 0.0019227225566282868, -0.005424588453024626, 0.058778487145900726, 0.014443544670939445, -0.027222050353884697, 0.001443693763576448, 0.0004645753651857376, 0.0009390616323798895, -0.12326567620038986, -0.004929041489958763, 0.0027453305665403605, -0.0024446987081319094, 0.01943865790963173, 0.00436411751434207, -0.0273277685046196, -0.014496402814984322, -0.009283248335123062, 0.009692899882793427, -0.009362535551190376, -0.013511915691196918, 0.0024678243789821863, -0.019782237708568573, -0.0015386735321953893, -0.010545240715146065, 0.01891007460653782, -0.02412983775138855, -0.01692788675427437, 0.040330920368433, -0.006898014806210995, -0.005543519742786884, -0.010902035050094128, -0.012633145786821842, 0.004862968344241381, -0.01894971914589405, -0.03456936031579971, 0.0010034827282652259, 0.006818727124482393, 0.005305657163262367, -0.0012760336976498365, -0.025557013228535652, 0.0017740584444254637, -0.013102264143526554, 0.0159764364361763, -0.00862912554293871, -0.01150329876691103, 0.011919558048248291, 0.012395283207297325, -0.01868542656302452, -0.0021407632157206535, 0.01168830320239067, 0.01954437606036663, -0.05092902109026909, -0.003052569692954421, -0.00014876735804136842, -0.020614756271243095, 0.017773620784282684, 0.008992526680231094, -0.014760694466531277, -0.017733976244926453, 0.016280371695756912, -0.025160575285553932, -0.013928175903856754, -0.007717319298535585, 0.0004459923366084695, 0.01024791318923235, 0.009527717716991901, 0.008444122038781643, 0.021909786388278008, -0.01577821746468544, -0.0107963178306818, -0.005705398507416248, -0.019055435433983803, 0.015394994989037514, -0.007968396879732609, -0.004935648757964373, -0.0021143341436982155, -0.014060321263968945, -0.05248834192752838, -0.030393552035093307, -0.004734125919640064, -0.028702083975076675, 0.0042484900914132595, 0.0007197820814326406, -0.000820130342617631, -0.013967819511890411, 0.0022745609749108553, 0.011714732274413109, -0.02597988024353981, -0.010175232775509357, -0.004843146540224552, 0.009732544422149658, -0.028147071599960327, 0.015791432932019234, 0.0016509975539520383, 0.0004214214568492025, 0.009646649472415447, -0.0008011343888938427, 0.00010509727144381031, -0.03205858916044235, 0.028332076966762543, -0.00036567242932505906, -0.013743171468377113, -0.007102841045707464, 0.007426598574966192, -0.001273555913940072, -0.0031103836372494698, 0.009091636165976524, -0.0016782527090981603, -0.0084837656468153, 0.0016485198866575956, -0.03895660489797592, 0.021909786388278008, -0.009712722152471542, -0.01256046537309885, 0.0153157077729702, -0.004810110200196505, 0.004344295710325241, -0.025504155084490776, -0.03523009270429611, 0.006055585108697414, -0.009877904318273067, 0.02743348479270935, 0.021235842257738113, -0.016848599538207054, -0.0143246129155159, -0.021473705768585205, 0.005804507527500391, 0.025755232200026512, 0.007003731559962034, 0.02238551154732704, 0.0013412806438282132, -0.009712722152471542, 0.026323458179831505, 0.009930762462317944, -0.010320592671632767, 0.0041758096776902676, -0.022398727014660835, 0.015751788392663002, -0.014020677655935287, -0.0041758096776902676, 0.0028411364182829857, -0.02397126331925392, 0.009085029363632202, 0.020852619782090187, 0.019399015232920647, 0.008008040487766266, 0.004093218594789505, 0.026112025603652, 0.016597522422671318, 0.06099853664636612, -0.009065207093954086, -0.01683538407087326, -0.005424588453024626, -0.03721227869391441, 0.023033026605844498, -0.010915249586105347, -0.02954781800508499, 0.019055435433983803, 0.025557013228535652, 0.004823324736207724, 0.02142084762454033, 0.011397582478821278, -0.006898014806210995, -0.007122662849724293, -0.0038751778192818165, -0.037238709628582, 0.0024446987081319094, 0.008120364509522915, -0.01150329876691103, -0.015024986118078232, 0.03179429844021797, -0.010948286391794682, 0.001663386239670217, -0.02373339980840683, 0.015461067669093609, -0.016399303451180458, 0.01044613216072321, 0.012818150222301483, -0.006871585734188557, -0.033750057220458984, -0.010571670718491077, -0.002182058757171035, 0.011542942374944687, 0.03208502009510994, 0.01110686082392931, 0.0022084880620241165, -0.001183531479910016, 0.03803158551454544, -0.005236280616372824, 0.02944210171699524, -0.009137887507677078, 0.01119275577366352, -0.0027189014945179224, 0.021777641028165817, 0.023521967232227325, 0.0235748253762722, 0.010320592671632767, -0.0016832081601023674, -0.002233265433460474, 0.003980894573032856, -0.003282173303887248, -0.01047916803508997, 0.004853057209402323, -0.01173455361276865, -0.002753589767962694, 0.02716919220983982, -0.007499278523027897, 0.008245903067290783, 0.01924044080078602, 0.01505141519010067, 0.015950007364153862, -0.01472105085849762, 0.009078421629965305, -0.015091059729456902, -0.022174078971147537, 0.04191667214035988, -0.0032408777624368668, -0.028305647894740105, 0.003954465501010418, 0.021103696897625923, 0.005384944379329681, -0.0009572317358106375, -0.010637743398547173, 0.0107963178306818, -0.015077845193445683, 0.018923290073871613, -0.0033862381242215633, -0.013307089917361736, -0.010789711028337479, 0.034410785883665085, -0.00445661973208189, 0.004189024213701487, 0.009547539986670017, -0.022028718143701553, -0.012732255272567272, 0.0244602020829916, 0.012210278771817684, 0.0038091049063950777, 0.006210856139659882, -0.018989363685250282, -0.009098243899643421, -0.01604251004755497, -0.010056301020085812, -0.008126971311867237, 0.011774198152124882, -0.011457047425210476, -0.0014808597043156624, 0.019887953996658325, -0.036234401166439056, 0.02089226432144642, -0.002771759871393442, -0.006891407538205385, -0.011014359071850777, 0.013089049607515335, 0.01961044780910015, 0.014126394875347614, -0.007968396879732609, -0.012203671969473362, -0.0037496392615139484, 0.015011771582067013, -0.0013957908377051353, 0.029785681515932083, -0.028939947485923767, -0.033591482788324356, -0.012012060731649399, 0.009018956683576107, -0.006191034335643053, 0.015157132409512997, 0.023482322692871094, 0.018315419554710388, 0.008642340078949928, 0.02327089011669159, 0.011001144535839558, -0.021011194214224815, 0.0073539181612432, 0.037978727370500565, -0.017271466553211212, -0.006696492433547974, -0.038507308810949326, -0.004694482311606407, 0.005943261086940765, -0.0559769943356514, -0.024869853630661964, 0.01431139837950468, -0.03377648815512657, -0.009805223904550076, -0.027089904993772507, -0.0004559032677207142, 0.013128693215548992, -0.012534036301076412, 0.003640618873760104, -0.02175121195614338, 0.0084837656468153, 0.004426886793226004, -0.003792586736381054, -0.004998418036848307, -0.0017971839988604188, -0.02838493511080742]}}, rets=[{'node': {'id_': 'bcf8cce5-6f77-4391-8eae-1bd1a3749094', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': 'f3ad7036-414b-4f27-adcd-52244cbb27eb', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '18b69e4c8c3b6423e75e5c31805f7ee1ac2e3d8b1a20ed7fabef56d3845272c7'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'c3d00853-0daa-4514-9f1c-1a8b67b5d58e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '0990d9b6dd8f781bc862b7c85f53eb0613d198313f968d3b4743544aa84ffe58'}}, 'text': '@Teslarati @KlenderJoey Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet, 391, 6778, 2022-09-21 15:59:15, Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich I was instrumental in creating OpenAI, which made DALL-E, 153, 2442, 2022-09-21 15:17:17, I was instrumental in creating OpenAI, which made DALL-E\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist), 81, 1416, 2022-09-21 15:09:59, Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist)\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America., 64, 1285, 2022-09-21 15:00:26, Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America.\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich For sure, 58, 1171, 2022-09-21 14:55:11, For sure\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(, 154, 2286, 2022-09-21 14:36:49, Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich This is true, 60, 1356, 2022-09-21 14:26:17, This is true\\n@Teslarati @ResidentSponge With some improvements, it will achieve 300 Mps, 651, 14338, 2022-09-21 14:02:11, With some improvements, it will achieve 300 Mps\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.\\n\\nAlso, I was there for 2.5 years, previously attending govt schools for 9 of 12 years., 89, 1694, 2022-09-21 13:55:04, Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.Also, I was there for 2.5 years, previously attending govt schools for 9 of 12 years.', 'start_char_idx': 82792, 'end_char_idx': 85782, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7469842252500036}, {'node': {'id_': 'de6f0f78-221e-4fe4-b85d-6a992f6a6f6d', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': '3e563428-76cc-4aa0-83ee-3e8049217a83', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '9d14ec9cda85a9c8cc3c34b1cdf75fb105795e73d699b35cfc8234703dc37635'}, <NodeRelationship.NEXT: '3'>: {'node_id': '473c5d14-ac7c-45f4-a7eb-fb6ecc383b07', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'fb859d528497fbefaebc580603c2b0b92cca496d280392da3942cd40754a5cfc'}}, 'text': 'Walter Isaacson is writing a book., 215, 4004, 2022-04-21 08:49:51, I should, but that would take a lot of time away from Tesla, SpaceX &amp; other work. Walter Isaacson is writing a book.\\n@PPathole @teslaownersSV @woonomic @VaibhavSisinty Exactly, 137, 3587, 2022-04-21 08:42:11, Exactly\\n@billhuang688 @woonomic @VaibhavSisinty I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. \\n\\nIf it was really his baby, he would never have stood by while it starved to death.\\n\\nhttps://t.co/ZDfYLD5qkL, 355, 4152, 2022-04-21 08:40:07, I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. If it was really his baby, he would never have stood by while it starved to death.\\n@teslaownersSV @woonomic @VaibhavSisinty But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.\\n\\nWe would’ve lost at least a few talented people if he was the real deal, but we lost no one., 395, 5547, 2022-04-21 08:33:10, But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.We would’ve lost at least a few talented people if he was the real deal, but we lost no one.\\n@teslaownersSV @woonomic @VaibhavSisinty Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years., 452, 6083, 2022-04-21 08:27:57, Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years.\\n@woonomic @VaibhavSisinty They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!, 1071, 10676, 2022-04-21 08:21:40, They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!\\n@woonomic @VaibhavSisinty When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all., 425, 8706, 2022-04-21 08:15:37, When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all.\\n@woonomic @VaibhavSisinty If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for https://t.co/bOUOek5Cvy (later renamed PayPal), but that’s not what founding means., 506, 8631, 2022-04-21 08:12:03, If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for  (later renamed PayPal), but that’s not what founding means.\\n@woonomic @VaibhavSisinty Not even close to that. It was a shell corp with no employees, no IP, no designs, no prototype, literally nothing but a biz plan to commercialize AC Propulsion’s Tzero car, which was introduced to me by JB Straubel, *not* Eberhard. \\n\\nEven name “Tesla Motors” was owned by others!, 1324, 17242, 2022-04-21 08:09:07, Not even close to that.', 'start_char_idx': 380380, 'end_char_idx': 383649, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7451874390981541}], error=None, perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 12, 0, 127713), end_time=datetime.datetime(2024, 2, 29, 16, 12, 1, 263548)), pid=32024, tid=20596), RecordAppCall(stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='retrieve'))], args={'query_bundle': {'query_str': 'What did the author do growing up?', 'image_path': None, 'custom_embedding_strs': None, 'embedding': [0.012144206091761589, -0.015698930248618126, 0.007664461154490709, -0.010062908753752708, -0.021275486797094345, 0.02139441855251789, -0.006689885165542364, -0.01778683438897133, -0.027882780879735947, -0.03047283925116062, 0.024301627650856972, 0.0009704462718218565, 0.0025801481679081917, 0.01097471546381712, 0.003690173616632819, 0.01857971027493477, 0.03898303583264351, -0.009613612666726112, -0.0009795313235372305, -0.019055435433983803, -0.0023604556918144226, 0.010723638348281384, 0.0035844568628817797, -0.008569660596549511, -0.003151679178699851, 0.006812119856476784, 0.025041643530130386, -0.028543509542942047, 0.023363390937447548, -0.0033779789227992296, 0.011450440622866154, -0.013901746831834316, -0.008450728841125965, -0.005563341546803713, -0.024883069097995758, -0.009395572356879711, -0.016082152724266052, -0.014469973742961884, 0.0017724066274240613, 0.00755874440073967, 0.016095368191599846, -0.0030955171678215265, -0.013888532295823097, -0.0018087467178702354, 0.02201550267636776, 0.007327489089220762, -0.010360237210988998, -0.0012966814683750272, -0.016068939119577408, -0.0013982686214148998, 0.03425221145153046, 0.013406199403107166, -0.004350902978330851, -0.024103408679366112, -0.019491517916321754, 0.0012314344057813287, 0.0026974277570843697, 0.014099964872002602, 0.0016691676573827863, -0.021275486797094345, -0.007023553363978863, 0.022174078971147537, -0.00299640791490674, 0.007413383573293686, -0.024750923737883568, 0.0007007861277088523, -0.013194765895605087, -0.005199940409511328, -0.008424299769103527, -0.0153157077729702, 0.035996537655591965, 0.00011222076136618853, -0.009904333390295506, -0.0027948853094130754, 0.019002577289938927, -0.010941678658127785, -0.00964004173874855, -0.007367132697254419, -0.009957191534340382, 0.010611314326524734, -0.00785607285797596, -0.0219230018556118, -0.0064322007820010185, 0.009560754522681236, 0.01099453680217266, -0.005642629228532314, 0.016808954998850822, 0.029627105221152306, -0.020257962867617607, -0.01844756491482258, -0.0026214439421892166, 0.017601830884814262, 0.01657109335064888, 0.015130703337490559, 0.014773909002542496, 0.0032887805718928576, -0.01564607210457325, 0.01129186525940895, -0.008840559050440788, -0.003425881965085864, -0.005804507527500391, 0.0041592917405068874, -0.007195343263447285, -0.012401890940964222, -0.01877792924642563, 0.013036190532147884, -0.0062967510893940926, -0.021909786388278008, 0.028305647894740105, -0.03242859989404678, -0.014086750335991383, 0.0202711783349514, 0.02759205922484398, -0.05277906358242035, -0.008543231524527073, -0.025226648896932602, 0.0007338225841522217, 0.000338417332386598, 0.008952883072197437, 0.0009630130953155458, 0.007968396879732609, 0.001654301187954843, 0.021962644532322884, -0.010399880819022655, 0.00944182276725769, -0.0012454749085009098, -0.015897149220108986, -0.00881412997841835, -0.011126683093607426, -0.007102841045707464, 0.017469685524702072, -0.0016848599771037698, 0.012481178157031536, -0.013478879816830158, -0.017667904496192932, 0.016610736027359962, -0.019029006361961365, 0.015342136844992638, -0.021473705768585205, -0.02525307796895504, 0.020482610911130905, 0.022914094850420952, -0.008179829455912113, 0.002928683068603277, 0.006983909755945206, 0.018473993986845016, 0.030552126467227936, 0.039062321186065674, 0.01008273009210825, -0.018698642030358315, 0.017469685524702072, -0.0025884073693305254, 0.007089626509696245, -0.008457336574792862, 0.01437747199088335, 0.035071514546871185, -0.015381780453026295, 0.012540644034743309, -0.016002865508198738, -0.01994081400334835, 0.0209319069981575, 0.025689158588647842, 0.0221476498991251, -0.008212866261601448, -0.01170151773840189, 0.03816372901201248, 0.023416249081492424, 0.014958913438022137, 0.0028097517788410187, -0.012679397128522396, -0.024050550535321236, 0.012593502178788185, -0.03351219370961189, 0.009435215964913368, 0.007281237747520208, 0.024235554039478302, -0.0026924721896648407, 0.015540354885160923, -0.01706003211438656, -0.003908214159309864, -0.012666182592511177, 0.001992099219933152, 0.02501521445810795, 0.04424244165420532, -0.008622518740594387, -0.00881412997841835, 0.00861591100692749, 0.008080720901489258, -0.004327777307480574, 0.0031896710861474276, -0.012124384753406048, 0.024473417550325394, -0.012501000426709652, -0.02449984662234783, -0.66982102394104, -0.012752077542245388, -0.0032127967569977045, -0.01967652142047882, 0.01149008423089981, 0.022755520418286324, -0.002528941724449396, 0.0365779809653759, -0.0306314155459404, 0.004301348235458136, -0.007320881821215153, 0.03858659788966179, 0.026719896122813225, 0.0004080004000570625, 0.011866699904203415, -0.011476869694888592, 0.010109160095453262, -0.02218729257583618, 0.00853001605719328, -0.015342136844992638, 0.00508431252092123, 0.02073368802666664, -0.007624817080795765, -0.0062571074813604355, 0.008576267398893833, 0.010743459686636925, 0.008992526680231094, -0.03311575576663017, 0.013835673220455647, 0.02571558766067028, -0.017839694395661354, 0.028807802125811577, 0.03647226095199585, 0.025794874876737595, 0.05513126030564308, -0.012322602793574333, -0.008378048427402973, 0.03536223620176315, 0.005566644947975874, 0.04387243092060089, -0.02062797173857689, -0.016676809638738632, 0.015619643032550812, 0.014205682091414928, 0.007710712030529976, -0.01119275577366352, 0.039564475417137146, 0.0032887805718928576, 0.035071514546871185, 0.019266869872808456, -0.005854062270373106, -0.001008438179269433, -0.001815353985875845, -0.007922145538032055, -0.0007602517725899816, -0.006504880730062723, 0.0012182198697701097, 0.015421424061059952, 0.007294452283531427, 0.017800049856305122, -0.006389253307133913, 0.03715942054986954, -0.023033026605844498, 0.004704393446445465, -0.0033829344902187586, 0.02241194061934948, -0.016544664278626442, 0.026204528287053108, -0.005606289021670818, -0.022636588662862778, 0.009778794832527637, 0.007585173472762108, -0.015289277769625187, -0.009474859572947025, 0.01335334125906229, 0.010538633912801743, 0.03274574875831604, 0.005295746028423309, -0.002448002342134714, -0.006716314237564802, -0.027023831382393837, -0.0037000845186412334, -0.04067450016736984, -0.010849176906049252, 0.02205514721572399, 0.006448718719184399, -0.03023497760295868, 0.007922145538032055, -0.003838837845250964, 0.007677675690501928, -0.01026773452758789, 0.03277217969298363, -0.007690890226513147, -0.026019522920250893, -0.012540644034743309, 0.009811831638216972, -0.013445843011140823, -0.001188487047329545, 0.0007317577837966383, 0.002581800101324916, 0.010611314326524734, -0.002264650072902441, 0.010221484117209911, 0.011338116601109505, 0.00550387566909194, 0.000641320482827723, 0.012434926815330982, 0.025530584156513214, 0.03927375376224518, -0.018923290073871613, 0.005708701908588409, 0.0009473207755945623, 0.0013354993425309658, -0.002353848423808813, -0.007869287393987179, -0.023151958361268044, 0.014033892191946507, 0.008292153477668762, 0.011001144535839558, -0.013459057547152042, -0.0014023981057107449, 0.010413095355033875, -0.00911145843565464, -0.018209701403975487, -0.005014935974031687, 0.023455893620848656, -0.009197353385388851, 0.010703816078603268, -0.011357937939465046, 0.0007561221718788147, -0.0007342355675064027, -0.010023265145719051, 0.013300483115017414, -0.01910829357802868, 0.013769600540399551, 0.0031285537406802177, -0.003805801272392273, -0.0061943382024765015, 0.00477707339450717, -0.00725480867549777, -0.020509039983153343, 0.004998418036848307, -0.005209851078689098, -0.003214448457583785, -0.017152534797787666, -0.02046939730644226, 0.01117293443530798, -0.002784974407404661, -0.012765292078256607, 0.02152656391263008, 0.003366416320204735, -0.013518523424863815, -0.022517656907439232, -0.015883933752775192, -0.01828899048268795, -0.016201084479689598, 0.01265957485884428, -0.005351908039301634, -0.01431139837950468, 0.009085029363632202, -0.0023109009489417076, 0.03628725931048393, -0.010928464122116566, 0.010155410505831242, -0.014218896627426147, 0.01036684401333332, 0.013181551359593868, 0.012296173721551895, 0.010868998244404793, -0.008701805956661701, 0.011272042989730835, -0.016148226335644722, 0.0034490074031054974, -0.005170207470655441, -0.007446420378983021, 0.012652968056499958, 0.0008837255882099271, -0.002722205128520727, -0.008477157913148403, 0.004420279525220394, -0.004479745402932167, -0.007902323268353939, -0.007882501929998398, -0.011575979180634022, 0.023706970736384392, -0.009382357820868492, 0.0012628190452232957, 0.03245502710342407, 0.016201084479689598, 0.03227002173662186, 7.371262472588569e-05, 0.014536046423017979, 0.006762565113604069, 0.009058600291609764, -0.0006429722998291254, -0.009461645036935806, 0.030049972236156464, 0.007049982436001301, 0.009930762462317944, 0.00785607285797596, -0.026191312819719315, 0.026191312819719315, 0.01399424858391285, 0.0019755808170884848, 0.017245037481188774, -0.037317994982004166, -0.004004020243883133, -0.02023153379559517, -0.008219473995268345, 0.0025223344564437866, 0.003528294852003455, -0.008153400383889675, 0.006025852169841528, -0.018989363685250282, -0.01437747199088335, 0.006141479592770338, -0.018077556043863297, 0.017152534797787666, -0.01047916803508997, 0.001551062217913568, 0.017932195216417313, -0.011873307637870312, -0.004704393446445465, -0.001744325621984899, -0.015223205089569092, 0.019861524924635887, 0.001973929116502404, -0.019385799765586853, 0.006937658414244652, -0.022068360820412636, -0.004836539272218943, -0.004796895198523998, 0.01090864185243845, 0.01802469789981842, -0.005242887884378433, 0.021143341436982155, 0.02218729257583618, -0.01544785313308239, 0.01835506223142147, -0.011675088666379452, -0.00881412997841835, 0.016861815005540848, 0.024737708270549774, -0.005655843764543533, 0.030552126467227936, -0.011959201656281948, 0.017773620784282684, -0.021169770509004593, -0.011893128976225853, -0.0057483455166220665, 0.003977591171860695, -0.012084740214049816, 0.0026743023190647364, -0.023680541664361954, 0.006250500213354826, 0.007895716466009617, 0.005355211906135082, -0.004192328080534935, 0.00868198461830616, 0.02957424707710743, -0.0042352755554020405, 0.009626827202737331, -0.003957768902182579, 0.01706003211438656, 0.0014511268818750978, 0.013703527860343456, -0.009659864008426666, -0.005081009119749069, -0.03219073638319969, -0.0105782775208354, -0.022755520418286324, -0.010644350200891495, 0.006716314237564802, -0.02126227132976055, 0.010743459686636925, 0.007307667285203934, -0.00924360379576683, -0.004988506902009249, 0.00911145843565464, -0.0010546892881393433, -0.02261015959084034, -0.0025702372658997774, 0.022821594029664993, 0.01716575026512146, 0.0043608141131699085, -0.007003731559962034, -0.007076411973685026, 0.0016361312009394169, -0.014033892191946507, -0.0004860490735154599, 0.005262709688395262, -0.002244828036054969, 0.0026528285816311836, -0.010849176906049252, 0.0041592917405068874, 0.016465377062559128, 0.03020854853093624, -0.023588038980960846, 0.023521967232227325, -0.02099798060953617, -0.03345933556556702, -0.009177531115710735, -0.022359082475304604, 0.0017509328899905086, 0.02838493511080742, 0.005150385666638613, -0.014641763642430305, -0.013095656409859657, -0.007003731559962034, -0.007023553363978863, -0.012302781455218792, 0.0009704462718218565, 0.0061447834596037865, 0.01338637713342905, 0.010360237210988998, -0.011278650723397732, -0.014694621786475182, -0.006039066705852747, 0.015936793759465218, 0.0032028856221586466, -0.026415960863232613, -0.021275486797094345, -0.029124950990080833, 0.012104562483727932, 0.1068795844912529, 0.014403901062905788, -0.01564607210457325, 0.015077845193445683, 0.009785402566194534, -0.011424011550843716, -0.016161441802978516, -0.021804070100188255, 0.01004969421774149, -0.0018682123627513647, -0.00121904572006315, -0.011873307637870312, 0.01411317940801382, -0.013716742396354675, -0.007076411973685026, -0.0016295238165184855, -3.1307215976994485e-05, -0.025755232200026512, -0.01523641962558031, -0.021315129473805428, 0.003528294852003455, 0.026230957359075546, 0.010095945559442043, 0.022636588662862778, 0.007809821516275406, -0.006267018150538206, 0.004080004058778286, 0.0010505596874281764, 0.0286492258310318, -0.01076328195631504, 0.00599281582981348, 0.0006190208368934691, 0.009461645036935806, -0.015513925813138485, 0.006415682379156351, 0.04109736904501915, 0.014086750335991383, 0.0287549439817667, 0.02525307796895504, -0.0040106275118887424, 0.031635724008083344, -0.009428608231246471, 0.014205682091414928, -0.0352565199136734, 0.014060321263968945, -0.026363102719187737, 0.0026974277570843697, 0.028173500671982765, 0.0040866113267838955, 0.0007334096007980406, 0.0024314841721206903, -0.0022894274443387985, -0.0057483455166220665, -0.007684282958507538, 0.014998557046055794, -0.00549726840108633, 0.013848887756466866, -0.0032161003910005093, -0.0018748196307569742, 0.02627060003578663, -0.010849176906049252, -0.031609293073415756, 0.009283248335123062, -0.026257386431097984, 0.007301060017198324, -0.02719562128186226, -0.013207980431616306, 0.0042716157622635365, -0.009501288644969463, -0.002657783916220069, -0.009236996993422508, -0.00550387566909194, -0.038110870867967606, -0.015831075608730316, 0.024024121463298798, 0.01835506223142147, 0.039564475417137146, -0.004829932004213333, -0.008424299769103527, 0.00785607285797596, -0.0031615900807082653, -0.02994425594806671, 0.018500423058867455, -0.018764715641736984, 0.00019966416584793478, 0.024830210953950882, -0.0059201354160904884, -0.01450961735099554, -0.020614756271243095, 0.008741449564695358, 0.0012396934907883406, 0.00745963491499424, -0.004582158289849758, -0.04244525730609894, -0.012798327952623367, -0.023218030110001564, -0.010670779272913933, 0.01257367990911007, 0.01641251891851425, -0.014443544670939445, 0.0061943382024765015, -0.023257674649357796, -0.028702083975076675, -0.006554435472935438, -0.0042484900914132595, -0.0007796607096679509, -0.005758256651461124, 0.019293298944830894, -0.022914094850420952, -0.02851708047091961, 0.03766157478094101, -0.006554435472935438, 0.0002335265453439206, 0.017588617280125618, 0.015791432932019234, 0.0345957912504673, -0.0055104829370975494, 0.008113756775856018, 0.0038586596492677927, -0.011324902065098286, -0.002576844533905387, -0.02644238993525505, 0.020680829882621765, 0.018460778519511223, -0.037873007357120514, -0.009085029363632202, -0.008060898631811142, -0.02225336618721485, -0.018870431929826736, -0.005137171130627394, 0.0004455793823581189, 0.02745991386473179, -0.031239286065101624, -0.013769600540399551, -0.012157420627772808, 0.00023311359109357, -0.007935360074043274, -5.884621259610867e-06, 0.002801492577418685, 0.0002467411395628005, -0.023482322692871094, -0.01690145768225193, -0.005612896289676428, -0.018619354814291, 0.002233265433460474, -0.02706347592175007, -0.003321816911920905, -0.006931051146239042, -0.012712433934211731, 0.03874517232179642, -0.008477157913148403, 0.0021110305096954107, 0.010003442876040936, 0.017918981611728668, -0.026521677151322365, -0.02571558766067028, -0.008721628226339817, 0.020125817507505417, 0.04067450016736984, -0.0031467238441109657, 0.019332941621541977, -0.00912467297166586, 0.03462221845984459, 0.008259117603302002, -0.00028266830486245453, 0.01584429107606411, 0.0076578538864851, 0.0031037763692438602, -0.005315567832440138, 0.03895660489797592, 0.014932484365999699, 0.0211565550416708, -0.0019425443606451154, 0.0017311109695583582, 0.007135877385735512, 0.024843424558639526, 0.0022084880620241165, -0.004228668287396431, -9.100515308091417e-05, -0.03557366877794266, 0.005692183505743742, 0.014218896627426147, -0.007109448313713074, 0.011252221651375294, 0.008622518740594387, 0.002487646182999015, 0.03425221145153046, 0.005391551647335291, 0.010545240715146065, -0.02267623320221901, 0.02231943979859352, -0.015302492305636406, 0.03382934629917145, -0.018170058727264404, -0.005325478967279196, -0.014575690031051636, -0.017311109229922295, -0.02119619958102703, 0.012421712279319763, 0.02161906659603119, 0.007836250588297844, 0.008358227089047432, -0.00020957511151209474, -0.013558167032897472, -0.00010520051000639796, -0.016082152724266052, 0.008371441625058651, 9.972884436137974e-05, 0.015209990553557873, 0.005536912474781275, 0.018196487799286842, -0.04149380698800087, -0.007505885791033506, -0.01712610572576523, 0.0013462360948324203, -0.01857971027493477, -0.018170058727264404, 0.023786257952451706, 0.0035745459608733654, -0.00955414678901434, -0.022914094850420952, -0.010750067420303822, 0.030974993482232094, -0.005814418662339449, 0.002353848423808813, -0.01875150017440319, -0.02194943092763424, -0.005705398507416248, 0.010717030614614487, 0.002978237811475992, -0.01326744630932808, 0.012692611664533615, 0.011794019490480423, 0.01437747199088335, -0.006904622074216604, 0.005127259995788336, 0.018341848626732826, -0.026455605402588844, -0.01613501086831093, -0.00467796390876174, 0.006508184596896172, -0.025966664776206017, 0.012395283207297325, -0.008219473995268345, -0.021605851128697395, 0.02666703797876835, -0.005768167786300182, -0.008543231524527073, -0.008800915442407131, -0.017945410683751106, -0.016068939119577408, 0.0101355891674757, 0.01584429107606411, 0.00632978742942214, -0.01809077151119709, 0.01089542731642723, -0.005470839329063892, -0.029257098212838173, 0.015355351381003857, 0.014258540235459805, -0.013571381568908691, 0.0068319421261549, 0.0018533458933234215, 0.02013903111219406, 0.0037661574315279722, -0.0012809891486540437, 0.0025355489924550056, -0.017733976244926453, -0.024209124967455864, 0.026402747258543968, 0.00548075046390295, -0.005021543242037296, -0.02036367915570736, 0.01242832001298666, 0.0077701774425804615, 0.026521677151322365, -0.018923290073871613, -0.009884512051939964, 0.016690025106072426, -0.009303069673478603, 0.010888820514082909, 0.004321170039474964, -0.03562653064727783, 0.009098243899643421, -0.009382357820868492, -0.018302204087376595, -0.014892840757966042, -0.011457047425210476, 0.012884222902357578, -0.031609293073415756, -0.04247168451547623, -0.0033251207787543535, 0.011014359071850777, 0.012553858570754528, -0.03647226095199585, -0.003501865779981017, 0.015738574787974358, 0.0273277685046196, -0.029759252443909645, 0.012606716714799404, -0.007228379603475332, -0.013412807136774063, -0.007677675690501928, 0.005880491808056831, -0.012699219398200512, -0.008444122038781643, 0.039035893976688385, -0.0013478879118338227, -0.025160575285553932, -0.009316284209489822, -0.0010183491976931691, -0.013201373629271984, -0.022570516914129257, -0.009580575861036777, -0.023521967232227325, 0.008933061733841896, -0.008767878636717796, -0.02849065139889717, -0.008193044923245907, 0.010188447311520576, 0.004208846017718315, -0.004070092923939228, 0.004790287930518389, -0.004780377261340618, -0.004453316330909729, 0.0062934476882219315, -0.003538205986842513, -0.0028543509542942047, -0.024975571781396866, 0.004783680662512779, 0.0001777775032678619, 0.014575690031051636, -0.022861236706376076, 0.00725480867549777, -0.02069404534995556, 0.002936942270025611, -0.0015444549499079585, 0.03364434093236923, 0.021975859999656677, 0.0025124235544353724, -0.003954465501010418, 0.028702083975076675, 0.005563341546803713, 0.012454749085009098, 0.01544785313308239, -0.009858082979917526, -0.0027106422930955887, 0.000394372851587832, -0.04323813319206238, -0.01038005854934454, 0.016346445307135582, 0.036392975598573685, 0.009904333390295506, -0.01277189888060093, -0.01957080513238907, -0.012296173721551895, -0.03821658715605736, -0.00755213713273406, 0.00736052542924881, 0.0006103487685322762, 0.017271466553211212, -0.008510194718837738, -0.007043375167995691, 0.007915537804365158, -0.017271466553211212, -0.0021870143245905638, -0.0005079357069917023, -0.010802925564348698, -0.020416539162397385, 0.01014880370348692, 0.01877792924642563, -0.001067077973857522, -0.0005087616154924035, -0.004076700191944838, 0.0007639683899469674, -0.011675088666379452, -0.002487646182999015, -0.02304624207317829, -0.01034702267497778, -0.002573540899902582, -0.009633434936404228, 0.03575867414474487, 0.013009761460125446, 0.00035803273203782737, 0.004268311895430088, -0.0286492258310318, -0.00962022040039301, -0.0018748196307569742, -0.020654400810599327, -0.0026511766482144594, -0.010062908753752708, 0.016623951494693756, -0.004195631481707096, 0.008675376884639263, 0.008232688531279564, -0.014932484365999699, 0.0153157077729702, 0.014284969307482243, 0.027116334065794945, -0.020773332566022873, -0.01881757378578186, 0.025134146213531494, -0.00631326949223876, -0.01242832001298666, -0.022663017734885216, 0.021711567416787148, -0.00233072298578918, -0.0020135727245360613, -0.014456759206950665, -0.03597010672092438, 0.014747479930520058, 0.00012626126408576965, -0.004995114170014858, -0.006785690784454346, 0.004674660507589579, -0.0038322305772453547, -0.0015279367798939347, 0.011113468557596207, 0.012593502178788185, -0.0002702796191442758, -0.019266869872808456, 0.006729528773576021, 0.013901746831834316, -0.006138176191598177, -0.00716230645775795, -0.0254512969404459, 0.02538522332906723, -0.024090193212032318, -0.021037623286247253, 0.005877187941223383, -0.013042798265814781, -0.0022217025980353355, -0.017522543668746948, 0.012481178157031536, -0.0006355390651151538, 0.004017234779894352, 0.24145695567131042, 0.006864978466182947, -0.0040866113267838955, 0.011364545673131943, -0.023984476923942566, 0.007188735995441675, 0.03734442591667175, 0.00950789637863636, -0.028702083975076675, 0.015672501176595688, -0.01670323871076107, -0.021605851128697395, 0.01712610572576523, 0.0035844568628817797, 0.007340703625231981, -0.011556156910955906, -0.029653534293174744, -0.0013875317526981235, -0.002456261543557048, -0.025808090344071388, 0.01828899048268795, -0.011853485368192196, -0.029098521918058395, -0.015461067669093609, 0.01877792924642563, 0.036260828375816345, 0.001434608711861074, 0.0008007214055396616, 0.010604706592857838, 0.005728523712605238, -0.0016047465614974499, -0.009871297515928745, 0.0009522762265987694, -0.011945987120270729, -0.027644917368888855, 0.0025983182713389397, 0.018738284707069397, -0.003311906009912491, -0.003343290649354458, -0.007049982436001301, 0.01822291687130928, 0.0036868699826300144, 0.011483476497232914, -0.007915537804365158, -0.021235842257738113, 0.05407409369945526, -0.01617465540766716, 0.0019029006361961365, -0.004714304115623236, 0.02056189812719822, -0.004892701283097267, -0.002264650072902441, 0.01961044780910015, 0.01904222182929516, -0.024962356314063072, 0.0018219612538814545, 0.011615622788667679, 0.007829642854630947, -0.007466242182999849, 0.01398103404790163, 0.00755213713273406, 0.02009938843548298, -0.02432805672287941, 0.01046595349907875, 0.00507440185174346, 0.012481178157031536, -0.011311687529087067, -0.004631713032722473, -0.0025933629367500544, -0.04506174474954605, -0.0015923578757792711, -0.008549838326871395, -0.021737996488809586, 0.00784285832196474, -0.026812398806214333, -0.014853197149932384, 0.004948863293975592, 0.03102785162627697, 0.018526852130889893, 0.026852043345570564, -0.0036241007037460804, -0.006055585108697414, 0.006878193002194166, -0.0007602517725899816, -0.0319528728723526, -0.04545818269252777, 0.01411317940801382, 0.0045359074138104916, -0.006713010836392641, -0.02756563015282154, 0.0008073287317529321, -0.0030294442549347878, -0.006765868980437517, 0.003452311037108302, 0.002398447599261999, -0.012071525678038597, -0.015209990553557873, 0.03385577350854874, -0.00861591100692749, -0.005451017525047064, -0.014562475495040417, -0.009177531115710735, 0.0221476498991251, 0.009375750087201595, -0.010043086484074593, 0.013505308888852596, 0.0018450868083164096, 0.01795862428843975, 0.0042848302982747555, -0.028464222326874733, -0.0039610727690160275, -0.032957181334495544, -0.0020251355599611998, 0.0030459624249488115, 0.008153400383889675, -0.0034853476099669933, -0.0031731529161334038, -0.021381203085184097, 0.006422289647161961, -0.005599681753665209, -0.03340647742152214, -0.001721200067549944, 0.002978237811475992, 0.018923290073871613, -0.003947858233004808, -0.003125250106677413, -0.019993672147393227, -0.002821314614266157, 0.005527001339942217, -0.046594638377428055, 0.01637287437915802, -0.015989651903510094, 0.003059177193790674, 0.00932289194315672, -0.012606716714799404, 0.01842113584280014, -0.010043086484074593, 0.001954107079654932, 0.0023241157177835703, 0.017324324697256088, 0.012375461868941784, -0.022068360820412636, 0.01736396923661232, 0.00045342554221861064, 0.007849465124309063, -0.03541509434580803, 0.012983332388103008, 0.02185692824423313, 0.004020538181066513, -0.029600676149129868, -0.008252509869635105, -0.004611891228705645, -0.012976725585758686, 0.01213099155575037, 0.010360237210988998, -0.04413672164082527, -0.04458601772785187, -0.027644917368888855, -0.011258828453719616, -0.009058600291609764, -0.007466242182999849, 0.012487785890698433, 0.018632568418979645, 0.010413095355033875, -0.0025173788890242577, -0.00808732770383358, -0.1692524403333664, 0.02666703797876835, 0.023429464548826218, -0.020376894623041153, 0.005873884540051222, -0.0025751928333193064, 0.033195044845342636, -0.01036684401333332, -0.01672966778278351, 0.024750923737883568, 0.019319728016853333, 0.006128265056759119, -0.02013903111219406, 0.02284802310168743, -0.007512493059039116, 0.013009761460125446, -0.014588904567062855, 0.03290432319045067, 0.019399015232920647, 0.0063694315031170845, 0.03182072937488556, 0.008536623790860176, 0.012798327952623367, 0.002411662368103862, 0.01339959166944027, 0.01505141519010067, 0.003465525573119521, 0.0018913379171863198, 0.0018748196307569742, -0.005150385666638613, 0.008734842762351036, -8.58432031236589e-05, -0.006950873415917158, -0.022623375058174133, 0.0300764013081789, 0.008820737712085247, -0.00755874440073967, -0.002223354298621416, 0.001600616960786283, 0.015897149220108986, 0.02970639429986477, 0.02089226432144642, 0.007789999712258577, 0.008093935437500477, -0.011424011550843716, 0.026230957359075546, 0.014086750335991383, -0.002994755981490016, 0.01379602961242199, -0.02013903111219406, 0.01437747199088335, -0.023099100217223167, -0.0015428031329065561, 0.00021948604262433946, 0.014879626221954823, -0.005137171130627394, 0.015817862004041672, 0.025279507040977478, -0.007611602544784546, -0.014549260959029198, 0.009435215964913368, -0.02227979525923729, 0.029098521918058395, 0.010307378135621548, 0.008503586985170841, -0.013056012801826, -0.014813552610576153, -0.015804646536707878, -0.03205858916044235, 0.021182984113693237, -0.02023153379559517, -0.008100542239844799, 0.01297011785209179, -0.0009754017810337245, 0.0031235981732606888, 0.004443405196070671, -0.004707696847617626, -0.027777064591646194, 0.0005570774665102363, 0.03594367951154709, -0.017086463049054146, 0.013320304453372955, -0.008021255023777485, 0.004902611952275038, -0.010703816078603268, 0.025279507040977478, -0.013505308888852596, 0.015461067669093609, -0.00882734451442957, -0.013379770331084728, 0.023482322692871094, -0.018936503678560257, -0.029600676149129868, -0.012421712279319763, 0.026164883747696877, 0.009910941123962402, 0.004843146540224552, 0.0033019951079040766, 0.0022117916960269213, -0.022517656907439232, 0.007796606980264187, -0.026759540662169456, 0.01180062722414732, -0.020218320190906525, 0.032296452671289444, 0.003075695363804698, 0.011298472993075848, 0.00964004173874855, 0.033353619277477264, -0.010300771333277225, -0.001560147269628942, -0.004195631481707096, 0.004839842673391104, 0.007268023211508989, -0.016544664278626442, 0.0378994382917881, -0.006105139385908842, -0.007083019241690636, 0.0019227225566282868, -0.005424588453024626, 0.058778487145900726, 0.014443544670939445, -0.027222050353884697, 0.001443693763576448, 0.0004645753651857376, 0.0009390616323798895, -0.12326567620038986, -0.004929041489958763, 0.0027453305665403605, -0.0024446987081319094, 0.01943865790963173, 0.00436411751434207, -0.0273277685046196, -0.014496402814984322, -0.009283248335123062, 0.009692899882793427, -0.009362535551190376, -0.013511915691196918, 0.0024678243789821863, -0.019782237708568573, -0.0015386735321953893, -0.010545240715146065, 0.01891007460653782, -0.02412983775138855, -0.01692788675427437, 0.040330920368433, -0.006898014806210995, -0.005543519742786884, -0.010902035050094128, -0.012633145786821842, 0.004862968344241381, -0.01894971914589405, -0.03456936031579971, 0.0010034827282652259, 0.006818727124482393, 0.005305657163262367, -0.0012760336976498365, -0.025557013228535652, 0.0017740584444254637, -0.013102264143526554, 0.0159764364361763, -0.00862912554293871, -0.01150329876691103, 0.011919558048248291, 0.012395283207297325, -0.01868542656302452, -0.0021407632157206535, 0.01168830320239067, 0.01954437606036663, -0.05092902109026909, -0.003052569692954421, -0.00014876735804136842, -0.020614756271243095, 0.017773620784282684, 0.008992526680231094, -0.014760694466531277, -0.017733976244926453, 0.016280371695756912, -0.025160575285553932, -0.013928175903856754, -0.007717319298535585, 0.0004459923366084695, 0.01024791318923235, 0.009527717716991901, 0.008444122038781643, 0.021909786388278008, -0.01577821746468544, -0.0107963178306818, -0.005705398507416248, -0.019055435433983803, 0.015394994989037514, -0.007968396879732609, -0.004935648757964373, -0.0021143341436982155, -0.014060321263968945, -0.05248834192752838, -0.030393552035093307, -0.004734125919640064, -0.028702083975076675, 0.0042484900914132595, 0.0007197820814326406, -0.000820130342617631, -0.013967819511890411, 0.0022745609749108553, 0.011714732274413109, -0.02597988024353981, -0.010175232775509357, -0.004843146540224552, 0.009732544422149658, -0.028147071599960327, 0.015791432932019234, 0.0016509975539520383, 0.0004214214568492025, 0.009646649472415447, -0.0008011343888938427, 0.00010509727144381031, -0.03205858916044235, 0.028332076966762543, -0.00036567242932505906, -0.013743171468377113, -0.007102841045707464, 0.007426598574966192, -0.001273555913940072, -0.0031103836372494698, 0.009091636165976524, -0.0016782527090981603, -0.0084837656468153, 0.0016485198866575956, -0.03895660489797592, 0.021909786388278008, -0.009712722152471542, -0.01256046537309885, 0.0153157077729702, -0.004810110200196505, 0.004344295710325241, -0.025504155084490776, -0.03523009270429611, 0.006055585108697414, -0.009877904318273067, 0.02743348479270935, 0.021235842257738113, -0.016848599538207054, -0.0143246129155159, -0.021473705768585205, 0.005804507527500391, 0.025755232200026512, 0.007003731559962034, 0.02238551154732704, 0.0013412806438282132, -0.009712722152471542, 0.026323458179831505, 0.009930762462317944, -0.010320592671632767, 0.0041758096776902676, -0.022398727014660835, 0.015751788392663002, -0.014020677655935287, -0.0041758096776902676, 0.0028411364182829857, -0.02397126331925392, 0.009085029363632202, 0.020852619782090187, 0.019399015232920647, 0.008008040487766266, 0.004093218594789505, 0.026112025603652, 0.016597522422671318, 0.06099853664636612, -0.009065207093954086, -0.01683538407087326, -0.005424588453024626, -0.03721227869391441, 0.023033026605844498, -0.010915249586105347, -0.02954781800508499, 0.019055435433983803, 0.025557013228535652, 0.004823324736207724, 0.02142084762454033, 0.011397582478821278, -0.006898014806210995, -0.007122662849724293, -0.0038751778192818165, -0.037238709628582, 0.0024446987081319094, 0.008120364509522915, -0.01150329876691103, -0.015024986118078232, 0.03179429844021797, -0.010948286391794682, 0.001663386239670217, -0.02373339980840683, 0.015461067669093609, -0.016399303451180458, 0.01044613216072321, 0.012818150222301483, -0.006871585734188557, -0.033750057220458984, -0.010571670718491077, -0.002182058757171035, 0.011542942374944687, 0.03208502009510994, 0.01110686082392931, 0.0022084880620241165, -0.001183531479910016, 0.03803158551454544, -0.005236280616372824, 0.02944210171699524, -0.009137887507677078, 0.01119275577366352, -0.0027189014945179224, 0.021777641028165817, 0.023521967232227325, 0.0235748253762722, 0.010320592671632767, -0.0016832081601023674, -0.002233265433460474, 0.003980894573032856, -0.003282173303887248, -0.01047916803508997, 0.004853057209402323, -0.01173455361276865, -0.002753589767962694, 0.02716919220983982, -0.007499278523027897, 0.008245903067290783, 0.01924044080078602, 0.01505141519010067, 0.015950007364153862, -0.01472105085849762, 0.009078421629965305, -0.015091059729456902, -0.022174078971147537, 0.04191667214035988, -0.0032408777624368668, -0.028305647894740105, 0.003954465501010418, 0.021103696897625923, 0.005384944379329681, -0.0009572317358106375, -0.010637743398547173, 0.0107963178306818, -0.015077845193445683, 0.018923290073871613, -0.0033862381242215633, -0.013307089917361736, -0.010789711028337479, 0.034410785883665085, -0.00445661973208189, 0.004189024213701487, 0.009547539986670017, -0.022028718143701553, -0.012732255272567272, 0.0244602020829916, 0.012210278771817684, 0.0038091049063950777, 0.006210856139659882, -0.018989363685250282, -0.009098243899643421, -0.01604251004755497, -0.010056301020085812, -0.008126971311867237, 0.011774198152124882, -0.011457047425210476, -0.0014808597043156624, 0.019887953996658325, -0.036234401166439056, 0.02089226432144642, -0.002771759871393442, -0.006891407538205385, -0.011014359071850777, 0.013089049607515335, 0.01961044780910015, 0.014126394875347614, -0.007968396879732609, -0.012203671969473362, -0.0037496392615139484, 0.015011771582067013, -0.0013957908377051353, 0.029785681515932083, -0.028939947485923767, -0.033591482788324356, -0.012012060731649399, 0.009018956683576107, -0.006191034335643053, 0.015157132409512997, 0.023482322692871094, 0.018315419554710388, 0.008642340078949928, 0.02327089011669159, 0.011001144535839558, -0.021011194214224815, 0.0073539181612432, 0.037978727370500565, -0.017271466553211212, -0.006696492433547974, -0.038507308810949326, -0.004694482311606407, 0.005943261086940765, -0.0559769943356514, -0.024869853630661964, 0.01431139837950468, -0.03377648815512657, -0.009805223904550076, -0.027089904993772507, -0.0004559032677207142, 0.013128693215548992, -0.012534036301076412, 0.003640618873760104, -0.02175121195614338, 0.0084837656468153, 0.004426886793226004, -0.003792586736381054, -0.004998418036848307, -0.0017971839988604188, -0.02838493511080742]}}, rets=[{'node': {'id_': 'bcf8cce5-6f77-4391-8eae-1bd1a3749094', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': 'f3ad7036-414b-4f27-adcd-52244cbb27eb', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '18b69e4c8c3b6423e75e5c31805f7ee1ac2e3d8b1a20ed7fabef56d3845272c7'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'c3d00853-0daa-4514-9f1c-1a8b67b5d58e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '0990d9b6dd8f781bc862b7c85f53eb0613d198313f968d3b4743544aa84ffe58'}}, 'text': '@Teslarati @KlenderJoey Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet, 391, 6778, 2022-09-21 15:59:15, Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich I was instrumental in creating OpenAI, which made DALL-E, 153, 2442, 2022-09-21 15:17:17, I was instrumental in creating OpenAI, which made DALL-E\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist), 81, 1416, 2022-09-21 15:09:59, Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist)\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America., 64, 1285, 2022-09-21 15:00:26, Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America.\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich For sure, 58, 1171, 2022-09-21 14:55:11, For sure\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(, 154, 2286, 2022-09-21 14:36:49, Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich This is true, 60, 1356, 2022-09-21 14:26:17, This is true\\n@Teslarati @ResidentSponge With some improvements, it will achieve 300 Mps, 651, 14338, 2022-09-21 14:02:11, With some improvements, it will achieve 300 Mps\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.\\n\\nAlso, I was there for 2.5 years, previously attending govt schools for 9 of 12 years., 89, 1694, 2022-09-21 13:55:04, Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.Also, I was there for 2.5 years, previously attending govt schools for 9 of 12 years.', 'start_char_idx': 82792, 'end_char_idx': 85782, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7469842252500036}, {'node': {'id_': 'de6f0f78-221e-4fe4-b85d-6a992f6a6f6d', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': '3e563428-76cc-4aa0-83ee-3e8049217a83', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '9d14ec9cda85a9c8cc3c34b1cdf75fb105795e73d699b35cfc8234703dc37635'}, <NodeRelationship.NEXT: '3'>: {'node_id': '473c5d14-ac7c-45f4-a7eb-fb6ecc383b07', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'fb859d528497fbefaebc580603c2b0b92cca496d280392da3942cd40754a5cfc'}}, 'text': 'Walter Isaacson is writing a book., 215, 4004, 2022-04-21 08:49:51, I should, but that would take a lot of time away from Tesla, SpaceX &amp; other work. Walter Isaacson is writing a book.\\n@PPathole @teslaownersSV @woonomic @VaibhavSisinty Exactly, 137, 3587, 2022-04-21 08:42:11, Exactly\\n@billhuang688 @woonomic @VaibhavSisinty I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. \\n\\nIf it was really his baby, he would never have stood by while it starved to death.\\n\\nhttps://t.co/ZDfYLD5qkL, 355, 4152, 2022-04-21 08:40:07, I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. If it was really his baby, he would never have stood by while it starved to death.\\n@teslaownersSV @woonomic @VaibhavSisinty But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.\\n\\nWe would’ve lost at least a few talented people if he was the real deal, but we lost no one., 395, 5547, 2022-04-21 08:33:10, But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.We would’ve lost at least a few talented people if he was the real deal, but we lost no one.\\n@teslaownersSV @woonomic @VaibhavSisinty Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years., 452, 6083, 2022-04-21 08:27:57, Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years.\\n@woonomic @VaibhavSisinty They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!, 1071, 10676, 2022-04-21 08:21:40, They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!\\n@woonomic @VaibhavSisinty When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all., 425, 8706, 2022-04-21 08:15:37, When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all.\\n@woonomic @VaibhavSisinty If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for https://t.co/bOUOek5Cvy (later renamed PayPal), but that’s not what founding means., 506, 8631, 2022-04-21 08:12:03, If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for  (later renamed PayPal), but that’s not what founding means.\\n@woonomic @VaibhavSisinty Not even close to that. It was a shell corp with no employees, no IP, no designs, no prototype, literally nothing but a biz plan to commercialize AC Propulsion’s Tzero car, which was introduced to me by JB Straubel, *not* Eberhard. \\n\\nEven name “Tesla Motors” was owned by others!, 1324, 17242, 2022-04-21 08:09:07, Not even close to that.', 'start_char_idx': 380380, 'end_char_idx': 383649, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7451874390981541}], error=None, perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 11, 59, 728897), end_time=datetime.datetime(2024, 2, 29, 16, 12, 1, 267547)), pid=32024, tid=20596), RecordAppCall(stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine, id=2416071854224, init_bindings=None), name='get_response')), RecordAppCallMethod(path=Lens().app._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.refine.Refine, id=2416071854224, init_bindings=None), name='get_response')), RecordAppCallMethod(path=Lens().app._response_synthesizer._llm, method=Method(obj=Obj(cls=llama_index.llms.openai.base.OpenAI, id=2416099018320, init_bindings=None), name='wrapped_llm_chat'))], args={'_self': {'callback_manager': {'__tru_non_serialized_object': {'cls': {'name': 'CallbackManager', 'module': {'package_name': 'llama_index.core.callbacks', 'module_name': 'llama_index.core.callbacks.base'}, 'bases': None}, 'id': 2416092799248, 'init_bindings': None}}, 'system_prompt': None, 'messages_to_prompt': {'__tru_non_serialized_object': {'cls': {'name': 'function', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}, 'id': 2416066936192, 'init_bindings': None}}, 'completion_to_prompt': {'__tru_non_serialized_object': {'cls': {'name': 'function', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}, 'id': 2416067517216, 'init_bindings': None}}, 'output_parser': None, 'pydantic_program_mode': <PydanticProgramMode.DEFAULT: 'default'>, 'query_wrapper_prompt': None, 'model': 'gpt-3.5-turbo', 'temperature': 0.1, 'max_tokens': None, 'additional_kwargs': {}, 'max_retries': 3, 'timeout': 60.0, 'default_headers': None, 'reuse_client': True, 'api_key': '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', 'api_base': 'https://api.openai.com/v1', 'api_version': ''}, 'messages': [{'role': <MessageRole.SYSTEM: 'system'>, 'content': \"You are an expert Q&A system that is trusted around the world.\\nAlways answer the query using the provided context information, and not prior knowledge.\\nSome rules to follow:\\n1. Never directly reference the given context in your answer.\\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines.\", 'additional_kwargs': {}}, {'role': <MessageRole.USER: 'user'>, 'content': 'Context information is below.\\n---------------------\\nfile_path: c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv\\n\\n@Teslarati @KlenderJoey Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet, 391, 6778, 2022-09-21 15:59:15, Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich I was instrumental in creating OpenAI, which made DALL-E, 153, 2442, 2022-09-21 15:17:17, I was instrumental in creating OpenAI, which made DALL-E\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist), 81, 1416, 2022-09-21 15:09:59, Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist)\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America., 64, 1285, 2022-09-21 15:00:26, Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America.\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich For sure, 58, 1171, 2022-09-21 14:55:11, For sure\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(, 154, 2286, 2022-09-21 14:36:49, Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich This is true, 60, 1356, 2022-09-21 14:26:17, This is true\\n@Teslarati @ResidentSponge With some improvements, it will achieve 300 Mps, 651, 14338, 2022-09-21 14:02:11, With some improvements, it will achieve 300 Mps\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.\\n\\nAlso, I was there for 2.5 years, previously attending govt schools for 9 of 12 years., 89, 1694, 2022-09-21 13:55:04, Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.Also, I was there for 2.5 years, previously attending govt schools for 9 of 12 years.\\n\\nfile_path: c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv\\n\\nWalter Isaacson is writing a book., 215, 4004, 2022-04-21 08:49:51, I should, but that would take a lot of time away from Tesla, SpaceX &amp; other work. Walter Isaacson is writing a book.\\n@PPathole @teslaownersSV @woonomic @VaibhavSisinty Exactly, 137, 3587, 2022-04-21 08:42:11, Exactly\\n@billhuang688 @woonomic @VaibhavSisinty I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. \\n\\nIf it was really his baby, he would never have stood by while it starved to death.\\n\\nhttps://t.co/ZDfYLD5qkL, 355, 4152, 2022-04-21 08:40:07, I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. If it was really his baby, he would never have stood by while it starved to death.\\n@teslaownersSV @woonomic @VaibhavSisinty But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.\\n\\nWe would’ve lost at least a few talented people if he was the real deal, but we lost no one., 395, 5547, 2022-04-21 08:33:10, But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.We would’ve lost at least a few talented people if he was the real deal, but we lost no one.\\n@teslaownersSV @woonomic @VaibhavSisinty Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years., 452, 6083, 2022-04-21 08:27:57, Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years.\\n@woonomic @VaibhavSisinty They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!, 1071, 10676, 2022-04-21 08:21:40, They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!\\n@woonomic @VaibhavSisinty When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all., 425, 8706, 2022-04-21 08:15:37, When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all.\\n@woonomic @VaibhavSisinty If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for https://t.co/bOUOek5Cvy (later renamed PayPal), but that’s not what founding means., 506, 8631, 2022-04-21 08:12:03, If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for  (later renamed PayPal), but that’s not what founding means.\\n@woonomic @VaibhavSisinty Not even close to that. It was a shell corp with no employees, no IP, no designs, no prototype, literally nothing but a biz plan to commercialize AC Propulsion’s Tzero car, which was introduced to me by JB Straubel, *not* Eberhard. \\n\\nEven name “Tesla Motors” was owned by others!, 1324, 17242, 2022-04-21 08:09:07, Not even close to that.\\n---------------------\\nGiven the context information and not prior knowledge, answer the query.\\nQuery: What did the author do growing up?\\nAnswer: ', 'additional_kwargs': {}}]}, rets={'message': {'role': <MessageRole.ASSISTANT: 'assistant'>, 'content': 'The author attended government schools for 9 out of 12 years before spending 2.5 years at a particular school.', 'additional_kwargs': {}}, 'raw': {'id': 'chatcmpl-8xknK58ONq9OJIRHLoDwrN3Rrl2T0', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'The author attended government schools for 9 out of 12 years before spending 2.5 years at a particular school.', 'role': 'assistant', 'function_call': None, 'tool_calls': None}}], 'created': 1709251922, 'model': 'gpt-3.5-turbo-0125', 'object': 'chat.completion', 'system_fingerprint': 'fp_2b778c6b35', 'usage': {'completion_tokens': 25, 'prompt_tokens': 2023, 'total_tokens': 2048}}, 'delta': None, 'additional_kwargs': {}}, error=None, perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 12, 2, 175851), end_time=datetime.datetime(2024, 2, 29, 16, 12, 3, 628191)), pid=32024, tid=20596), RecordAppCall(stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine, id=2416071854224, init_bindings=None), name='get_response')), RecordAppCallMethod(path=Lens().app._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.refine.Refine, id=2416071854224, init_bindings=None), name='get_response'))], args={'query_str': 'What did the author do growing up?', 'text_chunks': ['file_path: c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv\\n\\n@Teslarati @KlenderJoey Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet, 391, 6778, 2022-09-21 15:59:15, Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich I was instrumental in creating OpenAI, which made DALL-E, 153, 2442, 2022-09-21 15:17:17, I was instrumental in creating OpenAI, which made DALL-E\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist), 81, 1416, 2022-09-21 15:09:59, Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist)\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America., 64, 1285, 2022-09-21 15:00:26, Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America.\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich For sure, 58, 1171, 2022-09-21 14:55:11, For sure\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(, 154, 2286, 2022-09-21 14:36:49, Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich This is true, 60, 1356, 2022-09-21 14:26:17, This is true\\n@Teslarati @ResidentSponge With some improvements, it will achieve 300 Mps, 651, 14338, 2022-09-21 14:02:11, With some improvements, it will achieve 300 Mps\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.\\n\\nAlso, I was there for 2.5 years, previously attending govt schools for 9 of 12 years., 89, 1694, 2022-09-21 13:55:04, Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.Also, I was there for 2.5 years, previously attending govt schools for 9 of 12 years.\\n\\nfile_path: c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv\\n\\nWalter Isaacson is writing a book., 215, 4004, 2022-04-21 08:49:51, I should, but that would take a lot of time away from Tesla, SpaceX &amp; other work. Walter Isaacson is writing a book.\\n@PPathole @teslaownersSV @woonomic @VaibhavSisinty Exactly, 137, 3587, 2022-04-21 08:42:11, Exactly\\n@billhuang688 @woonomic @VaibhavSisinty I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. \\n\\nIf it was really his baby, he would never have stood by while it starved to death.\\n\\nhttps://t.co/ZDfYLD5qkL, 355, 4152, 2022-04-21 08:40:07, I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. If it was really his baby, he would never have stood by while it starved to death.\\n@teslaownersSV @woonomic @VaibhavSisinty But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.\\n\\nWe would’ve lost at least a few talented people if he was the real deal, but we lost no one., 395, 5547, 2022-04-21 08:33:10, But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.We would’ve lost at least a few talented people if he was the real deal, but we lost no one.\\n@teslaownersSV @woonomic @VaibhavSisinty Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years., 452, 6083, 2022-04-21 08:27:57, Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years.\\n@woonomic @VaibhavSisinty They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!, 1071, 10676, 2022-04-21 08:21:40, They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!\\n@woonomic @VaibhavSisinty When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all., 425, 8706, 2022-04-21 08:15:37, When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all.\\n@woonomic @VaibhavSisinty If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for https://t.co/bOUOek5Cvy (later renamed PayPal), but that’s not what founding means., 506, 8631, 2022-04-21 08:12:03, If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for  (later renamed PayPal), but that’s not what founding means.\\n@woonomic @VaibhavSisinty Not even close to that. It was a shell corp with no employees, no IP, no designs, no prototype, literally nothing but a biz plan to commercialize AC Propulsion’s Tzero car, which was introduced to me by JB Straubel, *not* Eberhard. \\n\\nEven name “Tesla Motors” was owned by others!, 1324, 17242, 2022-04-21 08:09:07, Not even close to that.'], 'prev_response': None}, rets='The author attended government schools for 9 out of 12 years before spending 2.5 years at a particular school.', error=None, perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 12, 1, 771685), end_time=datetime.datetime(2024, 2, 29, 16, 12, 3, 630191)), pid=32024, tid=20596), RecordAppCall(stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='query')), RecordAppCallMethod(path=Lens().app._response_synthesizer, method=Method(obj=Obj(cls=llama_index.core.response_synthesizers.compact_and_refine.CompactAndRefine, id=2416071854224, init_bindings=None), name='get_response'))], args={'query_str': 'What did the author do growing up?', 'text_chunks': ['file_path: c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv\\n\\n@Teslarati @KlenderJoey Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet, 391, 6778, 2022-09-21 15:59:15, Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich I was instrumental in creating OpenAI, which made DALL-E, 153, 2442, 2022-09-21 15:17:17, I was instrumental in creating OpenAI, which made DALL-E\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist), 81, 1416, 2022-09-21 15:09:59, Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist)\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America., 64, 1285, 2022-09-21 15:00:26, Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America.\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich For sure, 58, 1171, 2022-09-21 14:55:11, For sure\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(, 154, 2286, 2022-09-21 14:36:49, Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich This is true, 60, 1356, 2022-09-21 14:26:17, This is true\\n@Teslarati @ResidentSponge With some improvements, it will achieve 300 Mps, 651, 14338, 2022-09-21 14:02:11, With some improvements, it will achieve 300 Mps\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.\\n\\nAlso, I was there for 2.5 years, previously attending govt schools for 9 of 12 years., 89, 1694, 2022-09-21 13:55:04, Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.Also, I was there for 2.5 years, previously attending govt schools for 9 of 12 years.', 'file_path: c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv\\n\\nWalter Isaacson is writing a book., 215, 4004, 2022-04-21 08:49:51, I should, but that would take a lot of time away from Tesla, SpaceX &amp; other work. Walter Isaacson is writing a book.\\n@PPathole @teslaownersSV @woonomic @VaibhavSisinty Exactly, 137, 3587, 2022-04-21 08:42:11, Exactly\\n@billhuang688 @woonomic @VaibhavSisinty I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. \\n\\nIf it was really his baby, he would never have stood by while it starved to death.\\n\\nhttps://t.co/ZDfYLD5qkL, 355, 4152, 2022-04-21 08:40:07, I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. If it was really his baby, he would never have stood by while it starved to death.\\n@teslaownersSV @woonomic @VaibhavSisinty But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.\\n\\nWe would’ve lost at least a few talented people if he was the real deal, but we lost no one., 395, 5547, 2022-04-21 08:33:10, But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.We would’ve lost at least a few talented people if he was the real deal, but we lost no one.\\n@teslaownersSV @woonomic @VaibhavSisinty Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years., 452, 6083, 2022-04-21 08:27:57, Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years.\\n@woonomic @VaibhavSisinty They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!, 1071, 10676, 2022-04-21 08:21:40, They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!\\n@woonomic @VaibhavSisinty When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all., 425, 8706, 2022-04-21 08:15:37, When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all.\\n@woonomic @VaibhavSisinty If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for https://t.co/bOUOek5Cvy (later renamed PayPal), but that’s not what founding means., 506, 8631, 2022-04-21 08:12:03, If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for  (later renamed PayPal), but that’s not what founding means.\\n@woonomic @VaibhavSisinty Not even close to that. It was a shell corp with no employees, no IP, no designs, no prototype, literally nothing but a biz plan to commercialize AC Propulsion’s Tzero car, which was introduced to me by JB Straubel, *not* Eberhard. \\n\\nEven name “Tesla Motors” was owned by others!, 1324, 17242, 2022-04-21 08:09:07, Not even close to that.']}, rets='The author attended government schools for 9 out of 12 years before spending 2.5 years at a particular school.', error=None, perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 12, 1, 373716), end_time=datetime.datetime(2024, 2, 29, 16, 12, 3, 630191)), pid=32024, tid=20596), RecordAppCall(stack=[RecordAppCallMethod(path=Lens().app, method=Method(obj=Obj(cls=llama_index.core.query_engine.retriever_query_engine.RetrieverQueryEngine, id=2416097729744, init_bindings=None), name='query'))], args={'str_or_query_bundle': 'What did the author do growing up?'}, rets={'response': 'The author attended government schools for 9 out of 12 years before spending 2.5 years at a particular school.', 'source_nodes': [{'node': {'id_': 'bcf8cce5-6f77-4391-8eae-1bd1a3749094', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': 'f3ad7036-414b-4f27-adcd-52244cbb27eb', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '18b69e4c8c3b6423e75e5c31805f7ee1ac2e3d8b1a20ed7fabef56d3845272c7'}, <NodeRelationship.NEXT: '3'>: {'node_id': 'c3d00853-0daa-4514-9f1c-1a8b67b5d58e', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': '0990d9b6dd8f781bc862b7c85f53eb0613d198313f968d3b4743544aa84ffe58'}}, 'text': '@Teslarati @KlenderJoey Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet, 391, 6778, 2022-09-21 15:59:15, Ecological paradise plans from south portion of Giga Texas to river look great, however we must first get the factory financially on its feet\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich I was instrumental in creating OpenAI, which made DALL-E, 153, 2442, 2022-09-21 15:17:17, I was instrumental in creating OpenAI, which made DALL-E\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist), 81, 1416, 2022-09-21 15:09:59, Although his methods of instruction very strongly favored stick over carrot, I did learn a tremendous amount of engineering from him &amp; appreciation of art (he is an extremely talented artist)\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America., 64, 1285, 2022-09-21 15:00:26, Boys High was a good school, but it was government, not private. Brooklyn would be considered a middle income, not exclusive, neighborhood in America.\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich For sure, 58, 1171, 2022-09-21 14:55:11, For sure\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(, 154, 2286, 2022-09-21 14:36:49, Errol ran out of money in the 90’s. My brother &amp; I financially supported him &amp; his extended family in South Africa since then on condition that he not do bad things. Unfortunately, he did do bad things :(\\n@enn_nafnlaus @treygahigh @TaraBull808 @theliamnissan @RBReich This is true, 60, 1356, 2022-09-21 14:26:17, This is true\\n@Teslarati @ResidentSponge With some improvements, it will achieve 300 Mps, 651, 14338, 2022-09-21 14:02:11, With some improvements, it will achieve 300 Mps\\n@enn_nafnlaus @TaraBull808 @treygahigh @theliamnissan @RBReich Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.\\n\\nAlso, I was there for 2.5 years, previously attending govt schools for 9 of 12 years., 89, 1694, 2022-09-21 13:55:04, Incorrect. There was tuition for students from outside the school district, which was a small part of the student body, incl black students, but not for students in district, which I was.Also, I was there for 2.5 years, previously attending govt schools for 9 of 12 years.', 'start_char_idx': 82792, 'end_char_idx': 85782, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7469842252500036}, {'node': {'id_': 'de6f0f78-221e-4fe4-b85d-6a992f6a6f6d', 'embedding': None, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'excluded_embed_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'excluded_llm_metadata_keys': ['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': 'f25b95df-9897-4a0c-a335-66b1190b9f2c', 'node_type': <ObjectType.DOCUMENT: '4'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '2f8554bd1a4850f4a42c784058d2e7ec8826a253c33cdc0020be2d734e4c6bfa'}, <NodeRelationship.PREVIOUS: '2'>: {'node_id': '3e563428-76cc-4aa0-83ee-3e8049217a83', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'hash': '9d14ec9cda85a9c8cc3c34b1cdf75fb105795e73d699b35cfc8234703dc37635'}, <NodeRelationship.NEXT: '3'>: {'node_id': '473c5d14-ac7c-45f4-a7eb-fb6ecc383b07', 'node_type': <ObjectType.TEXT: '1'>, 'metadata': {}, 'hash': 'fb859d528497fbefaebc580603c2b0b92cca496d280392da3942cd40754a5cfc'}}, 'text': 'Walter Isaacson is writing a book., 215, 4004, 2022-04-21 08:49:51, I should, but that would take a lot of time away from Tesla, SpaceX &amp; other work. Walter Isaacson is writing a book.\\n@PPathole @teslaownersSV @woonomic @VaibhavSisinty Exactly, 137, 3587, 2022-04-21 08:42:11, Exactly\\n@billhuang688 @woonomic @VaibhavSisinty I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. \\n\\nIf it was really his baby, he would never have stood by while it starved to death.\\n\\nhttps://t.co/ZDfYLD5qkL, 355, 4152, 2022-04-21 08:40:07, I put in all I had, whereas Eberhard, who was worth over $10M at the time, refused to invest a penny to help Tesla. If it was really his baby, he would never have stood by while it starved to death.\\n@teslaownersSV @woonomic @VaibhavSisinty But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.\\n\\nWe would’ve lost at least a few talented people if he was the real deal, but we lost no one., 395, 5547, 2022-04-21 08:33:10, But if he was actually capable of creating a company like Tesla, he would have done so after he was fired for lying outrageously about cost &amp; progress of Roadster in mid 2007.We would’ve lost at least a few talented people if he was the real deal, but we lost no one.\\n@teslaownersSV @woonomic @VaibhavSisinty Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years., 452, 6083, 2022-04-21 08:27:57, Eberhard has been relentlessly pushing a false narrative about Tesla for 15 years. He’s a compelling liar. Fooled me for almost 3 years.\\n@woonomic @VaibhavSisinty They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!, 1071, 10676, 2022-04-21 08:21:40, They say history is written by the victors, but not on Wikipedia if the losing party is still alive &amp; has lots of time on their hands!\\n@woonomic @VaibhavSisinty When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all., 425, 8706, 2022-04-21 08:15:37, When Eberhard was fired unanimously by the board in July 2007 (for damn good reasons), no one left with him. That says it all.\\n@woonomic @VaibhavSisinty If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for https://t.co/bOUOek5Cvy (later renamed PayPal), but that’s not what founding means., 506, 8631, 2022-04-21 08:12:03, If filing a shell corp constitutes “founding a company”, then I’d be the only founder of PayPal, since I filed the original incorporation docs for  (later renamed PayPal), but that’s not what founding means.\\n@woonomic @VaibhavSisinty Not even close to that. It was a shell corp with no employees, no IP, no designs, no prototype, literally nothing but a biz plan to commercialize AC Propulsion’s Tzero car, which was introduced to me by JB Straubel, *not* Eberhard. \\n\\nEven name “Tesla Motors” was owned by others!, 1324, 17242, 2022-04-21 08:09:07, Not even close to that.', 'start_char_idx': 380380, 'end_char_idx': 383649, 'text_template': '{metadata_str}\\n\\n{content}', 'metadata_template': '{key}: {value}', 'metadata_seperator': '\\n'}, 'score': 0.7451874390981541}], 'metadata': {'bcf8cce5-6f77-4391-8eae-1bd1a3749094': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}, 'de6f0f78-221e-4fe4-b85d-6a992f6a6f6d': {'file_path': 'c:\\\\Users\\\\Evan\\\\Desktop\\\\training_llm\\\\rag_api\\\\rag_api\\\\uploads\\\\cleandata.csv', 'file_name': 'c:/Users/Evan/Desktop/training_llm/rag_api/rag_api/uploads/cleandata.csv', 'file_type': 'text/csv', 'file_size': 498541, 'creation_date': '2024-02-29', 'last_modified_date': '2024-02-29'}}}, error=None, perf=Perf(start_time=datetime.datetime(2024, 2, 29, 16, 11, 59, 342173), end_time=datetime.datetime(2024, 2, 29, 16, 12, 3, 630191)), pid=32024, tid=20596)], feedback_and_future_results=[(Feedback(tru_class_info=trulens_eval.feedback.feedback.Feedback, implementation=Method(obj=Obj(cls=trulens_eval.feedback.groundedness.Groundedness, id=2416098713568, init_bindings=Bindings(args=(), kwargs={'tru_class_info': {'name': 'Groundedness', 'module': {'package_name': 'trulens_eval.feedback', 'module_name': 'trulens_eval.feedback.groundedness'}, 'bases': [{'name': 'Groundedness', 'module': {'package_name': 'trulens_eval.feedback', 'module_name': 'trulens_eval.feedback.groundedness'}, 'bases': None}, {'name': 'WithClassInfo', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.pyschema'}, 'bases': None}, {'name': 'SerialModel', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.serial'}, 'bases': None}, {'name': 'BaseModel', 'module': {'package_name': 'pydantic', 'module_name': 'pydantic.main'}, 'bases': None}, {'name': 'object', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}]}, 'groundedness_provider': {'tru_class_info': {'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': [{'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': None}, {'name': 'LLMProvider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'Provider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'WithClassInfo', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.pyschema'}, 'bases': None}, {'name': 'SerialModel', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.serial'}, 'bases': None}, {'name': 'BaseModel', 'module': {'package_name': 'pydantic', 'module_name': 'pydantic.main'}, 'bases': None}, {'name': 'object', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}]}, 'model_engine': 'gpt-3.5-turbo'}})), name='groundedness_measure_with_cot_reasons'), aggregator=Method(obj=Obj(cls=trulens_eval.feedback.groundedness.Groundedness, id=2416098713568, init_bindings=Bindings(args=(), kwargs={'tru_class_info': {'name': 'Groundedness', 'module': {'package_name': 'trulens_eval.feedback', 'module_name': 'trulens_eval.feedback.groundedness'}, 'bases': [{'name': 'Groundedness', 'module': {'package_name': 'trulens_eval.feedback', 'module_name': 'trulens_eval.feedback.groundedness'}, 'bases': None}, {'name': 'WithClassInfo', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.pyschema'}, 'bases': None}, {'name': 'SerialModel', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.serial'}, 'bases': None}, {'name': 'BaseModel', 'module': {'package_name': 'pydantic', 'module_name': 'pydantic.main'}, 'bases': None}, {'name': 'object', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}]}, 'groundedness_provider': {'tru_class_info': {'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': [{'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': None}, {'name': 'LLMProvider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'Provider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'WithClassInfo', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.pyschema'}, 'bases': None}, {'name': 'SerialModel', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.serial'}, 'bases': None}, {'name': 'BaseModel', 'module': {'package_name': 'pydantic', 'module_name': 'pydantic.main'}, 'bases': None}, {'name': 'object', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}]}, 'model_engine': 'gpt-3.5-turbo'}})), name='grounded_statements_aggregator'), feedback_definition_id='feedback_definition_hash_340a2d56d352f366b5f58e211254f6b2', selectors={'source': Lens().__record__.app.query.rets.source_nodes[:].node.text.collect(), 'statement': Lens().__record__.main_output}, supplied_name=None, higher_is_better=True, imp=<bound method Groundedness.groundedness_measure_with_cot_reasons of Groundedness(tru_class_info=trulens_eval.feedback.groundedness.Groundedness, groundedness_provider=OpenAI(tru_class_info=trulens_eval.feedback.provider.openai.OpenAI, endpoint=Endpoint(openai), model_engine='gpt-3.5-turbo'))>, agg=<bound method Groundedness.grounded_statements_aggregator of Groundedness(tru_class_info=trulens_eval.feedback.groundedness.Groundedness, groundedness_provider=OpenAI(tru_class_info=trulens_eval.feedback.provider.openai.OpenAI, endpoint=Endpoint(openai), model_engine='gpt-3.5-turbo'))>), <Future at 0x2328d2dd650 state=running>), (Feedback(tru_class_info=trulens_eval.feedback.feedback.Feedback, implementation=Method(obj=Obj(cls=trulens_eval.feedback.provider.openai.OpenAI, id=2416098708768, init_bindings=Bindings(args=(), kwargs={'tru_class_info': {'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': [{'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': None}, {'name': 'LLMProvider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'Provider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'WithClassInfo', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.pyschema'}, 'bases': None}, {'name': 'SerialModel', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.serial'}, 'bases': None}, {'name': 'BaseModel', 'module': {'package_name': 'pydantic', 'module_name': 'pydantic.main'}, 'bases': None}, {'name': 'object', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}]}, 'model_engine': 'gpt-3.5-turbo'})), name='relevance'), aggregator=Function(module=Module(package_name='numpy', module_name='numpy'), cls=None, name='mean'), feedback_definition_id='feedback_definition_hash_9d7ee70c54dd0a581cb853de7a74a2d1', selectors={'prompt': Lens().__record__.main_input, 'response': Lens().__record__.main_output}, supplied_name=None, higher_is_better=True, imp=<bound method LLMProvider.relevance of OpenAI(tru_class_info=trulens_eval.feedback.provider.openai.OpenAI, endpoint=Endpoint(openai), model_engine='gpt-3.5-turbo')>, agg=<function mean at 0x00000232C08B56F0>), <Future at 0x2328b210ed0 state=running>), (Feedback(tru_class_info=trulens_eval.feedback.feedback.Feedback, implementation=Method(obj=Obj(cls=trulens_eval.feedback.provider.openai.OpenAI, id=2416098708768, init_bindings=Bindings(args=(), kwargs={'tru_class_info': {'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': [{'name': 'OpenAI', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.openai'}, 'bases': None}, {'name': 'LLMProvider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'Provider', 'module': {'package_name': 'trulens_eval.feedback.provider', 'module_name': 'trulens_eval.feedback.provider.base'}, 'bases': None}, {'name': 'WithClassInfo', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.pyschema'}, 'bases': None}, {'name': 'SerialModel', 'module': {'package_name': 'trulens_eval.utils', 'module_name': 'trulens_eval.utils.serial'}, 'bases': None}, {'name': 'BaseModel', 'module': {'package_name': 'pydantic', 'module_name': 'pydantic.main'}, 'bases': None}, {'name': 'object', 'module': {'package_name': '', 'module_name': 'builtins'}, 'bases': None}]}, 'model_engine': 'gpt-3.5-turbo'})), name='qs_relevance'), aggregator=Function(module=Module(package_name='numpy', module_name='numpy'), cls=None, name='mean'), feedback_definition_id='feedback_definition_hash_79a47d63392fa72054494539f2c7523f', selectors={'question': Lens().__record__.main_input, 'statement': Lens().__record__.app.query.rets.source_nodes[:].node.text}, supplied_name=None, higher_is_better=True, imp=<bound method LLMProvider.qs_relevance of OpenAI(tru_class_info=trulens_eval.feedback.provider.openai.OpenAI, endpoint=Endpoint(openai), model_engine='gpt-3.5-turbo')>, agg=<function mean at 0x00000232C08B56F0>), <Future at 0x2328edb2650 state=running>)], feedback_results=[<Future at 0x2328d2dd650 state=running>, <Future at 0x2328b210ed0 state=running>, <Future at 0x2328edb2650 state=running>])"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# The record of the app invocation can be retrieved from the `recording`:\n",
        "\n",
        "rec = recording.get() # use .get if only one record\n",
        "# recs = recording.records # use .records if multiple\n",
        "\n",
        "display(rec)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "0Liw9gaBGcgP"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Starting dashboard ...\n",
            "Config file already exists. Skipping writing process.\n",
            "Credentials file already exists. Skipping writing process.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "dc92c8085da44d4cbff16f70aa9232fb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Accordion(children=(VBox(children=(VBox(children=(Label(value='STDOUT'), Output())), VBox(children=(Label(valu…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dashboard started at http://192.168.1.29:8501 .\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Popen: returncode: None args: ['streamlit', 'run', '--server.headless=True'...>"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tru.run_dashboard()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "YoBPUdXRGcgP"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "groundedness_measure_with_cot_reasons 0.5\n",
            "relevance 0.8\n",
            "qs_relevance 0.15000000000000002\n"
          ]
        }
      ],
      "source": [
        "# The results of the feedback functions can be rertireved from\n",
        "# `Record.feedback_results` or using the `wait_for_feedback_result` method. The\n",
        "# results if retrieved directly are `Future` instances (see\n",
        "# `concurrent.futures`). You can use `as_completed` to wait until they have\n",
        "# finished evaluating or use the utility method:\n",
        "\n",
        "for feedback, feedback_result in rec.wait_for_feedback_results().items():\n",
        "    print(feedback.name, feedback_result.result)\n",
        "\n",
        "# See more about wait_for_feedback_results:\n",
        "# help(rec.wait_for_feedback_results)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "ZrlkrJcKGcgP"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>app_id</th>\n",
              "      <th>app_json</th>\n",
              "      <th>type</th>\n",
              "      <th>record_id</th>\n",
              "      <th>input</th>\n",
              "      <th>output</th>\n",
              "      <th>tags</th>\n",
              "      <th>record_json</th>\n",
              "      <th>cost_json</th>\n",
              "      <th>perf_json</th>\n",
              "      <th>ts</th>\n",
              "      <th>relevance</th>\n",
              "      <th>qs_relevance</th>\n",
              "      <th>groundedness_measure_with_cot_reasons</th>\n",
              "      <th>relevance_calls</th>\n",
              "      <th>qs_relevance_calls</th>\n",
              "      <th>groundedness_measure_with_cot_reasons_calls</th>\n",
              "      <th>latency</th>\n",
              "      <th>total_tokens</th>\n",
              "      <th>total_cost</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>LlamaIndex_App1</td>\n",
              "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
              "      <td>RetrieverQueryEngine(llama_index.core.query_en...</td>\n",
              "      <td>record_hash_ddf49602dea32aa32f54f9a400508f6b</td>\n",
              "      <td>\"What did the author do growing up?\"</td>\n",
              "      <td>\"The author attended government schools for 9 ...</td>\n",
              "      <td>-</td>\n",
              "      <td>{\"record_id\": \"record_hash_ddf49602dea32aa32f5...</td>\n",
              "      <td>{\"n_requests\": 2, \"n_successful_requests\": 2, ...</td>\n",
              "      <td>{\"start_time\": \"2024-02-29T16:09:24.217268\", \"...</td>\n",
              "      <td>2024-02-29T16:09:29.803582</td>\n",
              "      <td>0.8</td>\n",
              "      <td>0.15</td>\n",
              "      <td>0.5</td>\n",
              "      <td>[{'args': {'prompt': 'What did the author do g...</td>\n",
              "      <td>[{'args': {'question': 'What did the author do...</td>\n",
              "      <td>[{'args': {'source': ['@Teslarati @KlenderJoey...</td>\n",
              "      <td>5</td>\n",
              "      <td>2045</td>\n",
              "      <td>0.003063</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>LlamaIndex_App1</td>\n",
              "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
              "      <td>RetrieverQueryEngine(llama_index.core.query_en...</td>\n",
              "      <td>record_hash_56e53381af40be526ab632ae206ada68</td>\n",
              "      <td>\"What did the author do growing up?\"</td>\n",
              "      <td>\"The author attended government schools for 9 ...</td>\n",
              "      <td>-</td>\n",
              "      <td>{\"record_id\": \"record_hash_56e53381af40be526ab...</td>\n",
              "      <td>{\"n_requests\": 2, \"n_successful_requests\": 2, ...</td>\n",
              "      <td>{\"start_time\": \"2024-02-29T16:11:59.342173\", \"...</td>\n",
              "      <td>2024-02-29T16:12:03.631727</td>\n",
              "      <td>0.8</td>\n",
              "      <td>0.15</td>\n",
              "      <td>0.5</td>\n",
              "      <td>[{'args': {'prompt': 'What did the author do g...</td>\n",
              "      <td>[{'args': {'question': 'What did the author do...</td>\n",
              "      <td>[{'args': {'source': ['@Teslarati @KlenderJoey...</td>\n",
              "      <td>4</td>\n",
              "      <td>2056</td>\n",
              "      <td>0.003085</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            app_id                                           app_json  \\\n",
              "0  LlamaIndex_App1  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
              "1  LlamaIndex_App1  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
              "\n",
              "                                                type  \\\n",
              "0  RetrieverQueryEngine(llama_index.core.query_en...   \n",
              "1  RetrieverQueryEngine(llama_index.core.query_en...   \n",
              "\n",
              "                                      record_id  \\\n",
              "0  record_hash_ddf49602dea32aa32f54f9a400508f6b   \n",
              "1  record_hash_56e53381af40be526ab632ae206ada68   \n",
              "\n",
              "                                  input  \\\n",
              "0  \"What did the author do growing up?\"   \n",
              "1  \"What did the author do growing up?\"   \n",
              "\n",
              "                                              output tags  \\\n",
              "0  \"The author attended government schools for 9 ...    -   \n",
              "1  \"The author attended government schools for 9 ...    -   \n",
              "\n",
              "                                         record_json  \\\n",
              "0  {\"record_id\": \"record_hash_ddf49602dea32aa32f5...   \n",
              "1  {\"record_id\": \"record_hash_56e53381af40be526ab...   \n",
              "\n",
              "                                           cost_json  \\\n",
              "0  {\"n_requests\": 2, \"n_successful_requests\": 2, ...   \n",
              "1  {\"n_requests\": 2, \"n_successful_requests\": 2, ...   \n",
              "\n",
              "                                           perf_json  \\\n",
              "0  {\"start_time\": \"2024-02-29T16:09:24.217268\", \"...   \n",
              "1  {\"start_time\": \"2024-02-29T16:11:59.342173\", \"...   \n",
              "\n",
              "                           ts  relevance  qs_relevance  \\\n",
              "0  2024-02-29T16:09:29.803582        0.8          0.15   \n",
              "1  2024-02-29T16:12:03.631727        0.8          0.15   \n",
              "\n",
              "   groundedness_measure_with_cot_reasons  \\\n",
              "0                                    0.5   \n",
              "1                                    0.5   \n",
              "\n",
              "                                     relevance_calls  \\\n",
              "0  [{'args': {'prompt': 'What did the author do g...   \n",
              "1  [{'args': {'prompt': 'What did the author do g...   \n",
              "\n",
              "                                  qs_relevance_calls  \\\n",
              "0  [{'args': {'question': 'What did the author do...   \n",
              "1  [{'args': {'question': 'What did the author do...   \n",
              "\n",
              "         groundedness_measure_with_cot_reasons_calls  latency  total_tokens  \\\n",
              "0  [{'args': {'source': ['@Teslarati @KlenderJoey...        5          2045   \n",
              "1  [{'args': {'source': ['@Teslarati @KlenderJoey...        4          2056   \n",
              "\n",
              "   total_cost  \n",
              "0    0.003063  \n",
              "1    0.003085  "
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "records, feedback = tru.get_records_and_feedback(app_ids=[\"LlamaIndex_App1\"])\n",
        "\n",
        "records.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "1DsiilIyGcgP"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>relevance</th>\n",
              "      <th>groundedness_measure_with_cot_reasons</th>\n",
              "      <th>qs_relevance</th>\n",
              "      <th>latency</th>\n",
              "      <th>total_cost</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>app_id</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>LlamaIndex_App1</th>\n",
              "      <td>0.8</td>\n",
              "      <td>0.5</td>\n",
              "      <td>0.15</td>\n",
              "      <td>4.5</td>\n",
              "      <td>0.003074</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                 relevance  groundedness_measure_with_cot_reasons  \\\n",
              "app_id                                                              \n",
              "LlamaIndex_App1        0.8                                    0.5   \n",
              "\n",
              "                 qs_relevance  latency  total_cost  \n",
              "app_id                                              \n",
              "LlamaIndex_App1          0.15      4.5    0.003074  "
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tru.get_leaderboard(app_ids=[\"LlamaIndex_App1\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aUF0VlYTGcgP"
      },
      "source": [
        "## Explore in a Dashboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "9wXvH288GcgP"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Starting dashboard ...\n",
            "Config file already exists. Skipping writing process.\n",
            "Credentials file already exists. Skipping writing process.\n",
            "Dashboard already running at path:   Network URL: http://192.168.1.29:8501\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Popen: returncode: None args: ['streamlit', 'run', '--server.headless=True'...>"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tru.run_dashboard() # open a local streamlit app to explore\n",
        "\n",
        "# tru.stop_dashboard() # stop if needed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E6eMIwMKGcgP"
      },
      "source": [
        "Alternatively, you can run `trulens-eval` from a command line in the same folder to start the dashboard."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.11.4 ('agents')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    },
    "vscode": {
      "interpreter": {
        "hash": "7d153714b979d5e6d08dd8ec90712dd93bff2c9b6c1f0c118169738af3430cd4"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}