Jim/vectordb update

.github/workflows/deploy.yml

@@ -11,7 +11,7 @@ permissions:
 jobs:
   deploy:
     runs-on: ubuntu-latest
-    if: ${{ github.event.head_commit.message != 'Merge pull request*' }} # Ensure it's not a merge commit
+    if: ${{ github.event.head_commit.message != 'Merge pull request*' }}
 
     steps:
       - uses: actions/checkout@v3
@@ -56,6 +56,9 @@ jobs:
           WEBUI_SECRET_KEY: ${{ secrets.WEBUI_SECRET_KEY }}
           DEV_ADMIN_EMAILS: ${{ secrets.DEV_ADMIN_EMAILS }}
           DEV_USER_EMAILS: ${{ secrets.DEV_USER_EMAILS }}
+          RAG_OPENAI_API_BASE_URL: ${{ secrets.RAG_OPENAI_API_BASE_URL }}
+          RAG_OPENAI_API_KEY: ${{ secrets.RAG_OPENAI_API_KEY }}
+          RAG_EMBEDDING_ENGINE: ${{ secrets.RAG_EMBEDDING_ENGINE }}
         with:
           cf_username: ${{ secrets.CLOUDGOV_USERNAME }}
           cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
@@ -80,6 +83,9 @@ jobs:
             --var WEBUI_SECRET_KEY="$WEBUI_SECRET_KEY"
             --var DEV_ADMIN_EMAILS="$DEV_ADMIN_EMAILS"
             --var DEV_USER_EMAILS="$DEV_USER_EMAILS"
+            --var RAG_OPENAI_API_BASE_URL="$RAG_OPENAI_API_BASE_URL"
+            --var RAG_OPENAI_API_KEY="$RAG_OPENAI_API_KEY"
+            --var RAG_EMBEDDING_ENGINE="$RAG_EMBEDDING_ENGINE"
 
   bail:
     runs-on: ubuntu-latest

.husky/pre-commit

@@ -1,2 +1,5 @@
+echo "Scanning commit history, ~30-60s... 💅🥱⏳"
+gitleaks detect --log-opts="-n 1"
+npm install
 npm run format
 npm run i18n:parse

README.md

@@ -2,7 +2,7 @@
 
 1. **Ensure Python 3.11 is installed**:
 
-   - You can check your Python version by running:
+   - You can check your Python3.11.x version by running:
 
    ```bash
    python3.11 --version
@@ -39,22 +39,43 @@
    nvm use 20.15.1
    ```
 
-5. **Install deps, build and run**:
+5. **Install gitleaks**:
 
-   - Install, build and run with hot reloading:
+   - Install with homebrew, then start a new terminal:
 
    ```bash
-   rm ./backend/data/webui.db && \
+   brew install gitleaks
+   ```
+
+6. **Run redis**:
+
+   - Check for redis on the default `lsof -i:6379`, if it's there, make sure it's using the default creds, or kill it and run:
+
+   ```bash
+   docker run -d --name redis -p 6379:6379 -p 8001:8001 redis/redis-stack:latest
+   ```
+
+7. **Install deps, build and run**:
+
+   - Make sure you've got the .env file set up, then install, build and run with hot reloading:
+
+   ```bash
+   loadenv .env && \
+   rm ./backend/data/webui.db || true && \
    pip install -r ./backend/requirements.txt && \
-   rm -rf node_modules && \
+   rm -rf node_modules || true && \
    npm install && \
+   npx husky init && \
+   cp pre-commit .husky/pre-commit && \
    npm run build && \
    ./backend/start.sh
    ```
 
-   - You should see the pipelines server running at 9099, the static files should be compiled and the webui server should be running at http://0.0.0.0:8080.
+   - The first user to sign up to a new installation should get the admin role. You can also predefine user roles in the .env file. Github auth checks that email domain is in ['gsa.gov'], but you can easily modify it at `backend/apps/webui/routers/auths.py:233`. Eventually we'll need to make github for local dev only for compliance reasons.
+   - After the first install, you can just run `./backend/start.sh`. First app startup will take a minute even after it says `Uvicorn running on http://0.0.0.0:8080`, once you see the ascii art, all of the features should be available. You may see a 500 the first time and need to refresh. You can run a front end dev server that hot reloads via `npm run dev` but connecting it to the backend and getting auth redirects with live servers working is unresolved due to the frontend and back running on different ports. We probably need to mock auth locally.
+   - ollama is not required for the app to run, but it is assumed, you can ignore the 500s if its not running. If you want to use it, you can install it with `brew install ollama`. You can then run `ollama serve` to start the server. You can then add a model to ollama with `ollama run mistral`.
 
-6. **Set up pipelines to access models via API**:
+8. **Set up pipelines to access models via API**:
 
-- The first user to sign up to a new installation should get the admin role. Once you're in, navigate to the Admin Panel > Settings > Connections > OpenAI API section. Set the API URL to http://localhost:9099 and the API key to 0p3n-w3bu! and hit refresh to see if it connects.
-- After completing these steps, the model specified in the pipeline should be available in the drop down at the upper left when you create a new conversation.
+- Once you're in, you should see the four default models available in the chat. If not, check that the pipelines server is running on 9099 and in the UI click on your user in the lower left > Admin Panel > Settings > Connections > OpenAI API section. Set the API URL to [<http://localhost:9099](http://localhost:9099>) and the API key to 0p3n-w3bu! and hit refresh to see if it connects to the pipeline server.
+- After completing these steps, the models specified in the pipeline settings should be available in the drop down at the upper left when you create a new conversation.

TODO.md

@@ -1,9 +1,94 @@
 # TODOs
 
-- Integrate cloud SSO
-- Use postgres for db
-- Build step during CI.
-- Consider checking for state and code in front-end auth route and then redirecting to /auth (and clearing params) on any auth failures. This will allow us to avoid flashing the sign in screen before showing "signing into ai sandbox..."
-- Save and retrieve valves.json data from env instead of json
-- Remove pipeline add from admin settings > pipelines
-- Investigation RAG set up
+## Current Project TODOs
+
+- Put up a warning banner on publicly shared links that says "This is a test for technology evaluation purposes only, this is not a decisional document" Make demo Videos: playground switching sys prompts, transfer sys prompt to custom model with docs
+- Cite sources in rag (do this for other models, too):
+- Allow bulk select even when nothing is tagged: <https://github.com/GSA-TTS/temp-10x-ai-sandbox/blob/66acbd1b866e9c93edc4457048af323c2fc0a5d0/src/lib/components/workspace/Documents.svelte#L280-L281>
+- Models and docs need ownership and visibility in database, owner str, visibility [str]
+- gray out vision for custom models that don't have vision
+- Ability to share docs/models by pasting an email of another user ("This email is belongs to a sandbox user. They will now have access to the model" / "This email does not belong to a sandbox user"). Probably want comma separation...
+- Add http links to Doc upload options, seems mostly done
+- stop checking for ollama models?
+- Fix tagging ux, add tags should be clickable, should save when you hit save
+- Check and batch-tag multiple docs
+- documents tab needs a spinner when processing docs
+- existing tags should be selectable when adding to docs
+- Need initial (Type a message down there 👇 to get started / explain that this is a chat interface) message for new users
+- Soft outline around new chat and documents buttons
+- Fix translations for docs & more and search your chats, as well as sign in screen
+- create a couple default custom prompts shortcuts like /quick-summary ([type something here]) and /quick-summary-from-paste {{clipboard}}
+- Unhide Tools from Docs & more layout
+- Chroma is kept in ram... need to swap for redisVL
+- tell run format to ignore package.json so we don't have to npm install in pre-commit to get formatting to match in the ci check
+- change cors all origins to front end and pipelines only
+
+## TODOs Found in Codebase
+
+### Git Hooks
+
+- Replace with appropriate checks (e.g. spell checking)
+- Replace with appropriate checks for this patch (e.g. checkpatch.pl)
+- Replace with appropriate checks for the whole series (e.g. quick build, coding style checks, etc.)
+
+### Passlib
+
+- Support the drupal phpass variants (see phpass homepage)
+- Remove redundant category deprecations
+- Verify digest size (if digest is known)
+- Find out what crowd's policy is re: unicode
+- Find out what grub's policy is re: unicode
+- Try to detect incorrect 8bit/wraparound hashes
+- Check for 2x support
+- Figure out way to skip these tests when not needed
+- Would like to implementing verify() directly, to skip need for parsing hash strings
+- Would like to dynamically pick this based on system
+- Make default block size configurable via using() and deprecatable via .needs_update()
+- Could support the optional 'data' parameter
+- Switch to working w/ str or unicode
+- Factor out variable checksum size support into a mixin
+- Add in 'encoding' support once that's finalized in 1.8 / 1.9
+
+### Pandas
+
+- Use inspect.VALUE here, and make the annotations lazily evaluated
+- Handle huge files in some other way
+- Remove in 1.0 release
+- Check -numweeks for next year
+- Remove the '$' check after JMESPath supports it
+- Make this configurable in the future
+- Support text-indent, padding-left -> alignment.indent
+- Allow for special properties -excel-pattern-bgcolor and -excel-pattern-type
+- Refactor to remove code duplication
+- Handle record value which are lists
+- Do this timedelta properly in objToJSON.c
+- Figure out why these two versions of `meta` dont always match
+- Support other fill patterns
+- Handle cell width and height: needs support in pandas.io.excel
+- Try to consolidate the concat visible rows methods
+- Expand to handle datetime to integer conversion
+- Handle datetime to integer conversion
+- Expand this to handle a default datetime format
+
+### Dateutil
+
+- Check week number 1 of next year as well
+- Remove after deprecation period
+
+### Boto3
+
+- Remove the '$' check after JMESPath supports it
+- Make this configurable in the future
+
+### Openpyxl
+
+- This can probably be sped up using a regex
+
+### Other
+
+- Support %
+- Don't lowercase case sensitive parts of values (strings)
+- Hack in buffer capability for pyxlsb
+- There is no way to distinguish between floats and datetimes in pyxlsb
+- Can we use current color as initial value to comply with CSS standards?
+- Warn user if item entered more than once (e.g. "border: red green")