Merge branch 'enricoros:main' into main

.dockerignore

@@ -1,7 +1,12 @@
 # big-AGI non-code files
 /docs/
+/dist/
 README.md
 
+# Ignore build and log files
+Dockerfile
+/.dockerignore
+
 # Node build artifacts
 /node_modules
 /.pnp

.github/workflows/docker-image.yml

@@ -57,4 +57,5 @@ jobs:
           file: Dockerfile
           push: true
           tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
+          labels: ${{ steps.meta.outputs.labels }}
+          build-args: NEXT_PUBLIC_GA4_MEASUREMENT_ID=${{ secrets.GA4_MEASUREMENT_ID }}

Dockerfile

@@ -2,6 +2,7 @@
 FROM node:18-alpine AS base
 ENV NEXT_TELEMETRY_DISABLED 1
 
+
 # Dependencies
 FROM base AS deps
 WORKDIR /app
@@ -14,10 +15,15 @@ COPY src/server/prisma ./src/server/prisma
 ENV NODE_ENV development
 RUN npm ci
 
+
 # Builder
 FROM base AS builder
 WORKDIR /app
 
+# Optional argument to configure GA4 at build time (see: docs/deploy-analytics.md)
+ARG NEXT_PUBLIC_GA4_MEASUREMENT_ID
+ENV NEXT_PUBLIC_GA4_MEASUREMENT_ID=${NEXT_PUBLIC_GA4_MEASUREMENT_ID}
+
 # Copy development deps and source
 COPY --from=deps /app/node_modules ./node_modules
 COPY . .
@@ -29,6 +35,7 @@ RUN npm run build
 # Reduce installed packages to production-only
 RUN npm prune --production
 
+
 # Runner
 FROM base AS runner
 WORKDIR /app

docs/config-feature-browse.md

@@ -3,11 +3,16 @@
 Allows users to load web pages across various components of `big-AGI`. This feature is supported by Puppeteer-based
 browsing services, which are the most common way to render web pages in a headless environment.
 
-Once configured, the Browsing service provides this functionality:
+Once configured, the Browsing service provides the following functionality:
 
-- **Paste a URL**: Simply paste/drag a URL into the chat, and `big-AGI` will load and attach the page (very effective)
-- **Use /browse**: Type `/browse [URL]` in the chat to command `big-AGI` to load the specified web page
-- **ReAct**: ReAct will automatically use the `loadURL()` function whenever a URL is encountered
+- ✅ **Paste a URL**: Simply paste/drag a URL into the chat, and `big-AGI` will load and attach the page (very effective)
+- ✅ **Use /browse**: Type `/browse [URL]` in the chat to command `big-AGI` to load the specified web page
+- ✅ **ReAct**: ReAct will automatically use the `loadURL()` function whenever a URL is encountered
+
+It does not yet support the following functionality:
+
+- ✖️ **Auto-browsing by LLMs**: if an LLM encounters a URL, it will NOT load the page and will likely respond
+  that it cannot browse the web - No technical limitation, just haven't gotten to implement this yet outside of `/react` yet
 
 First of all, you need to procure a Puppteer web browsing service endpoint. `big-AGI` supports services like:
 
@@ -109,3 +114,5 @@ If you encounter any issues or have questions about configuring the browse funct
 ---
 
 Enjoy the enhanced browsing experience within `big-AGI` and explore the web without ever leaving your chat!
+
+Last updated on Feb 27, 2024 ([edit on GitHub](https://github.com/enricoros/big-AGI/edit/main/docs/config-feature-browse.md))

docs/deploy-analytics.md

@@ -0,0 +1,63 @@
+# big-AGI Analytics
+
+The open-source big-AGI project provides support for the following analytics services:
+
+- **Vercel Analytics**: automatic when deployed to Vercel
+- **Google Analytics 4**: manual setup required
+
+The following is a quick overview of the Analytics options for the deployers of this open-source project.
+big-AGI is deployed to many large-scale and enterprise though various ways (custom builds, Docker, Vercel, Cloudflare, etc.),
+and this guide is for its customization.
+
+## Service Configuration
+
+### Vercel Analytics
+
+- Why: understand coarse traction, and identify deployment issues - all without tracking individual users
+- What: top pages, top referrers, country of origin, operating system, browser, and page speed metrics
+
+Vercel Analytics and Speed Insights are local API endpoints deployed to your domain, so everything stays within your
+domain. Furthermore, the Vercel Analytics service is privacy-friendly, and does not track individual users.
+
+This service is avaialble to system administrators when deploying to Vercel. It is automatically enabled when deploying to Vercel.
+The code that activates Vercel Analytics is located in the `src/pages/_app.tsx` file:
+
+```tsx
+const MyApp = ({ Component, emotionCache, pageProps }: MyAppProps) => <>
+  ...
+  {isVercelFromFrontend && <VercelAnalytics debug={false} />}
+  {isVercelFromFrontend && <VercelSpeedInsights debug={false} sampleRate={1 / 2} />}
+  ...
+</>;
+```
+
+When big-AGI is served on Vercel hosts, the ```process.env.NEXT_PUBLIC_VERCEL_URL``` environment variable is trueish, and
+analytics will be sent by default to the Vercel Analytics service which is deployed by Vercel IF configured from the
+Vercel project dashboard.
+
+In summary: to turn it on: activate the `Analytics` service in the Vercel project dashboard.
+
+### Google Analytics 4
+
+- Why: user engagement and retention, performance insights, personalization, content optimization
+- What: https://support.google.com/analytics/answer/11593727
+
+Google Analytics 4 (GA4) is a powerful tool for understanding user behavior and engagement.
+This can help optimize big-AGI, understanding which features are needed/users and which aren't.
+
+To enable Google Analytics 4, you need to set the `NEXT_PUBLIC_GA4_MEASUREMENT_ID` environment variable
+before starting the local build or the docker build (i.e. at build time), at which point the
+server/container will be able to report analytics to your Google Analytics 4 property.
+
+As of Feb 27, 2024, this feature is in development.
+
+## Configurations
+
+| Scope                                                                                   | Default          | Description / Instructions                                                                                              | 
+|-----------------------------------------------------------------------------------------|------------------|-------------------------------------------------------------------------------------------------------------------------|
+| Your source builds of big-AGI                                                           | None             | **Vercel**: enable Vercel Analytics from the dashboard. · **Google Analytics**: set environment variable at build time. |
+| Your docker builds of big-AGI                                                           | None             | **Vercel**: n/a. · **Google Analytics**: set environment variable at `docker build` time.                               |
+| [big-agi.com](https://big-agi.com)                                                      | Vercel + Google  | The main website ([privacy policy](https://big-agi.com/privacy)) hosted for free for anyone.                            |
+| [official Docker packages](https://github.com/enricoros/big-AGI/pkgs/container/big-agi) | Google Analytics | **Vercel**: n/a · **Google Analytics**: set to the big-agi.com Google Analytics for analytics and improvements.         |
+
+Note: this information is updated as of Feb 27, 2024 and can change at any time.

docs/environment-variables.md

@@ -28,6 +28,8 @@ AZURE_OPENAI_API_KEY=
 ANTHROPIC_API_KEY=
 ANTHROPIC_API_HOST=
 GEMINI_API_KEY=
+LOCALAI_API_HOST=
+LOCALAI_API_KEY=
 MISTRAL_API_KEY=
 OLLAMA_API_HOST=
 OPENROUTER_API_KEY=
@@ -55,15 +57,22 @@ BACKEND_ANALYTICS=
 # Backend HTTP Basic Authentication (see `deploy-authentication.md` for turning on authentication)
 HTTP_BASIC_AUTH_USERNAME=
 HTTP_BASIC_AUTH_PASSWORD=
+
+# Frontend variables
+NEXT_PUBLIC_GA4_MEASUREMENT_ID=
+NEXT_PUBLIC_PLANTUML_SERVER_URL=
 ```
 
-## Variables Documentation
+## Backend Variables
+
+These variables are used only by the server-side code, at runtime. Define them before running the nextjs local server (in development or
+cloud deployment), or pass them to Docker (--env-file or -e) when starting the container.
 
 ### Database
 
-For Database configuration see [deploy-database.md](deploy-database.md).
+To enable Chat Link Sharing, you need to connect the backend to a database. We currently support Postgres and MongoDB.
 
-To enable features such as Chat Link Sharing, you need to connect the backend to a database. We currently support Postgres and MongoDB.
+For Database configuration see [deploy-database.md](deploy-database.md).
 
 ### LLMs
 
@@ -80,13 +89,15 @@ requiring the user to enter an API key
 | `ANTHROPIC_API_KEY`         | The API key for Anthropic                                                                                                     | Optional                                                          |
 | `ANTHROPIC_API_HOST`        | Changes the backend host for the Anthropic vendor, to enable platforms such as [config-aws-bedrock.md](config-aws-bedrock.md) | Optional                                                          |
 | `GEMINI_API_KEY`            | The API key for Google AI's Gemini                                                                                            | Optional                                                          |
+| `LOCALAI_API_HOST`          | Sets the URL of the LocalAI server, or defaults to http://127.0.0.1:8080                                                      | Optional                                                          |  
+| `LOCALAI_API_KEY`           | The (Optional) API key for LocalAI                                                                                            | Optional                                                          |
 | `MISTRAL_API_KEY`           | The API key for Mistral                                                                                                       | Optional                                                          |
 | `OLLAMA_API_HOST`           | Changes the backend host for the Ollama vendor. See [config-local-ollama.md](config-local-ollama)                             |                                                                   |
 | `OPENROUTER_API_KEY`        | The API key for OpenRouter                                                                                                    | Optional                                                          |
 | `PERPLEXITY_API_KEY`        | The API key for Perplexity                                                                                                    | Optional                                                          |
 | `TOGETHERAI_API_KEY`        | The API key for Together AI                                                                                                   | Optional                                                          |
 
-### Model Observability: Helicone
+### LLM Observability: Helicone
 
 Helicone provides observability to your LLM calls. It is a paid service, with a generous free tier.
 It is currently supported for:
@@ -98,7 +109,7 @@ It is currently supported for:
 |--------------------|--------------------------|
 | `HELICONE_API_KEY` | The API key for Helicone |
 
-### Specials
+### Features
 
 Enable the app to Talk, Draw, and Google things up.
 
@@ -108,16 +119,31 @@ Enable the app to Talk, Draw, and Google things up.
 | `ELEVENLABS_API_KEY`       | ElevenLabs API Key - used for calls, etc.                                                                               |
 | `ELEVENLABS_API_HOST`      | Custom host for ElevenLabs                                                                                              |
 | `ELEVENLABS_VOICE_ID`      | Default voice ID for ElevenLabs                                                                                         |
+| **Text-To-Image**          | [Prodia](https://prodia.com/) is a reliable image generation service                                                    |
+| `PRODIA_API_KEY`           | Prodia API Key - used with '/imagine ...'                                                                               |
 | **Google Custom Search**   | [Google Programmable Search Engine](https://programmablesearchengine.google.com/about/)  produces links to pages        |
 | `GOOGLE_CLOUD_API_KEY`     | Google Cloud API Key, used with the '/react' command - [Link to GCP](https://console.cloud.google.com/apis/credentials) |
 | `GOOGLE_CSE_ID`            | Google Custom/Programmable Search Engine ID - [Link to PSE](https://programmablesearchengine.google.com/)               |
-| **Text-To-Image**          | [Prodia](https://prodia.com/) is a reliable image generation service                                                    |
-| `PRODIA_API_KEY`           | Prodia API Key - used with '/imagine ...'                                                                               |
 | **Browse**                 |                                                                                                                         |
-| `PUPPETEER_WSS_ENDPOINT`   | Puppeteer WebSocket endpoint - used for browsing, etc.                                                                  |
+| `PUPPETEER_WSS_ENDPOINT`   | Puppeteer WebSocket endpoint - used for browsing (pade downloadeing), etc.                                              |
 | **Backend**                |                                                                                                                         | 
 | `BACKEND_ANALYTICS`        | Semicolon-separated list of analytics flags (see backend.analytics.ts). Flags: `domain` logs the responding domain.     |
 | `HTTP_BASIC_AUTH_USERNAME` | See the [Authentication](deploy-authentication.md) guide. Username for HTTP Basic Authentication.                       |
 | `HTTP_BASIC_AUTH_PASSWORD` | Password for HTTP Basic Authentication.                                                                                 |
 
+### Frontend Variables
+
+The value of these variables are passed to the frontend (Web UI) - make sure they do not contain secrets.
+
+| Variable                          | Description                                                                              |
+|:----------------------------------|:-----------------------------------------------------------------------------------------|
+| `NEXT_PUBLIC_GA4_MEASUREMENT_ID`  | The measurement ID for Google Analytics 4. (see [deploy-analytics](deploy-analytics.md)) |
+| `NEXT_PUBLIC_PLANTUML_SERVER_URL` | The URL of the PlantUML server, used for rendering UML diagrams. (code in RederCode.tsx) |
+
+> Important: these variables must be set at build time, which is required by Next.js to pass them to the frontend.
+> This is in contrast to the backend variables, which can be set when starting the local server/container.
+
 ---
+
+For a higher level overview of backend code and environemnt customization,
+see the [big-AGI Customization](customizations.md) guide.