The Agent Framework is designed for building realtime, programmable participants that run on servers. Use it to create conversational, multi-modal voice agents that can see, hear, and understand.
This is a Node.js distribution of the LiveKit Agents framework, originally written in Python.
We're partnering with OpenAI on a new MultimodalAgent API in the Agents framework. This class completely wraps OpenAI’s Realtime API, abstract away the raw wire protocol, and provide an ultra-low latency WebRTC transport between GPT-4o and your users’ devices. This same stack powers Advanced Voice in the ChatGPT app.
- Try the Realtime API in our playground [code]
- Check out our guide to building your first app with this new API
Warning
This SDK is in Developer Preview. During this period, you may encounter bugs, and the APIs may change. Currently, only the OpenAI Realtime voice assistant is available as a plugin.
For production, we recommend using the more mature version of this framework, built with Python, which also supports other integrations.
We welcome and appreciate any feedback or contributions. You can create issues here or chat live with us in the LiveKit Community Slack.
To install the core Agents library:
pnpm install @livekit/agentsTo install the OpenAI plugin, for optional support for realtime voice assistants:
pnpm install @livekit/agents-plugin-openaiFirst, a few concepts:
- Agent: A function that defines the workflow of a programmable, server-side participant. This is your application code.
- Worker: A container process responsible for managing job queuing with LiveKit server. Each worker is capable of running multiple agents simultaneously.
- Plugin: A library class that performs a specific task, e.g. speech-to-text, from a specific provider. An agent can compose multiple plugins together to perform more complex tasks.
Your main file for an agent is built of two parts:
- The boilerplate code that runs when you run this file, creating a new worker to orchestrate jobs
- The code that is exported when this file is imported into Agents, to be ran on all jobs (which includes your entrypoint function, and an optional prewarm function)
Refer to the minimal voice assistant example to understand how to build a simple voice assistant with function calling using OpenAI's model.
The framework exposes a CLI interface to run your agent. To get started, you'll need the following environment variables set:
LIVEKIT_URLLIVEKIT_API_KEYLIVEKIT_API_SECRETOPENAI_API_KEY
The following command will start the worker and wait for users to connect to your LiveKit server:
node my_agent.js startTo run the worker in dev mode (outputting colourful pretty-printed debug logs), run it using dev:
node my_agent.js devTo ease the process of building and testing an agent, we've developed a versatile web frontend called "playground". You can use or modify this app to suit your specific requirements. It can also serve as a starting point for a completely custom agent application.
To join a LiveKit room that's already active, you can use the connect command:
node my_agent.ts connect --room <my-room>When you follow the steps above to run your agent, a worker is started that opens an authenticated
WebSocket connection to a LiveKit server instance(defined by your LIVEKIT_URL and authenticated
with an access token).
No agents are actually running at this point. Instead, the worker is waiting for LiveKit server to give it a job.
When a room is created, the server notifies one of the registered workers about a new job. The notified worker can decide whether or not to accept it. If the worker accepts the job, the worker will instantiate your agent as a participant and have it join the room where it can start subscribing to tracks. A worker can manage multiple agent instances simultaneously.
If a notified worker rejects the job or does not accept within a predetermined timeout period, the server will route the job request to another available worker.
The orchestration system was designed for production use cases. Unlike the typical web server, an agent is a stateful program, so it's important that a worker isn't terminated while active sessions are ongoing.
When calling SIGTERM on a worker, the worker will signal to LiveKit server that it no longer wants additional jobs. It will also auto-reject any new job requests that get through before the server signal is received. The worker will remain alive while it manages any agents connected to rooms.
This project is licensed under Apache-2.0, and is REUSE-3.2 compliant.
Refer to the license for details.
| LiveKit Ecosystem | |
|---|---|
| Realtime SDKs | Browser · iOS/macOS/visionOS · Android · Flutter · React Native · Rust · Node.js · Python · Unity · Unity (WebGL) |
| Server APIs | Node.js · Golang · Ruby · Java/Kotlin · Python · Rust · PHP (community) |
| UI Components | React · Android Compose · SwiftUI |
| Agents Frameworks | Python · Node.js · Playground |
| Services | LiveKit server · Egress · Ingress · SIP |
| Resources | Docs · Example apps · Cloud · Self-hosting · CLI |