-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Browser AI API for Utilizing On-Device Models #178
Comments
(Meta comment: Are you aware of the Prompt API proposal?) |
Yes, I saw those, and have worked with the prompt api, which is what inspired this proposal. I see this as an alternative approach to implementing AI APIs, one that makes them more open and flexible. This proposal isn’t about focusing on specific tasks like prompt generation or summarization. Instead, it’s about creating a bridge to a model runtime that gives access to models suited to each specific use case. I think building separate APIs for each task—like prompt-specific or translation-specific APIs—is the wrong direction. It’s better to have an API that lets the developer or user choose the model with the right capabilities for their needs. |
(Great, was just wondering, since your proposal didn't mention these previous efforts.) |
Thanks for the feedback. I updated the description to include "Other Considerations" which discusses this. |
@kenzic, hello. I have two quick feedback items. Firstly, your proposed API includes models' names and versions together in // Request permission for a specific model
const session = await window.ai.model.connect({
model: "llama", version: "3.2"
}); Or, perhaps, adding more parameters? // Request permission for a specific model
const session = await window.ai.model.connect({
model: "llama", version: "3.2", size: "11B", lang: "en", publicKeyToken: "..."
}); Secondly, it appears that your API allows iterating available models: // Get list of available models
const models = await window.ai.permissions.models(); and that the Prompt API presently doesn't. I think that this feature would be useful for an eventual API to enable content negotiation between computers. A client might have multiple models available with which to, for instance, create an embedding vector, and a server might recognize embedding vectors from multiple models. The two computers could perform content negotiation to select which available model to make use of to generate the embedding vector with. Did I explain that well? |
Hi @AdamSobieski - Thanks for the feedback. I agree that having a version parameter could be helpful. The reason I didn't include it is because most of the providers, such as openai, ollama, anthropic include the version in the name. For example,
From what I understand, you're saying this API allows the website accessing the api to see which models have been made available to it by the browser, which is helpful. This is something the Prompt API doesn't do, and would be helpful to add to it as well? |
Benefits of separating out the version for these purposes include that the semantics of connecting to a model without providing an explicit version could be such as to request the newest, or latest, version available. So: // Request permission for a specific model
const session = await window.ai.model.connect({
model: "llama"
}); could describe a request to connect to the highest version of However, there are also the numbers of model parameters to consider. With respect to
Yes. Thank you. In my opinion, being able to list the available models would be useful – both those locally available on a client device and those available to it per downloading. I'm trying to envision how two computers might negotiate which model or models to utilize when each could make use of more than one model (e.g., to exchange embedding vectors). It could be that one party, e.g., a server, would provide a set of content-negotiation headers with values describing model aspects (like the |
You raise some good points about the versions and sizes. I'd like to keep the api as consistent with other providers as possible. Let me think about it. Regarding negotiating models with prompt api: I don't manage that spec, but I'd recommend you reach out them. It's a great idea, and one I thought important to add to this spec. To do it with this implementation you'd do something like: // Get list of available models
const models = await window.ai.permissions.models();
if (models.includes('llama3.2')) {
// Request permission for a specific model
const granted = await window.ai.permissions.request({
model: "llama3.2"
});
if (!granted) {
// handle
} else {
const session = await window.ai.model.connect({
model: "llama3.2"
});
}
} |
Introduction
The Browser AI API is a proposal for a new browser feature that makes AI models accessible directly on users' devices through the browser. By offering a simple API available on the
window
object, this approach would allow websites to leverage AI without sending data to the cloud, preserving privacy, reducing latency, and enabling offline functionality.This is about empowering developers to integrate advanced AI into web apps—without needing heavy infrastructure—while giving users control over their data and the models they choose to run. Imagine a world where on-device AI enhances web apps in real-time, with no data leaving the device and no reliance on external servers.
The API would let developers:
By running models directly on the user's hardware, we’re opening up new possibilities for AI-driven web apps while keeping things secure, private, and available offline.
Prototype
I created a prototype of this concept for review: https://github.com/kenzic/browser.ai
API Overview
The proposed API would be exposed on the
window.ai
object with the following high-level structure:Permissions
Before using any models, websites must first query available models and request permission:
Model Sessions
Once permission is granted, websites can create sessions to interact with models:
Chat
Embed
WebIDL
Key Benefits
Use Cases
This API opens the door for all kinds of innovative web applications:
Technical Considerations
Other Considerations
There are similar proposals, such as Prompt API proposal
I see this as an alternative approach to implementing AI APIs, one that makes them more open and flexible. This proposal isn’t about focusing on specific tasks like prompt generation or summarization. Instead, it’s about creating a bridge to a model runtime that gives access to models suited to each specific use case. I think building separate APIs for each task—like prompt-specific or translation-specific APIs—is the wrong direction. It’s better to have an API that lets the developer or user choose the model with the right capabilities for their needs.
Feedback
Please provide all feedback below.
The text was updated successfully, but these errors were encountered: