Firefox AI features (Summarize, etc.) does not work with llamafile

[TFWol]

Discussed in #633

^{Originally posted by TFWol November 19, 2024}
Windows 10 (not WSL)
I'm not entirely sure where the problem lies, but I haven't been able to get llamafiles to work with Firefox. It worked before.

A handful of separate commands I've tried running it with - all have the same outcome:

llamafile

@REM all commands were tried with various ports and IPs and changing corresponding ff about:config settings

llamafile.exe -m "modelname.llamafile.exe" --gpu nvidia --server --host 127.0.0.1 --port 8080 --nobrowser --temp 0 -ngl 999

llamafile.exe -m "modelname.llamafile" --recompile --gpu auto

llamafile.exe -m "modelname.llamafile" --server
llamafile.exe -m "modelname.llamafile.exe" --server
modelname.llamafile.exe --server

Firefox

Setting	Value	Additional Parameters
browser.ml.chat.enabled	true
browser.ml.chat.hideLocalhost	false
browser.ml.chat.prompt.prefix	{"l10nId":"genai-prompt-prefix-selection"}
browser.ml.chat.prompts.0	{"id":"summarize","l10nId":"genai-prompts-summarize"}
browser.ml.chat.prompts.1	{"id":"simplify","l10nId":"genai-prompts-simplify"}
browser.ml.chat.prompts.2	{"id":"quiz","l10nId":"genai-prompts-quiz","targeting":"!provider|regExpMatch('gemini')"}
browser.ml.chat.prompts.3	{"id":"explain","l10nId":"genai-prompts-explain","targeting":"channel=='nightly'"}
browser.ml.chat.provider	http://localhost:8080
browser.ml.chat.shortcuts	true
browser.ml.chat.shortcuts.custom	true
browser.ml.chat.shortcuts.longPress	60000
browser.ml.chat.sidebar	true
browser.ml.enable	true
browser.ml.logLevel	Error
browser.ml.modelCacheMaxSizeBytes	1073741824
browser.ml.modelCacheTimeout	120000
browser.ml.modelHubRootUrl	https://model-hub.mozilla.org/
browser.ml.modelHubUrlTemplate	{model}/{revision}

More info

• going directly to http://localhost:8080/ or http://127.0.0.1:8080/ in a browser tab and typing works, but not when using the firefox built-in ML stuff like 'Summarize'.

• Setting browser.ml.chat.sidebar to "False" and using firefox ml feature from context menu will open a tab with a url like http://localhost:8080/?q=I’m+on+page+“Test+Webpage”+with+“foobar”+selected.%0A%0APlease+summarize+the+selection+using+precise+and+concise+language.+Use+headers+and+bulleted+lists+in+the+summary%2C+to+make+it+scannable.+Maintain+the+meaning+and+factual+accuracy. and I can see firefox talk to llamafile server via the commandprompt window, but there won't be a response back from the llamafile server. Even typing in the chatbox and submitting it doesn't yield a response.

• In Dev Tools, I see a 404 in the POST:
  

[sizvix]

Yes , it's just a too simple "replace" insteed of use " URL() constructor " in the line https://github.com/Mozilla-Ocho/llamafile/blob/main/llama.cpp/server/public/index.html#L422
So it send "/?q=*******/completion" ...

[TFWol]

I got excited when I saw a new release...they didn't even bother to fix the issue.

I haven't really looked at what this "lamafiler" thing is yet, but I don't feel like going through more disappointment right now.

[pawel665j]

everything will be fine, thank you

…

________________________________ From: TFWol ***@***.***> Sent: Monday, December 2, 2024 12:53 PM To: Mozilla-Ocho/llamafile ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [Mozilla-Ocho/llamafile] Firefox AI features (Summarize, etc.) does not work with llamafile (Issue #640) I got excited when I saw a new release...they didn't even bother to fix the issue. I haven't really looked at what this "lamafiler" thing is yet, but I don't feel like going through more disappointment right now. — Reply to this email directly, view it on GitHub<#640 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOOFRCDKE2F2QBMFMGPUSRT2DQU2VAVCNFSM6AAAAABSNNGQ3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJRGA3DAOBUG4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[TFWol]

@sizvix would you happen to know how to use github Actions to build just the lamafile binary for placing into an artifact to download? I'm not familiar with builds that have a ton of imports like this project has.
I've tried and failed several times in the fork I copied with your modifications.