-
-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Andoird libllama.so error #8
Comments
Heya, sorry to hear that. This is the file that the app seems to be missing on your device: https://github.com/android/src/main/jniLibs/arm64-v8a/libllama.so As seen from the file location, it is located in a folder named "arm64-v8a". This is not a made up folder name, it's a reserved name that specifies the CPU architecture of the device. If your device is running x86_64 for example, it will not be able to find the file. I'm pretty sure I can compile different libllama.so files for different architectures, I'd need to spend some time setting that up. To confirm my guess, could you let me know which device you have that is running Android? You may also tell me the CPU architecture if you can find that out, but otherwise just the device's name. Thanks! Let's see if we can solve this. |
I'm having the same issue. Your comment makes sense and i can confirm my emulator on my windows PC is configured to run with the system image x86_64. The other images (arm64-v8a) are not supported to run on my PC - at least not past API 27 - Oreo. Would be happy if it could run on x86_64 :) |
Thanks for sharing your insights. So, to clarify this for myself: Your emulator is running Android (x86_64) if I understood that correctly, and you're facing this same issue. Thus, likely facing the issue highlighted in my previous comment. If so, that's good. Means I could verify that easily and fix it. I've just moved to a different country and took a break from development. I'll try to replicate this the same way (emulator) & fix this issue before the end of this week. |
Yes when you set up an emulator you have to choose a system image. There I had to choose some Android API with x86_64 (since arm64-v8a doesn't work). I assume this means the android emulator runs on that architecture and that I am indeed experiencing the issue you described. I assume it's due to the check you have in your I'm trying to get a local LLM on device and up and running by Tuesday - so it would be great if it gets fixed by the end of the week. Thanks for providing this lib 👍🏻 |
Ah, got it. No worries, I'll get it done this week. I've just put together a new workstation and am tinkering with VM's/containers for fun (and CI/CD stuff). Nonetheless, I planned to work on this repo somewhere in the upcoming days. It'll probably take me a day or two fixing/updating all open issues. |
@escottgoodwin @d-kls can you pull the latest changes and give it a try? I have included Android x86_64 support and synced upstream with llama.cpp. Let me know if you're still running into an issue. |
@BrutalCoding Now when im trying to build i get the following error (user specific paths replaced with <path>):
Some additional information from
|
@d-kls I see a few lines about CMake requiring a higher version than what you have installed locally. I see you got 3.18. Nonetheless, annoying experience for sure. I did not expect that my changes would cause this. CMake seems to require you to run 3.19 or higher, can you try installing the latest version of CMake and try again? I'm unable to continue development today. Would it help if I compile and share an APK tomorrow that supports x86_64? That way, you can still show it off on your device. This issue should still be fixed of course, which I hope is as simple as installing the latest CMake for you. Let me know if you want me to do produce that APK, and it will be the first thing I'd do tomorrow morning. Please try installing the latest version of CMake first, hopefully that'll work. Cheers, |
@BrutalCoding Yes i did notice that too. However it seems that you have a specific ndk version (23.1.7779620) specified, which does not allow me to up the ndk. It seems when the ndk is not updated, the cmake version defaults to 3.18. This is the output i get when im trying to build the application, having removed cmake 3.18 and ndk 23..while having installed ndk 26.. and cmake 3.22
When i uninstall both of them, configure android to use the newest versions, i get the following error:
Also i appreciate your offer, really, but it is no use for me in another app. i have to integrate it into my current project. I still have the whole day tomorrow to get it done somehow.. |
@BrutalCoding could you provide the shared object file |
I've provided 2 different .so's, it should automatically be picked up when you built the app with Flutter. Here's the x86_64 artifact (notice the x86_64 dir name): To confirm the architecture, at least on a Mac, you can run: Sorry, I'm unable to test this on x86_64 without asking someone to borrow their laptop with that architecture. I've been struggling to get x86_64 running in my VM, my main machine is my MBP M1 (ARM) so I went on spending hours setting up a Windows env on another machine of mine running Proxmox (a hypervisor) -> installed Windows with all the Flutter/Android tooling successfully but cant get the Android emulator to boot up. Stuck at the boot. Too long to explain, but it has probably something to do with nested virtualizations which my CPU, BIOS etc does support and is enabled but yeah - I gave up after half a day. I'll be setting up a dual-boot with Windows sometime today and install all required stuff again (IDE/Flutter/Android x86_64 SDK etc). Will let you know if I can confirm it running. |
@BrutalCoding This is indeed the one i have been using. As i mentioned i was using the one from your repo for x86_64. But when i use yours, it throws an error saying that while it is indeed built for x86_64, it is more specifically built for AARCH_x86_64 and not EM_x86_64 i believe.. |
@BrutalCoding Ok i have built the correct
This is also consistent with the experience of using a physical phone that has the arm64-v8a I have tracked this down to your bindings and the specific call that i think might cause the issue. In the llama_context_params llama_context_default_params() {
log("call to _llama_context_default_params");
log(_llama_context_default_params().toString());
return _llama_context_default_params();
} It logs the first message fine. But the second one (or ones in the calling method outside this file after) are not called, meaning the segfault (likely) happens in this call to |
@d-kls I have just updated the Dart bindings, but I haven't been able to built the binaries for Android x86_64 while I was attempting to build it on my new Windows setup. I ran into some issues I need to resolve. But since you got the correct .so binaries, plus combined with the recent change I pushed to update llama.cpp bindings, give it another try. Thanks for the work you put into this issue, once again. -- Edit -- > objdump -x libllama_elf.so | grep "file format"
libllama_elf.so: file format elf64-x86-64 This binary was built on a x86_64 Linux machine just to be extra sure, previously I cross-compiled it on my M1 Mac. Nonetheless, both machines seemed to produce the right binaries. That context binding is certainly still valid, you're likely running into an issue with loading the binary file. Can you give me some specifics of your host device and the exact Android x86_64 device that you're trying to compile for? |
@BrutalCoding Ok awesome it seems to have been a bindings issue. I don't get a segfault anymore when calling default params. Thanks for updating. Right now im using your released version (on pub) with the bindings of the main branched locally patched. Since i had the other issue with NDK and CMake with the commits before the newest ones. I'll have to test loading a 1.2gb model now from my assets, this seems to not be easy with flutter limiting the size of About the binaries: Im definitely not saying that they are not x86_64. Im saying i had the error mentioning that they are for AARCH_x86_64 and not EM_x86_64 when using them. That was the exact error it was telling me when it was trying to open them. My emulator is a Google Pixel 6 Pro with Android API 33 and the corresponding x86_64 image. |
Cool, glad you hacked it together to make it work. It's annoying that it didn't work for you the first time (That's on me). With my initial project, shady.ai, that's exactly how I started but quickly found out that it's the wrong tool for the job. It sounds very reasonable to add a model as an asset to your app but there are several reasons on why not. This is from memory so take this with a grain of salt, I might be wrong: Reasons to avoid assets folder
Anyways, I did get it to work with tiny models (tiny stories series 100m etc, not 1B, 7B etc). But I had to read it in memory, convert the bytes, write it back to a temp (or support) directory, and then finally I was able to pass the path to that converted model to llama.cpp. How to proceed without assetsThere are 2 solutions:
Have a look at:
I initially wanted to go this way (downloading models in-app), because that's the only way I see it for the average Joe. However, I also needed a way to test new models. Hence why I went with a file picker approach first, more dev friendly but absolutely not user friendly haha, I'm very aware of that hahaha. Hope that this helped you in some way. PS. The example app will include models available to download automatically, but I'm unable to estimate when I can can prioritize that. Should be fairly easy to implement, I'd reckon that it takes a novice 1-3 hours. |
@BrutalCoding Yeah that's completely reasonable. When i opt to load it from the file picker (which is fine for a demo, i think the other option is more sustainable) then it works quite fine besides the fact that there is always an empty response using the model phi-2.Q4_K_M.gguf. Do you have any idea why that could be? There is no error thrown or logged. Just start and then end token and it's over. Remember im using the version on pub for the |
I remember having that issue yeah, but I took care of it. I'll publish a new version to pub today, that should resolve that issue and bring some more improvements. --Edit-- I'll sync the package on pub.dev this weekend. |
Hello, |
Hi, @BrutalCoding, I have the same issue with not finding libllama.so, I am using your latest version from pub.dev |
Ran the app on an android device. It seemed to cmake build the llama.cpp. Downloaded GGUF file and loaded it.
When I asked a question I got:
ArgumentError (Invalid argument(s): Failed to load dynamic library 'libllama.so': dlopen failed: library "libllama.so" not found)
The text was updated successfully, but these errors were encountered: