Andoird libllama.so error about aub.ai HOT 18 OPEN

Yes when you set up an emulator you have to choose a system image. There I had to choose some Android API with x86_64 (since arm64-v8a doesn't work). I assume this means the android emulator runs on that architecture and that I am indeed experiencing the issue you described.

I assume it's due to the check you have in your _dylib function. Where the platform is indeed Android but it is not ARM. https://stackoverflow.com/a/75959707 - perhaps this answer can help with that.

I'm trying to get a local LLM on device and up and running by Tuesday - so it would be great if it gets fixed by the end of the week. Thanks for providing this lib 👍🏻

Ah, got it. No worries, I'll get it done this week.

I've just put together a new workstation and am tinkering with VM's/containers for fun (and CI/CD stuff).

Nonetheless, I planned to work on this repo somewhere in the upcoming days. It'll probably take me a day or two fixing/updating all open issues.

from aub.ai.

@BrutalCoding Yeah that's completely reasonable. When i opt to load it from the file picker (which is fine for a demo, i think the other option is more sustainable) then it works quite fine besides the fact that there is always an empty response using the model phi-2.Q4_K_M.gguf. Do you have any idea why that could be? There is no error thrown or logged. Just start and then end token and it's over. Remember im using the version on pub for the aub_ai.dart file.

I remember having that issue yeah, but I took care of it. I'll publish a new version to pub today, that should resolve that issue and bring some more improvements.

--Edit--
Couldn't stick to my words to publish a fix to pub.dev yesterday, my apologies. Kinda overwhelmed with a different project that I'd like to add here too.

I'll sync the package on pub.dev this weekend.

from aub.ai.

Heya, sorry to hear that.

This is the file that the app seems to be missing on your device:

https://github.com/android/src/main/jniLibs/arm64-v8a/libllama.so

As seen from the file location, it is located in a folder named "arm64-v8a". This is not a made up folder name, it's a reserved name that specifies the CPU architecture of the device. If your device is running x86_64 for example, it will not be able to find the file.

I'm pretty sure I can compile different libllama.so files for different architectures, I'd need to spend some time setting that up.

To confirm my guess, could you let me know which device you have that is running Android? You may also tell me the CPU architecture if you can find that out, but otherwise just the device's name.

Thanks! Let's see if we can solve this.

from aub.ai.

I'm having the same issue. Your comment makes sense and i can confirm my emulator on my windows PC is configured to run with the system image x86_64. The other images (arm64-v8a) are not supported to run on my PC - at least not past API 27 - Oreo. Would be happy if it could run on x86_64 :)

from aub.ai.

I'm having the same issue. Your comment makes sense and i can confirm my emulator on my windows PC is configured to run with the system image x86_64. The other images (arm64-v8a) are not supported to run on my PC - at least not past API 27 - Oreo. Would be happy if it could run on x86_64 :)

Thanks for sharing your insights.

So, to clarify this for myself: Your emulator is running Android (x86_64) if I understood that correctly, and you're facing this same issue. Thus, likely facing the issue highlighted in my previous comment. If so, that's good. Means I could verify that easily and fix it.

I've just moved to a different country and took a break from development. I'll try to replicate this the same way (emulator) & fix this issue before the end of this week.

from aub.ai.

Yes when you set up an emulator you have to choose a system image. There I had to choose some Android API with x86_64 (since arm64-v8a doesn't work). I assume this means the android emulator runs on that architecture and that I am indeed experiencing the issue you described.

I assume it's due to the check you have in your _dylib function. Where the platform is indeed Android but it is not ARM. https://stackoverflow.com/a/75959707 - perhaps this answer can help with that.

I'm trying to get a local LLM on device and up and running by Tuesday - so it would be great if it gets fixed by the end of the week. Thanks for providing this lib 👍🏻

from aub.ai.

@escottgoodwin @d-kls can you pull the latest changes and give it a try?

I have included Android x86_64 support and synced upstream with llama.cpp.

Let me know if you're still running into an issue.

from aub.ai.

@BrutalCoding Now when im trying to build i get the following error (user specific paths replaced with <path>):

Execution failed for task ':aub_ai:configureCMakeRelWithDebInfo[arm64-v8a]'.
error when building with cmake using <path>\git\aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e\src\CMakeLists.txt:
Build command failed.


Error while executing process <path>\Android\Sdk\cmake\3.18.1\bin\cmake.exe with arguments {-H<path>\git\aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e\src -DCMAKE_SYSTEM_NAME=Android -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_SYSTEM_VERSION=28 -DANDROID_PLATFORM=android-28 -DANDROID_ABI=arm64-v8a -DCMAKE_ANDROID_ARCH_ABI=arm64-v8a -DANDROID_NDK=<path>\Android\Sdk\ndk\23.1.7779620 -DCMAKE_ANDROID_NDK=<path>\Android\Sdk\ndk\23.1.7779620 -DCMAKE_TOOLCHAIN_FILE=<path>\Android\Sdk\ndk\23.1.7779620\build\cmake\android.toolchain.cmake -DCMAKE_MAKE_PROGRAM=<path>\Android\Sdk\cmake\3.18.1\bin\ninja.exe -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=<path>\build\aub_ai\intermediates\cxx\RelWithDebInfo\58506m46\obj\arm64-v8a -DCMAKE_RUNTIME_OUTPUT_DIRECTORY=<path>\build\aub_ai\intermediates\cxx\RelWithDebInfo\58506m46\obj\arm64-v8a -DCMAKE_BUILD_TYPE=RelWithDebInfo -B<path>\git\aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e\android\.cxx\RelWithDebInfo\58506m46\arm64-v8a -GNinja}
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: <path>/Android/Sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/windows-x86_64/bin/clang.exe - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: <path>/Android/Sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/windows-x86_64/bin/clang++.exe - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Configuring incomplete, errors occurred!
  See also "<path>/aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e/android/.cxx/RelWithDebInfo/58506m46/arm64-v8a/CMakeFiles/CMakeOutput.log".

  CMake Warning at <path>/Android/Sdk/ndk/23.1.7779620/build/cmake/android-legacy.toolchain.cmake:416 (message):
    An old version of CMake is being used that cannot automatically detect
    compiler attributes.  Compiler identification is being bypassed.  Some
    values may be wrong or missing.  Update to CMake 3.19 or newer to use
    CMake's built-in compiler identification.
  Call Stack (most recent call first):
    <path>/Android/Sdk/ndk/23.1.7779620/build/cmake/android.toolchain.cmake:55 (include)
    <path>/Android/Sdk/cmake/3.18.1/share/cmake-3.18/Modules/CMakeDetermineSystem.cmake:93 (include)
    CMakeLists.txt:6 (project)


  CMake Warning at <path>/Android/Sdk/ndk/23.1.7779620/build/cmake/android-legacy.toolchain.cmake:416 (message):
    An old version of CMake is being used that cannot automatically detect
    compiler attributes.  Compiler identification is being bypassed.  Some
    values may be wrong or missing.  Update to CMake 3.19 or newer to use
    CMake's built-in compiler identification.
  Call Stack (most recent call first):
    <path>/Android/Sdk/ndk/23.1.7779620/build/cmake/android.toolchain.cmake:55 (include)
    <path>/aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e/android/.cxx/RelWithDebInfo/58506m46/arm64-v8a/CMakeFiles/3.18.1-g262b901-dirty/CMakeSystem.cmake:6 (include)
    <path>aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e/android/.cxx/RelWithDebInfo/58506m46/arm64-v8a/CMakeFiles/CMakeTmp/CMakeLists.txt:2 (project)


  CMake Warning at <path>/Android/Sdk/ndk/23.1.7779620/build/cmake/android-legacy.toolchain.cmake:416 (message):
    An old version of CMake is being used that cannot automatically detect
    compiler attributes.  Compiler identification is being bypassed.  Some
    values may be wrong or missing.  Update to CMake 3.19 or newer to use
    CMake's built-in compiler identification.
  Call Stack (most recent call first):
    <path>/Android/Sdk/ndk/23.1.7779620/build/cmake/android.toolchain.cmake:55 (include)
    <path>/git/aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e/android/.cxx/RelWithDebInfo/58506m46/arm64-v8a/CMakeFiles/3.18.1-g262b901-dirty/CMakeSystem.cmake:6 (include)
   <path>/aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e/android/.cxx/RelWithDebInfo/58506m46/arm64-v8a/CMakeFiles/CMakeTmp/CMakeLists.txt:2 (project)


  CMake Error at CMakeLists.txt:9 (add_subdirectory):
    The source directory

     <path>/aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e/src/llama.cpp

    does not contain a CMakeLists.txt file.

Some additional information from CMakeOutput.log

The target system is: Android - 1 - aarch64
The host system is: Windows - 10.0.22621 - AMD64
Detecting C compiler ABI info compiled with the following output:
Change Dir:<path>/git/aub.ai-5c95b9874b4cbf980e763d950e2a86d19934499e/android/.cxx/Debug/255v1n4m/arm64-v8a/CMakeFiles/CMakeTmp

Run Build Command(s):<path>\Android\Sdk\cmake\3.18.1\bin\ninja.exe cmTC_04866 && [1/2] Building C object CMakeFiles/cmTC_04866.dir/CMakeCCompilerABI.c.o
Android (7714059, based on r416183c1) clang version 12.0.8 (https://android.googlesource.com/toolchain/llvm-project c935d99d7cf2016289302412d708641d52d2f7ee)
Target: aarch64-none-linux-android28
Thread model: posix
InstalledDir: <path>\Android\Sdk\ndk\23.1.7779620\toolchains\llvm\prebuilt\windows-x86_64\bin
Found candidate GCC installation: <path>/Android/Sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/windows-x86_64/lib/gcc/aarch64-linux-android\4.9.x
Selected GCC installation: <path>/Android/Sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/windows-x86_64/lib/gcc/aarch64-linux-android/4.9.x
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: <path>\NVIDIA GPU Computing Toolkit, version 11.0

from aub.ai.

@d-kls I see a few lines about CMake requiring a higher version than what you have installed locally. I see you got 3.18.

Nonetheless, annoying experience for sure. I did not expect that my changes would cause this. CMake seems to require you to run 3.19 or higher, can you try installing the latest version of CMake and try again?

I'm unable to continue development today. Would it help if I compile and share an APK tomorrow that supports x86_64? That way, you can still show it off on your device.

This issue should still be fixed of course, which I hope is as simple as installing the latest CMake for you.

Let me know if you want me to do produce that APK, and it will be the first thing I'd do tomorrow morning. Please try installing the latest version of CMake first, hopefully that'll work.

Cheers,
Daniel

from aub.ai.

@BrutalCoding Yes i did notice that too. However it seems that you have a specific ndk version (23.1.7779620) specified, which does not allow me to up the ndk. It seems when the ndk is not updated, the cmake version defaults to 3.18. This is the output i get when im trying to build the application, having removed cmake 3.18 and ndk 23..while having installed ndk 26.. and cmake 3.22

Checking the license for package NDK (Side by side) 23.1.7779620 in <path>\Android\Sdk\licenses
License for package NDK (Side by side) 23.1.7779620 accepted.
Preparing "Install NDK (Side by side) 23.1.7779620 (revision: 23.1.7779620)".

...

Checking the license for package CMake 3.18.1 in <path>\Android\Sdk\licenses
License for package CMake 3.18.1 accepted.
Preparing "Install CMake 3.18.1 (revision: 3.18.1)".
"Install CMake 3.18.1 (revision: 3.18.1)" ready.

When i uninstall both of them, configure android to use the newest versions, i get the following error:

A problem occurred configuring project ':aub_ai'.
> [CXX1104] NDK from ndk.dir at <path>\Android\Sdk\ndk\26.1.10909125 had version [26.1.10909125] which disagrees with android.ndkVersion [23.1.7779620]

Also i appreciate your offer, really, but it is no use for me in another app. i have to integrate it into my current project. I still have the whole day tomorrow to get it done somehow..

from aub.ai.

@BrutalCoding could you provide the shared object file libllama.so compiled for EM_X86_64? The one in your repo seems to be for AARCH_64?

from aub.ai.

@BrutalCoding could you provide the shared object file libllama.so compiled for EM_X86_64? The one in your repo seems to be for AARCH_64?

I've provided 2 different .so's, it should automatically be picked up when you built the app with Flutter. Here's the x86_64 artifact (notice the x86_64 dir name):
https://github.com/BrutalCoding/aub.ai/blob/5c95b9874b4cbf980e763d950e2a86d19934499e/android/src/main/jniLibs/x86_64/libllama.so

To confirm the architecture, at least on a Mac, you can run:
file -b libllama.so
Which returns:
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=82464acea604d70315ee7e739221a057ed057b6b, with debug_info, not stripped

Sorry, I'm unable to test this on x86_64 without asking someone to borrow their laptop with that architecture. I've been struggling to get x86_64 running in my VM, my main machine is my MBP M1 (ARM) so I went on spending hours setting up a Windows env on another machine of mine running Proxmox (a hypervisor) -> installed Windows with all the Flutter/Android tooling successfully but cant get the Android emulator to boot up. Stuck at the boot. Too long to explain, but it has probably something to do with nested virtualizations which my CPU, BIOS etc does support and is enabled but yeah - I gave up after half a day.

I'll be setting up a dual-boot with Windows sometime today and install all required stuff again (IDE/Flutter/Android x86_64 SDK etc). Will let you know if I can confirm it running.

from aub.ai.

@BrutalCoding This is indeed the one i have been using. As i mentioned i was using the one from your repo for x86_64. But when i use yours, it throws an error saying that while it is indeed built for x86_64, it is more specifically built for AARCH_x86_64 and not EM_x86_64 i believe..

from aub.ai.

@BrutalCoding Ok i have built the correct libllama.so and libggml-shared.so by myself. It now successfully opens the Dynamic Library. However i now get a segfault (i believe) and the app crashes:

F/libc    (15846): Fatal signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 0x0 in tid 15994 (DartWorker), pid 15846

This is also consistent with the experience of using a physical phone that has the arm64-v8a .so files you generated. There in the release version it crashes unexpectedly (i wasn't able to reproduce this before with my emulator but now have the same issue with correct .so files). Could this be due to mismatched bindings? I built the .so files out of version b1892 of the llama.cpp project, but the bindings i assume are not up to date?

I have tracked this down to your bindings and the specific call that i think might cause the issue. In the aub_ai_bindings_generated.dart i have the following code:

llama_context_params llama_context_default_params() {
    log("call to _llama_context_default_params");
    log(_llama_context_default_params().toString());
    return _llama_context_default_params();
}

It logs the first message fine. But the second one (or ones in the calling method outside this file after) are not called, meaning the segfault (likely) happens in this call to _llama_context_default_params. And i dont reach beyond here (with or without the second log line)

from aub.ai.

@BrutalCoding Ok i have built the correct libllama.so and libggml-shared.so by myself. It now successfully opens the Dynamic Library. However i now get a segfault (i believe) and the app crashes:

F/libc    (15846): Fatal signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 0x0 in tid 15994 (DartWorker), pid 15846

This is also consistent with the experience of using a physical phone that has the arm64-v8a .so files you generated. There in the release version it crashes unexpectedly (i wasn't able to reproduce this before with my emulator but now have the same issue with correct .so files). Could this be due to mismatched bindings? I built the .so files out of version b1892 of the llama.cpp project, but the bindings i assume are not up to date?

I have tracked this down to your bindings and the specific call that i think might cause the issue. In the aub_ai_bindings_generated.dart i have the following code:

llama_context_params llama_context_default_params() {
    log("call to _llama_context_default_params");
    log(_llama_context_default_params().toString());
    return _llama_context_default_params();
}

It logs the first message fine. But the second one (or ones in the calling method outside this file after) are not called, meaning the segfault (likely) happens in this call to _llama_context_default_params. And i dont reach beyond here (with or without the second log line)

@d-kls I have just updated the Dart bindings, but I haven't been able to built the binaries for Android x86_64 while I was attempting to build it on my new Windows setup. I ran into some issues I need to resolve.

But since you got the correct .so binaries, plus combined with the recent change I pushed to update llama.cpp bindings, give it another try.

Thanks for the work you put into this issue, once again.

-- Edit --
I just confirmed that the (previous and current) Android x86_64 binary (.so) file are built for the right architecture which can be tested this way:

> objdump -x libllama_elf.so | grep "file format"
libllama_elf.so:	file format elf64-x86-64

This binary was built on a x86_64 Linux machine just to be extra sure, previously I cross-compiled it on my M1 Mac. Nonetheless, both machines seemed to produce the right binaries.

That context binding is certainly still valid, you're likely running into an issue with loading the binary file.

Can you give me some specifics of your host device and the exact Android x86_64 device that you're trying to compile for?

from aub.ai.

@BrutalCoding Ok awesome it seems to have been a bindings issue. I don't get a segfault anymore when calling default params. Thanks for updating. Right now im using your released version (on pub) with the bindings of the main branched locally patched. Since i had the other issue with NDK and CMake with the commits before the newest ones. I'll have to test loading a 1.2gb model now from my assets, this seems to not be easy with flutter limiting the size of rootBundle.load(...) to 1gb. Im trying to bundle the model with the app, not have the user filepick the model themselves. Is this something you considered when coding the way the model is loaded?

About the binaries: Im definitely not saying that they are not x86_64. Im saying i had the error mentioning that they are for AARCH_x86_64 and not EM_x86_64 when using them. That was the exact error it was telling me when it was trying to open them.

My emulator is a Google Pixel 6 Pro with Android API 33 and the corresponding x86_64 image.

from aub.ai.

@BrutalCoding Ok awesome it seems to have been a bindings issue. I don't get a segfault anymore when calling default params. Thanks for updating. Right now im using your released version (on pub) with the bindings of the main branched locally patched. Since i had the other issue with NDK and CMake with the commits before the newest ones. I'll have to test loading a 1.2gb model now from my assets, this seems to not be easy with flutter limiting the size of rootBundle.load(...) to 1gb. Im trying to bundle the model with the app, not have the user filepick the model themselves. Is this something you considered when coding the way the model is loaded?

About the binaries: Im definitely not saying that they are not x86_64. Im saying i had the error mentioning that they are for AARCH_x86_64 and not EM_x86_64 when using them. That was the exact error it was telling me when it was trying to open them.

My emulator is a Google Pixel 6 Pro with Android API 33 and the corresponding x86_64 image.

Cool, glad you hacked it together to make it work. It's annoying that it didn't work for you the first time (That's on me).

With my initial project, shady.ai, that's exactly how I started but quickly found out that it's the wrong tool for the job.

It sounds very reasonable to add a model as an asset to your app but there are several reasons on why not. This is from memory so take this with a grain of salt, I might be wrong:

Reasons to avoid assets folder

1. If you plan to release the app on official app stores, they impose max file size limits. On Play Store, it's <= 200mb.

2. Not too sure about this but I believe that when you load an asset (e.g. 1gb model in assets dir) it will first be copied in memory (1gb+ RAM), and then written in a temp directory (aka app cache). It's ineffecient, slow and is happening on the main thread where Flutter (UI) is running too. Thus, app will be frozen for a while until it's done.

3. Compile time increases because the debugger is copying over all assets to your device (including to a emulator/simulator). Trust me, it's not fun.

4. I think that my debugger kept crashing / disconnecting when I had a model that was only a few hundred MB's (certainly a 1GB model).

Anyways, I did get it to work with tiny models (tiny stories series 100m etc, not 1B, 7B etc). But I had to read it in memory, convert the bytes, write it back to a temp (or support) directory, and then finally I was able to pass the path to that converted model to llama.cpp.

How to proceed without assets

There are 2 solutions:

1. Like the majority of games in the app stores -> download the assets on-demand. Could be done in background I suppose, but each OS has its limitations so be aware.

2. Do like I did, let the user pick a file.

Have a look at:

• https://developer.android.com/google/play/expansion-files
• https://help.apple.com/xcode/mac/current/#/devbbdc5ce4f (In particular: "On-Demand Resources (iOS, tvOS)", they mention that Apple will host these resources on their servers)
• https://developer.apple.com/help/app-store-connect/reference/maximum-build-file-sizes/ (Note, it mentions a max limit of 500 MB for the app size)

I initially wanted to go this way (downloading models in-app), because that's the only way I see it for the average Joe. However, I also needed a way to test new models. Hence why I went with a file picker approach first, more dev friendly but absolutely not user friendly haha, I'm very aware of that hahaha.

Hope that this helped you in some way.

PS. The example app will include models available to download automatically, but I'm unable to estimate when I can can prioritize that. Should be fairly easy to implement, I'd reckon that it takes a novice 1-3 hours.

from aub.ai.

@BrutalCoding Yeah that's completely reasonable. When i opt to load it from the file picker (which is fine for a demo, i think the other option is more sustainable) then it works quite fine besides the fact that there is always an empty response using the model phi-2.Q4_K_M.gguf. Do you have any idea why that could be? There is no error thrown or logged. Just start and then end token and it's over. Remember im using the version on pub for the aub_ai.dart file.

from aub.ai.