Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: cannot build ios loadable extension #190

Closed
iliapnmrv opened this issue Nov 28, 2024 · 11 comments
Closed

[Bug]: cannot build ios loadable extension #190

iliapnmrv opened this issue Nov 28, 2024 · 11 comments

Comments

@iliapnmrv
Copy link

What happened?

I want to add tokenizer for my db (I know that now I can create a custom tokenizer and it works perfectly for small custom tokenizers).
I'm trying to use sqlite-better-trigram. For android I've just downloaded a .so files from artefacts for each architecture

But I cannot make ios work
I'm following this apple guide

  1. By default library compiles to .dylib, so I've created a fork that compiles also to .a
  2. I've created a universal .a file using lipo to combine iossimulator architectures
lipo -create -output better-trigram-universal.a sqlite-better-trigram-iossimulator-aarch64-extension/better-trigram.a sqlite-better-trigram-iossimulator-x86_64-extension/better-trigram.a
  1. I'm using xcodebuild -create-xcframework with - library option
xcodebuild -create-xcframework
    -library sqlite-better-trigram-ios-aarch64-extension/better-trigram.a -headers better-trigram.h
    -library better-trigram-universal.a -headers better-trigram.h
    -output xcframeworks/better-trigram.xcframework
  1. Then I'm trying to import better-trigram.xcframework into project
    I've tried different ways but overall my file structure followed the one in the guide
/ios
  /your_lib.xcframework
    /device_framework <-- ios-arm64
      /Headers
        your_lib.h
      your_lib
    /sim_framework <-- ios-arm64_x86_64-simulator
      /Headers <-- in my case I only have `Headers` file
        your_lib.h
      your_lib <-- I have correct name but my file has `.a` extension
  Info.plist
  1. Ios builds, but then I have an error
'Error: Exception in HostFunction: dlopen(better-trigram.dylib, 0x000A): tried: \'/usr/lib/system/introspection/better-trigram.dylib\' (no such file, not in dyld cache), \'better-trigram.dylib\' (no such file), \'/private/preboot/Cryptexes/OSbetter-trigram.dylib\' (no such file), \'/usr/lib/swift/better-trigram.dylib\' (no such file, not in dyld cache),, js engine: hermes', { [Component Stack] name: 'Component Stack' }

Is there any place I should pay attention at? Could you please share more info

op-sqlite version

10.1.0

React Native version

0.75.4

Reproducible Example

https://github.com/iliapnmrv/op-sqlite-load-extension-ios

@ospfranco
Copy link
Contributor

You cannot dynamically load static libraries, you need to use a .framework and not a .xcframework. It's an entire topic, iOS dynamic frameworks have no extension and require to be compiled differently. You need to manually create the framework and also modify the rpath, etc etc... you will need to figure this one out yourself, but your problem has nothing to do with this library.

@ospfranco
Copy link
Contributor

Ah, I forgot I wrote about this:

https://ospfranco.com/generating-a-xcframework-with-dylibs-for-ios/

But please don't create a ticket, you will need to compile it and load it yourself, nothing wrong with the library

@iliapnmrv
Copy link
Author

iliapnmrv commented Nov 28, 2024

Sure, I know the library works perfectly and the problem is on my side
Here at least I can ask for a minor help from you or community. This will definitely help, thank you!

@ospfranco
Copy link
Contributor

If you want help from the community it's better if you join discord.

@ospfranco
Copy link
Contributor

Actually, it's been a while since I've taken a look at extension loading. I strongly discouraged it because it's such a complex process. I don't think you can load the dynamic frameworks on iOS without modifying the C++ code. Unlike Android they are not automacally loaded, so telling sqlite to load them for you doesn't work.

Creating a static framework might load, but I'm not sure how to tell sqlite how to call the entry poiint without it calling dlopen which is the dynamic linker library that tries to load a dynamic library.

@ospfranco
Copy link
Contributor

Looking at your library you want to link, I don't see why you couldn't use custom tokenizer approach for it, just copy the source files and implement the correct signature.

@ospfranco
Copy link
Contributor

I've added a new method to get dylibs runtime paths on iOS, this should allow to load any library that has been properly compiled as a dylib for iOS into sqlite. Here is the PR which the updated docs:

#192

Once it's merged you should be able to load any extension.

@iliapnmrv
Copy link
Author

Thank you so much! I've tried custom tokenizers and it worked perfectly for ios. I have multiple tokenizer files and one of
then had fts5 include. So I just put the function on my tokenizers.cpp file

Right now I'm facing android build problem (no problem on ios)

/*  better-trigram.h */
#ifndef SQLITE_BETTER_TRIGRAM_H
#define SQLITE_BETTER_TRIGRAM_H

#ifndef SQLITE_CORE
#include "sqlite3ext.h"
#else
#include "sqlite3.h"
#endif

#ifndef SQLITE_PRIVATE
#define SQLITE_PRIVATE static
#endif
/Users/ilia/Library/Android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ --target=aarch64-none-linux-android23 --sysroot=/Users/ilia/Library/Android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -DTOKENIZERS_HEADER_PATH=\"../c_sources/tokenizers.h\" -Dop_sqlite_EXPORTS -I/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/../cpp -I/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/../cpp/sqlcipher -I/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/../cpp/libsql -isystem /Users/ilia/.gradle/caches/8.8/transforms/fb92af41f860783fe60d03c1e1dbd7ef/transformed/fbjni-0.6.0/prefab/modules/fbjni/include -isystem /Users/ilia/.gradle/caches/8.8/transforms/1f00369f60fb558f4c67464032b9f89f/transformed/react-android-0.75.4-debug/prefab/modules/jsi/include -isystem /Users/ilia/.gradle/caches/8.8/transforms/1f00369f60fb558f4c67464032b9f89f/transformed/react-android-0.75.4-debug/prefab/modules/turbomodulejsijni/include -isystem /Users/ilia/.gradle/caches/8.8/transforms/1f00369f60fb558f4c67464032b9f89f/transformed/react-android-0.75.4-debug/prefab/modules/react_nativemodule_core/include -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security  -O2 -fexceptions -DONANDROID -fno-limit-debug-info  -fPIC -std=c++20 -MD -MT CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o -MF CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o.d -o CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o -c '/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/c_sources/tokenizers.cpp'
In file included from /Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/c_sources/tokenizers.cpp:14:
  /Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/c_sources/better-trigram.h:5:10: fatal error: 'sqlite3ext.h' file not found
  #include "sqlite3ext.h"

Is there any limitations on android side?

@ospfranco
Copy link
Contributor

sqlite3ext.h is not part of the source files compiled on op-sqlite, so you probably want just to define that flag, on your package.json on the "sqliteFlags" you can add any flags you want and add "-DSQLITE_CORE" or just remove that line and include "sqlite.h"

@iliapnmrv
Copy link
Author

iliapnmrv commented Nov 30, 2024

I've tried both variants and now I'm stuck on file not found. npx pod-install correctly copies my files from c_sources but as I run ./gradlew assemble[Debug/Release] I get error. Is it something on my side?

In c_sources I have 4 files

  • better-trigram.h
  • tokenizer.c
  • tokenizers.cpp
  • tokenizers.h
What went wrong:
Execution failed for task ':op-engineering_op-sqlite:buildCMakeDebug[arm64-v8a]'.
com.android.ide.common.process.ProcessException: ninja: Entering directory `/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/.cxx/Debug/72401o3i/arm64-v8a'
  [1/3] Building CXX object CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o
  FAILED: CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o 
  /Users/ilia/Library/Android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ --target=aarch64-none-linux-android23 --sysroot=/Users/ilia/Library/Android/sdk/ndk/26.1.10909125/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -DSQLITE_CORE -DTOKENIZERS_HEADER_PATH=\"../c_sources/tokenizers.h\" -Dop_sqlite_EXPORTS -I/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/../cpp -I/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/../cpp/sqlcipher -I/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/../cpp/libsql -isystem /Users/ilia/.gradle/caches/8.8/transforms/fb92af41f860783fe60d03c1e1dbd7ef/transformed/fbjni-0.6.0/prefab/modules/fbjni/include -isystem /Users/ilia/.gradle/caches/8.8/transforms/1f00369f60fb558f4c67464032b9f89f/transformed/react-android-0.75.4-debug/prefab/modules/jsi/include -isystem /Users/ilia/.gradle/caches/8.8/transforms/1f00369f60fb558f4c67464032b9f89f/transformed/react-android-0.75.4-debug/prefab/modules/turbomodulejsijni/include -isystem /Users/ilia/.gradle/caches/8.8/transforms/1f00369f60fb558f4c67464032b9f89f/transformed/react-android-0.75.4-debug/prefab/modules/react_nativemodule_core/include -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security  -O2 -fexceptions -DONANDROID -fno-limit-debug-info  -fPIC -std=c++20 -MD -MT CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o -MF CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o.d -o CMakeFiles/op-sqlite.dir/c_sources/tokenizers.cpp.o -c '/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/c_sources/tokenizers.cpp'
  /Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/c_sources/tokenizers.cpp:15:10: fatal error: 'tokenizer.c' file not found
  #include "tokenizer.c"
           ^~~~~~~~~~~~~
  1 error generated.
  [2/3] Building CXX object 'CMakeFiles/op-sqlite.dir/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/cpp/bridge.cpp.o'
  ninja: build stopped: subcommand failed.
  
  C++ build system [build] failed while executing:
      /Users/ilia/Library/Android/sdk/cmake/3.22.1/bin/ninja \
        -C \
        /Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android/.cxx/Debug/72401o3i/arm64-v8a \
        op-sqlite
    from /Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/android

Then if I rename file to .cpp I get another error

C/C++: ld.lld: error: undefined symbol: opsqlite::opsqlite_better_trigram_init(sqlite3*, char**, sqlite3_api_routines const*)
C/C++: >>> referenced by bridge.cpp:121 (/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/cpp/bridge.cpp:121)
C/C++: >>>               CMakeFiles/op-sqlite.dir/Users/ilia/projects/ticketscloud/scanner/node_modules/@op-engineering/op-sqlite/cpp/bridge.cpp.o:(opsqlite::opsqlite_open(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&))
C/C++: clang++: error: linker command failed with exit code 1 (use -v to see invocation)
C/C++: ninja: build stopped: subcommand failed.

but my tokenizers.cpp has namespace opsqlite and method opsqlite_better_trigram_init

#ifdef SQLITE_CORE
SQLITE_PRIVATE int opsqlite_better_trigram_init(sqlite3 *db) {
  return fts5BetterTrigramInit(db);
}
#else
SQLITE_BETTER_TRIGRAM_API int
opsqlite_better_trigram_init(sqlite3 *db, char **error,
                           const sqlite3_api_routines *api) {
  SQLITE_EXTENSION_INIT2(api);
  UNUSED_PARAM(error);

  return fts5BetterTrigramInit(db);
}
#endif

@ospfranco
Copy link
Contributor

You cannot include a .c file, includes are for header .h files, the linker then takes care of linking the symbols during compilation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants