[Feature Request]: Support Resource Dir through CppInterOp

[anutosh491]

Description of new feature

Two main reasons why this would be helpful is that

1. Xeus-cpp's non-wasm based kernels can make use of it through kernel.json

  "argv": [
      "/Users/anutosh491/micromamba/envs/xeus-cpp/bin/xcpp",
      "-f",
      "{connection_file}",
      "-resource-dir", "/Users/anutosh491/micromamba/envs/xeus-cpp/lib/clang/19",
      "-I", "/Users/anutosh491/micromamba/envs/xeus-cpp/include",
      "-std=c++20"
  ]

Currently we do provide a location "/Users/anutosh491/micromamba/envs/xeus-cpp/lib/clang/19" but this is just a placeholder right now and such a location doesn't exist during the setup.

2. Xeus-cpp's wasm based kernel can make use of it through emscripten's preloading feature (--preload-file or --embed-file

Once we have a resource dir through cppinterop, we just move these files to dedicated location that also has contains emscripten's sysroot and then we preload all of these files together.

[anutosh491]

IIRC some discussion was done on discord for the same but not sure we had a concrete way forward. Opening an issue to keep track of the same.
Maybe someone could educate me with what needs to be done if we have a solution.

[mcbarton]

@anutosh491 @vgvassilev If we build the resource directory as part of the build process then we can mount it the virtual file system of the Jupyter lite build. When mounting we can pick the exact location of the resource dir. Its one of the options of the Jupyter lite build, so would be simple to implement.

[kr-2003]

@anutosh491 @vgvassilev
I am interested in this issue and want to work on this. Till now, I have understood that Xeus-cpp can't use some resource dir header files of clang. For example, if I try to import msa.h header(present in clang headers) in my jupyter notebook(xeus-cpp), then it gives me file not found error.
Also, while running the jupyter notebook, this is the log I get regarding resource dir:

clang -cc1 version 19.1.6 based upon LLVM 19.1.6 default target x86_64-conda-linux-gnu
ignoring nonexistent directory "/home/abhinav/micromamba/envs/xeus-cpp/lib/clang/19/include"

So, this is what I think we should do:

1. CppInterOp should include a mechanism to detect and manage Clang's resource directory, which contains the necessary header files.
2. Modify the CMakeLists.txt of CppInterOp to copy the Clang's header files into it own directory.
3. Create an API kind of thing in CppInteOp so that other projects like xeus-cpp can use to access these headers.
4. xeus-cpp can now acess these header files through the above proposed API in CppInterOp.

Please correct me if I'm wrong anywhere in my understanding or proposed approach.

[mcbarton]

@kr-2003 You do not need stage 2. Emscripten has a virtual file system, which is accessed through an API (see https://emscripten.org/docs/api_reference/Filesystem-API.html ). What you would need to do is use this API to populate the virtual file system Emscripten builds with the resource directory. Once you have devised how to use Emscriptens API to populate the virtual file system, you can then use the Emscripten API to access the headers (making some CppInterOp API which accesses the virtual file system through the code you write to access the virutal file system using Emscriptens API make sense to me).

[anutosh491]

The emscripten use case is for the future. First we need to focus on supporting resource dir through cppinterop for the native cases.

[mcbarton]

The emscripten use case is for the future. First we need to focus on supporting resource dir through cppinterop for the native cases.

@anutosh491 My understanding is that we already support detecting the resource directory for native platforms to a certain extent (see

CppInterOp/unittests/CppInterOp/InterpreterTest.cpp

Line 128 in 9c818c5

TEST(InterpreterTest, DetectResourceDir) {

). Its the case where the compiler has not been built is where the problem lies (see compiler-research/xeus-cpp#115 (comment) ). A more resilient method would not be consulting the compiler for the resource directory, and would also fix #194 I think.

[anutosh491]

My understanding is that we already support detecting the resource directory for native platforms to a certain extent

Hmmm, I don't think so. The feature for detecting the resource dir and providing a resource dir are different things isn't it.

For eg

1. While we install CppInterOp through conda-forge I see the required header ending up in a Parent location respected by clang (i.e clang/Interpreter)

(xeus-cpp) anutosh491@Anutoshs-MacBook-Air Interpreter % pwd 
/Users/anutosh491/micromamba/envs/xeus-cpp/include/clang/Interpreter
(xeus-cpp) anutosh491@Anutoshs-MacBook-Air Interpreter % ls
CppInterOp.h

2. So technically we should see lib/clang/XX where XX stands for the current version cppinterop is running with i.e 19.
   .I currently see

(xeus-cpp) anutosh491@Anutoshs-MacBook-Air clang % pwd
/Users/anutosh491/micromamba/envs/xeus-cpp/lib/clang
(xeus-cpp) anutosh491@Anutoshs-MacBook-Air clang % ls
16      19.1.7

So basically these locations aren't coming out of cppinterop but if you fetch clangdev you would see lib/clang/19 ... just like the above case, we need to find the resource headers at a known location.

[kr-2003]

The emscripten use case is for the future. First we need to focus on supporting resource dir through cppinterop for the native cases.

@anutosh491 My understanding is that we already support detecting the resource directory for native platforms to a certain extent (see

CppInterOp/unittests/CppInterOp/InterpreterTest.cpp

Line 128 in 9c818c5

TEST(InterpreterTest, DetectResourceDir) {

). Its the case where the compiler has not been built is where the problem lies (see compiler-research/xeus-cpp#115 (comment) ). A more resilient method would not be consulting the compiler for the resource directory, and would also fix #194 I think.

I have a doubt.
From the PRs and comments related to resource-dir, I got to know that the DetectResourceDir will only work when clang is present(because it runs clang -print-resource-dir).
But why would there be a case of that clang is not present. Since, CppInterOp depends on clang and should install it while building itself.

[mcbarton]

The emscripten use case is for the future. First we need to focus on supporting resource dir through cppinterop for the native cases.

@anutosh491 My understanding is that we already support detecting the resource directory for native platforms to a certain extent (see

  [CppInterOp/unittests/CppInterOp/InterpreterTest.cpp](https://github.com/compiler-research/CppInterOp/blob/9c818c5e2f1a31248168334de84127b33756e895/unittests/CppInterOp/InterpreterTest.cpp#L128)


     Line 128
  in
  [9c818c5](/compiler-research/CppInterOp/commit/9c818c5e2f1a31248168334de84127b33756e895)





    
      
       TEST(InterpreterTest, DetectResourceDir) { 

). Its the case where the compiler has not been built is where the problem lies (see compiler-research/xeus-cpp#115 (comment) ). A more resilient method would not be consulting the compiler for the resource directory, and would also fix #194 I think.

I have a doubt. From the PRs and comments related to resource-dir, I got to know that the DetectResourceDir will only work when clang is present(because it runs clang -print-resource-dir). But why would there be a case of that clang is not present. Since, CppInterOp depends on clang and should install it while building itself.

@kr-2003 You don't actually need to build the clang compiler to pass any of the other tests, you just need libclang to be built (which is built as part of the clang target when building llvm). You need the clang project to be enabled to build CppInterOp, but don't actually need the clang compiler for any other test other than DetectResourceDir. You can see this by following the build instructions in this repo, by changing the clang target for libclang. It would be nice in my opinion if we could have a way to detect the resource directory directly from libclang, but have no idea if that can actually be done.

[kr-2003]

It would be nice in my opinion if we could have a way to detect the resource directory directly from libclang, but have no idea if that can actually be done.

@mcbarton
How about using clang::CompilerInvocation::GetResourcePath()? Can we use this through libclang(even though the whole clang compiler is not installed)?

Gets the directory where the compiler headers reside, relative to the compiler binary (found by the passed in arguments).

[mcbarton]

@kr-2003 The truth is that I don't know. My intuition says no, since it says relative to the compiler binary . Others in the group will probably be able to give you a definitive answer.