libnet and libnio #409
libnet and libnio #409
Conversation
Need to devise a way to get the platform specific headers from IKVM.Java. This requires new IKVM.Net.SDK support, and to split up IKVM.Java into platform specific versions. Which introduces our ref-assembly problem. Needs to be solved.
…ded since they don't exactly match the headers generated by IKVM.Java. Some important utiltiies are available, however. For instance, Net.socket0, for creating sockets, can now be used. FileDispatcherImpl will be an important one to get building, but it doesn't build just yet. ServerSocketchannelImpl does, SocketDispatcher, SocketChannelImpl, LinuxWatchService.c builds. UnixNativeDispatch Trying to figure out how to materialize a .NET framework socket out of a handle.
…ends can be used to set the FD from the call to socket0(). Core makes this easy, as it will take an existing handle. Framework needs to invoke the SocketInformation ctor, with information provided by a duplicate socket. Mono is unimplemented. But, if on Unix, we can probably pack some stuff into SocketInformation. Leave that up to the future. Fix a few JNI calls.
Update Clang version to release.
…s technically still Windows machines with Framework out there with ARM32.... but, I think the chances of them using IKVM are zero.
…d not available in musl.
Always use LoadLibraryEx on Windows. NativeLibrary seems to have some uncertainty about when it tries to do path searches and stuff.
…plicated things needlessly between Framework/Core. Rename libjava to libikvm. I hate this. But, .NET Core produces a java.dll file, which is loaded into the process using LoadLibrary, even though it's not native. And screws up dependency resolution for net.dll and nio.dll. dotnet/runtime#90839 Will research the single file thing. Need to figure out how to apply it to everything in files to publish.
@wasabii Can you share some of the complexity being observed here? I think we can all appreciate the inexact semantic behavior between the two approaches, but falling back to /cc @elinor-fung @jkotas |
|
Yeah. We're trying to implement the exact semantics of OpenJDK on the OS we're running on. It's not that we can't get this working by going through .NET Core's NativeLibrary implementation by being careful: it's that it's an additional level of indirection to think through. One which we cannot use on Framework anyways. For instance, passing an Assembly, and it resolving from the AssemblyContext ResolveUnmanagedDll. Or how it implements it's own cache. Or how it (in nativelibrary.cpp) tries to do the DllImport logic of appending .so and .dll, and prepending lib, etc. OpenJDK already contains their own version of all that logic (written in Java, in ClassLoader), and makes it's own determiniation of when to pass through the libname unmolested, etc. For instance, in tracking down the libjava.dll issue I linked from that commit, I got misdirected a number of times by assuming the fact that Framework and Core functioned differently were because of the additional logic in SRIS.NativeLibrary. It wasn't until I removed SRIS.NativeLibrary until I realized it was a deeper issue with how Win32 LoadLibrary itself functioned. I spent a day going over the .NET Core NativeLibrary code, trying to spot why it was failing in this particular case.... maybe it was changing the path. Maybe it was passing some unexpected option... etc. If we've introduced a security issue, as long as it's the exact same security issue present in OpenJDK when running on Windows, we're doing the right thing. So, it's not that we can't use it, and get it to work. It's that there is no point. It only makes it harder, not easier. |
Just call Load a second time. Same result. libnio needed a blank JNI_OnLoad. Not sure on this one. Doesn't exist in OpenJDK. Wrap dlopen() in libjvm. P/Invoke wasn't capable of resolving symbol properly on Linux, with libdl being deprecated in Ubuntu > 22.
Enable native initIDs method. Add a datagram test to the Java lang tests. Add a test to check that adhoc send fails when connected. Add some tests for bind with null values.
Add a test for joining IPv6 multicast groups. Skip unaligned long compare and swap tests on ARM[64]. Crashes with SIGBUS or similar. ARM doesn't support unaligned atomics. Add missing socket functions ot jvm.cpp.
…de the same functionality. Fix test for osx-arm64.
…rgetName).dylib, which should allow linking libs to resolve from relative paths? Maybe.
Use native FileDispatcher. We had to make a local copy of FileDispatcher.c in order to use lock0, since the signatures differ between solaris/windows versions. This is a minor fork, and gets locking working... maybe on OS X too? Remove lock0/release0 implementation from C#, use C version.
…or non-java related IKVM native functionality apparently.
…ndencies on OpenJDK related things. dl functions moved here. This is to prevent the bootstrapping issue with being forced to load libjvm itself using DllImport: the path of libjvm has to follow Java library load rules. libikvm follows .NET loading rules.
…the native OpenJDK libraries. Unified the logic for building .runtime images into IKVM.Image/IKVM.Image.runtime.targets. We reuse IkvmImageItem processing, but still pack. This puts the native libraries exclusively in the image. Which will let us eliminate our search path processing for runtimes/native/*. The sun boot path can now be set to the image path, while the user lib path can be set to runtimes/ or the application dir. Need to do that still.
NativeLibrary once again using NativeLibrary where available. This makes more sense now that we ensure full resolution before invoking. NativeLibrary has to bootstrap ikvm.dll on Windows without Mono by calling LoadLibrary directly, since the .NET SDK has the potential for putting the DLL files under runtimes/rid even under Framework. But sometimes it doesn't. boot path replaced with Image path. user path replaced with .NET paths. DestinationSubDirectory still needed for files without extensions. We preload jvm.dll through System.loadLibrary, so that it resolves out of JVM system paths. But, then we also load it by hand in JNI to get the JNI trampoline functions. Ensure we finalize all the manually loaded native libraries.
…C_PATH to @loader_path, and names itself @rpath/libjvm.dylib. We preload libjvm using NativeLibrary (or libikvm), against a full path.
… Regardless, DllImport couldn't find it, even after a successful preload. Linux behaves same as OSX in that regard. Just use out NativeLibrary to pull out symbols.
|
I do not see a problem with your approach. The managed |
|
Final design in here. This gets complicated. There are two paths to loading native libraries in IKVM: 1) loading regular native libraries for use in .NET code 2) native libraries loaded for usage within JNI (through System.loadLibrary).
IKVM.Runtime.NativeLibrary might not have System.Runtime.InteropServices.NativeLibrary (Framework). In that case, it uses libikvm's IKVM_dl_open/dl_close/dl_sym functions. These functions are implemented in C, instead of PInvoking to dlopen/dlsym and dlclose, because the library that contains dlopen/dlsym/dlclose isn't fixed. Older versions of Linux may have it in libdl. Newer versions have it in libc. There used to be a compatibility library. Sometimes there isn't. So, P/Invoke can't always find dlopen/dlclose/dlsym. However, IKVM.Runtime.NativeLibrary can't use libikvm without loading libikvm. Chicken meet egg. So, libikvm is has to be loaded differently. On Core, we can use System..NativeLibrary. On Framework on Windows we P/Invoke to LoadLibrary. And on Mono we can use DllImport with dllmaps. So, loading libikvm itself goes through the most available .NET way of doing things, either S..NativeLibrary or P/Invoke. Two methods which don't have much flexibility, but that's okay, because libikvm has no dependencies. So that flexibility doesn't matter. 2 JNI. This is handled through JNI.JNINativeLoader, which invokes JVM_LoadLibrary and JVM_UnloadLibrary directly out of libjvm. This is so for this path we can rely on the OpenJDK C based logic. New bootstrapping problem: calling JVM_LoadLibrary from JNINativeLoader (C#) requires us to load libjvm. This is handled by LibJVM.cs. In here, we use IKVM..NativeLibrary. But with a well known path off of IKVM_HOME: IKVMHOME/bin. And in this case, we cannot use DllImport, but have to obtain function pointers and marshal to delegates by hand, since our path is dynamic. Since we have a well-known absolute path, IKVM..NativeLibrary to load libjvm does not perform any searching logic, that logic is contained in LibJVM.cs. libjvm is loaded on demand (first requirement to load a JNI library). Other native libraries will go through I..NativeLibrary, which MAY use S..NativeLibrary, or libikvm, depending on the requirements of the particular platform. So, JNI stuff is loaded using JVM_LoadLibrary out of libjvm. libjvm is itself loaded by IKVM.Runtime.NativeLibrary, but with a well known path. IKVM.Runtime.NativeLibrary, on some platforms, requires libikvm. libikvm is itself loaded by the lowest common denominator of System.Runtime.InteropServices.NativeLibrary, or calls to LoadLibrary, and on Mono we can use DllImport with a dllmap file. Eventually a user's JNI library runs (loaded by JVM_LoadLibrary), and because of the @rpath stuff mentioned previously that resolves to the same libjvm as was preloaded by LibJVM.cs, preserving the static methods we use for callbacks for JVM_CreateJavaVM and those guys. So the end result is two paths for loading. One path for loading plain .NET stuff, the other for loading JNI stuff. IKVM distributes the built-in JNI libraries outside of the normal .NET path locations: we put them in IKVM_HOME. They aren't loaded through .NET paths, so this is okay. And we already have a RID specific IKVM_HOME drop available. So moving them in there makes a lot of sense. The files get shared by IKVM.Image.JRE and IKVM.Image.JDK, with .runtime specific packages IKVM.Image.runtime.win7-x64, etc. .NET-ish native libraries end up distributed normally, either copied to the user's bin directory, or runtimes/rid. In the Java loading path, System.loadLibrary goes through two sets of paths: sun.boot.library.path and java.library.path. The first is meant for finding the built-in libraries. It's search path is setup to be IKVM_HOME/bin. java.library.path however is different per OS: inheriting the search logic of OpenJDK, with different paths for each OS. We add our own little magic into java.library.path though: we include the .NET search locations in ther. So, libraries located in .NET standard locations can also be loaded through JNI. This lets the user build .NET projects which use IKVM as a dependency, and package their native libraries in NuGet, but still access them through IKVM. Lot of work. |
…rt for initializating based on file based FD or handle (using libikvm to detect if a handle is a file or socket). Add initIDs back to our local fork. We might be pretty close to discarding this fork and adding fields/properties through map.xml. Replace RandomAccessFile.open0 with native code. Maybe this will impact OS X locking? Ensure libjava (libiava) is loaded before any JNI call. This isn't loaded by any OpenJDK code that I can find. Seems to be built-in/internal. But, we need it to be loaded to bind JNI methods, and we don't wnat to edit OpenJDK Java files. isnanf() doesn't exist in MUSL. Manually define to isnan(). Move LibIkvm to separate file.
…Loader. In here we prepopulate nativeLibraries with libjava (libiava). We do that on the boostrap class loader, instead of just in time, to avoid recursion with string comparison stuff that also relies on libjava. libjava itself needed to be loaded through libjvm (it is a JNI library).
… implement it using mJNI, intercept VFS calls, and then forward to native open0 implementation using JNIFrame by hand. Works fine. Add Flush wherever FileDispatcherImpl calls write, since that seems to actually matter.
…dle to the loaded library, and handles proper finalization. Ensures finalizers run after everything else. Might have to do something similar for libJVM's LoadLibrary call.
Throw IOException properly on unauthorized exception.
Importer tests should have runtime libs. Update Core, which fixes PublishProjectReferences to not ignore nested directories.
Dispose in proper order in tests.
Integrate libnet and libnio projects, taking OpenJDK C files. We've got a bunch of stuff included in these, and a bunch of stuff still excluded. But over time we can enable more and more of it.
This PR also removes our forked copies of Net.java and IOUtil.java, since the standard versions can now be called through libnio.
Introduced IKVM.Image.runtime.* packages for each target RID. Native libraries now get packaged into these, instead of the main IKVM.Image project. The targets file has changed a bit to support Pack operations (used intree). PublishProjectReference was cleaned up to output to IkvmImageItem.
Native libraries moved into the Image: under ikvm/bin. As they would be in a real JVM. IKVM sun.boot library path set to resolve from ikvm.home, out of the image. User path set to resolve out of .NET standard locations (app dir, runtimes/*). This puts user libraries in the proper path.
Removed everything from the built-in library set. On IKVM nothing is statically compiled. Introduced an empty libawt to satisfy that.
libjvm needs to be preloaded by the JNI stuff to set up callbacks.