Compiled code isn't standalone

[SWu]

I don't think the compiled function is standalone right now, as ops inside the function are registered at compile-time as hashed references to JIT functions (i.e. here), which show up in the generated source.cc as e.g. runtime::Registry::Get("op_-3048088960110736787");

I believe this means that outside of the context of the python interpreter where the function is first compiled, these references won't be valid. In particular, I don't think it's possible to directly use the source.cc as integration in a C++ app.

Is there a way to generate truly standalone native code?

[slyubomirsky]

There is apparently a C backend for TVM (idk if it's been merged into mainline yet) that generates C implementations of the operators, which we could possibly inline into these generated programs. I am not sure about the resulting performance, though. @MarisaKirisame do you have any thoughts?

[MarisaKirisame]

Does cross-compiling work for you? It will still emit those Get("") files, but they will be the right architecture.

[SWu]

I'm not sure where the target of the Get() calls live; it appears they are registered in python via self.engine.jit + register_func during the compile process, so I think they are created in memory (or at least the referenced names are)?

[MarisaKirisame]

@SWu they are so/dlls.

[SWu]

@MarisaKirisame do you know where those are? I don't see any .so other than the compiled librelay_aot_1.so from source.cc

[MarisaKirisame]

@SWu tvm will emit source code and compile them into so/dll, see https://github.com/dmlc/tvm/blob/51785062553b19a219b2639cd230b7da7455cd0d/src/codegen/source_module.cc#L97. It is possible to send them onto another machine to execute. This process will be similiar for aot, but require changes to aot. Is a native dll standalone enough?

[MarisaKirisame]

Or you can cross compile all the source once tvm finished codegen.

[SWu]

Is there some example of how to directly call codegen? Does aot currently call that codegen for the underlying ops?

[MarisaKirisame]

@jroesch I am not very familiar with low level part of tvm. can we meet and talk about changes need to make this happend.