Generate p/invoke methods automatically

[ffMathy]

There is a library out there called SharpDX. They allow full DirectX 11 support for C# through PInvoke-ish APIs, but the cool thing is that they are all auto-generated and auto-converted to C#.

Could we instead of what we are doing now get inspiration from them, and make a tool which auto-generates all C# code?

It would also make it a lot easier to maintain.

[ffMathy]

This is the tool I am talking about: https://github.com/sharpdx/SharpDX/tree/master/Source/Tools/SharpGen

Notice how it also has a CPP parser etc.

[AArnott]

Thanks for the idea. Anything we can do to expedite increasing our API coverage with less effort is worth investigating.

FWIW, I've already been using pinvoke.net and the code generation feature of the P/Invoke Interop Assistant as a tool to help generate the signatures and struct types. I've been disappointed by the code generation of that tool though, as the method is may not be in the database, or it fails to parse the header files and produces nothing, or it produces a bunch of signatures that have to be fixed up simply because the header file used PUCHAR instead of void* resulting in byte* in the managed code when it should have been void* or IntPtr. So I usually just write out the signature myself, but refer to pinvoke.net or the tool output to see if it can save me time along the way.

Ultimately the very best generation tool in the world would still emit methods that would need to be hand-inspected, IntPtr replaced with SafeHandle where appropriate, helper methods added in a few cases, xml doc comments added, etc.

Hopefully SharpGen does a better job. But even if it did, I'm not sure we'd use it to replace the process of maintaining source code for the library with simply generating the source code at build time. Not sure if that was what you were suggesting. If you (or anyone else) can try out SharpGen and let us know how useful it is for some methods you want to contribute, I'm sure folks would find it interesting -- especially if your report is positive.

[ffMathy]

As for SafeHandle vs IntPtr, can't we use SafeHandle in all cases? It uses an IntPtr internally anyway.

I was considering writing a plugin which takes P/Invoke Interop Assistant's output, parsing it with Roslyn, and then using Roslyn, re-transpiling it into something we want (with fixes and tweaks).

[AArnott]

As for SafeHandle vs IntPtr, can't we use SafeHandle in all cases?

No. SafeHandle is supposed to represent native handles -- things that some sort of native release method must be called on. IntPtr can represent many things, including just a pointer-sized number (like the size of a buffer). Other times when they do represent handles, such as for structs that are initialized by the native code, the fields can't be SafeHandles because .NET can't marshal that way.

Massaging the output of a code generation tool sounds interesting. This is to aid us in checking in source code, I assume? Or is it to replace our source code? I'd be much more quick to adopt the former (as a tool to help us author source code) than the latter (a replacement for source code).

[ffMathy]

I think it is worth investing in Roslyn to see if it can replace our code. If not, it can always aid us. I am quite experienced in Roslyn, and think I can make a program that can very easily generate automatic code that follows exactly the same structure as we use now.

But what to do about SafeHandle vs IntPtr? Is there any way we can see the difference between the two? Let's think out of the box and think over-engineering, just for the sake of brainstorming. Could we use a Selenium parser on the MSDN article of the given PInvoke call and see it there?

[vbfox]

I don't think that a generated solution would be a good one in the long run. Lot of API require thinking on how to adapt them to good C#.

The only thing something automated would be able to produce is bad C# :

• IntPtr versus SafeHandle is a problem, SIZE_T is an IntPtr and have no sense as a SafeHandle.
• There isn't one SafeHandle but many, one per resource type.
• Enums are problematic, parsing C would in a lot of case produce just a bunch of const as they might not exists in the C code, and parsing documentation risk producing too much enums in cases where functions allow only a subset of values from a common enum.
• void* is the correct marshaling for buffers to allow all usages but in some cases byte[] is good enough for most use-cases.
• string or StringBuilder depends on usage
• Cases where the NullableUIntPtr type or similar is needed are hard to guess and using it everywhere will make the API hard to read. In most cases ref uint or out uint is good enough.
• Documentation need to be adapted to document correctly the API, not just copy the native API.

And that's only the case I can think of that poped from the API already in there, i'm sure we will encounter a LOT more.

---

That said a selenium based tool to generate at least a good baseline from the MSDN page would be extremely handy. Especially around documentation, cleaning it up by hand is ok but the copy-paste from MSDN part is just boring mechanical work.

[ffMathy]

Alright, let me try to gather more intel then given the above information (thanks by the way).

• As for IntPtr versus SafeHandle in the case of SIZE_T, we just make it have all SIZE_T items converted into IntPtr instead.
• What do you mean regarding "many SafeHandle, one per resource type"?
• Yes, enums could be a problem. However, they are all typically prefixed by something common (for instance, WM_ for window message constants like WM_MOUSEMOVE). This could be used to our advantage, and to generate enums (or constants) in a smarter way.
• As for void* versus byte[], why not just have both? I think all this talk about performance issues is premature optimization. If one thing is preferred over the other in most cases anyway, we could even have that as the default (for instance, having a MyPInvokeCall and MyPInvokeCallVoid or whatever).
• Why the need for adapting the documentation to the API? Why can't it be copied from MSDN?

Thanks for your clarification so far.

[vbfox]

What do you mean regarding "many SafeHandle, one per resource type"?

The same native library can contain lots of functions that take a HANDLE, some of theses are to be closed by CloseHandle, other FreeLibrary, and a few others. Any tool automating that would need to contain that hardcoded as it's not always obvious from the documentation (Often it' references handles created by function Foo, and it's the documentation of function Foo that explain how to free the resource).

Yes, enums could be a problem. However, they are all typically prefixed by something common

Some are, some aren't completely depends on the API, we would have lots of special cases that can't be generated, and the more special casing the less useful an automated too would be.

As for void* versus byte[], why not just have both?

It's possible but it might polute the API surface with lots of variants that have no chance to ever be used.

Why the need for adapting the documentation to the API? Why can't it be copied from MSDN?

Documentation sometimes link to generic articles on MSDN, references constants values one by one (where we would use enums and so just reference it), specifies what happens when some values are null that we marshalled as ref or out (And so can't be null)

There are a few changes like that that make the doc easier to consume for libraries users. (Nothing really critical but it's nice to have the library go the extra mile and provide adapted documentation that take the marshaling in account)

[ffMathy]

Hmmm, I think we can close this then. Damn! Really wanted it to work somehow, but I get your points.

[AArnott]

I may have missed part of the conversation somewhere. Which points condemned this idea?

[AArnott]

Oh, nm. I got my issues confused.

[AArnott]

I'm pleased to say we have a very good option opening up to automatically generate All The Things.

Reactivating to track.

[AArnott]

This is now closed by #565.