Cleanup VARIANT marshalling and convert to managed (#102498)

* Handle simple cases of MarshalComVariantForOleVariant * Pseudo code for MarshalOleVariantForComVariant * Complete and setup managed call for MarshalObjectForOleVariant * Setup MarshalHelperConvertObjectToVariant except record * Setup MarshalHelperCastVariant * Cleanup Variant struct and FCall * Handle record case * Cleanup VariantOleToCom/VariantComToOle routines * Remove VariantData definition at unmanaged side * Eliminate a dead branch for ref * Remove CVTypes in native * Share GetComIPFromObjectRef * Add tests for managed to native side * Add test for native to managed side * Add test for byref returning to BYREF * Fix BYREF handling * Fix array marshalling * Fix record marshalling * Fix VT_VARIANT case * Cleanup and fix IUnknown case * Add more test for VT_UNKNOWN * Cleanup for VT_VARIANT * Cleanup SR and comment * Fix contract in GetTypeHandleForVarType * Move VariantChangeTypeEx to coreclr * Add unmanaged fast path for (U)Int64 * Make ComVariant immutable * Add GCPROTECT in ConvertSystemColorToOleColor * Fix variant init * Fix BYREF|EMPTY * Initialize ComVariant * Move GCPROTECT to the QCall * Pass UnknownWrapper through SetFieldsObject * Apply suggestions from code review Co-authored-by: Aaron Robinson <[email protected]> * Suggestions at native side * Rename ComIP * Add comments for ConvertWrappedObject * Try convert to RECORD for all cases * Dispose Variant on failure * Update formatting * Fix NO_MAPPING definition --------- Co-authored-by: Aaron Robinson <[email protected]>
dotnet · Aug 28, 2024 · c1a9f26 · c1a9f26
1 parent 2df9935
commit c1a9f26
Show file tree

Hide file tree

Showing 26 changed files with 941 additions and 2,101 deletions.
diff --git a/src/coreclr/System.Private.CoreLib/System.Private.CoreLib.csproj b/src/coreclr/System.Private.CoreLib/System.Private.CoreLib.csproj
@@ -285,6 +285,9 @@
     <Compile Include="$(CommonPath)Interop\Windows\OleAut32\Interop.VariantClear.cs">
       <Link>Common\Interop\Windows\OleAut32\Interop.VariantClear.cs</Link>
     </Compile>
+    <Compile Include="$(CommonPath)Interop\Windows\OleAut32\Interop.VariantChangeTypeEx.cs">
+      <Link>Common\Interop\Windows\OleAut32\Interop.VariantChangeTypeEx.cs</Link>
+    </Compile>
   </ItemGroup>
   <ItemGroup Condition="'$(FeatureObjCMarshal)' == 'true'">
     <Compile Include="$(BclSourcesRoot)\System\Runtime\InteropServices\ObjectiveCMarshal.CoreCLR.cs" />

diff --git a/src/coreclr/System.Private.CoreLib/src/Microsoft/Win32/OAVariantLib.cs b/src/coreclr/System.Private.CoreLib/src/Microsoft/Win32/OAVariantLib.cs
@@ -151,19 +151,10 @@ private static ComVariant ToOAVariant(object input)
                 null => default,
                 Missing => throw new NotSupportedException(SR.NotSupported_ChangeType),
                 DBNull => ComVariant.Null,
-                _ => GetComIPFromObjectRef(input) // Convert the object to an IDispatch/IUnknown pointer.
+                _ => Variant.GetIUnknownOrIDispatchFromObject(input) // Convert the object to an IDispatch/IUnknown pointer.
             };
         }
 
-        private static ComVariant GetComIPFromObjectRef(object? obj)
-        {
-            IntPtr pUnk = GetIUnknownOrIDispatchForObject(ObjectHandleOnStack.Create(ref obj), out bool isIDispatch);
-            return ComVariant.CreateRaw(isIDispatch ? VarEnum.VT_DISPATCH : VarEnum.VT_UNKNOWN, pUnk);
-        }
-
-        [LibraryImport(RuntimeHelpers.QCall, EntryPoint = "MarshalNative_GetIUnknownOrIDispatchForObject")]
-        private static partial IntPtr GetIUnknownOrIDispatchForObject(ObjectHandleOnStack o, [MarshalAs(UnmanagedType.Bool)] out bool isIDispatch);
-
         private static object? FromOAVariant(ComVariant input) =>
             input.VarType switch
             {