Get-D365Module -ExcludeBinaryModules is broken if model references a binary model

[TrudAX]

Updated to the latest version and found out that Get-D365Module -ExcludeBinaryModules generates an error if a model contains a reference to a binary model

When running, it generates the following exception:
Exception calling "ListModulesInDependencyOrder" with "0" argument(s): "Module reference 'SDSRel' in model 'DEVExternalIntegration' cannot be located. This is an indication that model store is in a broken state."

We have a DEVExternalIntegration model(with a code) that references SDSRel, and SDSRel is a binary model

The expected result - this should not cause any exception.

[FH-Inway]

Thanks, I can reproduce that on my end. Seems this issue was introduced by the changes for #745 .
The metadata api disk provider now tries to list the modules in dependency order. As disk provider, I believe it can't "see" binary only modules, it needs the source code along with the binaries.

d365fo.tools/d365fo.tools/functions/get-d365module.ps1

Line 164 in 4170454

$diskModules = $metadataProviderViaDisk.ModelManifest.ListModulesInDependencyOrder()

[Splaxi]

....

And here I thought the change was an improvement 🤯

I'll do some compare on the changes and see if we can produce a working script. Then you can run it locally and let me know if all scenarios are covered.

[TrudAX]

Thanks.
Just a note - The problem I am trying to resolve is to get the modules for compile in the correct order(for example when you installing a new VM)

initially, it was the problem that this didn't work if you never compiled models(for example, for new VMs)

The standard full compile somehow resolves this problem(if you run a full compile, modules will be compiled in the correct order)

$modules = Get-D365Module -ExcludeBinaryModules -InDependencyOrder | Get-D365Model -ExcludeMicrosoftModels -CustomizableOnly | Select-Object -Property Module -Unique
    foreach ($model in $modules)
    {
        Invoke-D365ProcessModule -Module $model.Module  -ExecuteCompile 
    }

[TrudAX]

Just an idea. Maybe instead of fixing Get-D365Module -ExcludeBinaryModules you can create a powershell script that compiles a solution(the same what Build server does). The compile in this case is happening in the correct order.

This is a log from the build server that does this(I have no idea how to put it to PowerShell)

`
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\amd64\msbuild.exe" "C:\DynamicsSDK\VSOAgent_work\1\s\AXModulesBuild.proj" /nologo /nr:false /dl:CentralLogger,"C:\DynamicsSDK\VSOAgent_work_tasks\MSBuild_c6c4c611-aa2e-4a33-b606-5eaba2196824\1.217.0\node_modules\azure-pipelines-tasks-msbuildhelpers\tools\Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll";"RootDetailId=a2407575-e485-4c6e-920e-8c04814ff3c3|SolutionDir=C:\DynamicsSDK\VSOAgent_work\1\s|enableOrphanedProjectsLogs=true"*ForwardingLogger,"C:\DynamicsSDK\VSOAgent_work_tasks\MSBuild_c6c4c611-aa2e-4a33-b606-5eaba2196824\1.217.0\node_modules\azure-pipelines-tasks-msbuildhelpers\tools\Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll" /p:OutputPath="C:\DynamicsSDK\VSOAgent_work\1\Bin" /p:LogPath="C:\DynamicsSDK\VSOAgent_work\1\Logs" /p:platform="Any CPU" /p:configuration="Release" /p:_MSDeployUserAgent="VSTS_09cdf03e-120a-4c77-bc86-e10dacc3b007_build_1_0"

`

[TrudAX]

Just a note - it seems other functions are broken with the latest changes. For example if I execute
Invoke-D365ModuleFullCompile -Module CTM (for the provided in the first post structure)
it now produces the error
Exception calling "ListModulesInDependencyOrder" with "0" argument(s): "Module reference 'SDSRel' in model 'DEVExternalIntegration' cannot be located. This is an indication that model store is in a broken state.

DEVExternalIntegration references binary SDSRel, so this situation should not cause the exception

[FH-Inway]

I played around with this a bit. To reproduce the scenario, I created two modules, AAABinary and AAANonBinary. The contents are very simple, AAABinary references ApplicationPlatform and has a runnable class. AAANonBinary references AAABinary and has a class that inherits from the runnable class in AAANonBinary. So the dependency order is
ApplicationPlatform < AAABinary < AAANonBinary

I was not able to reproduce the issue when running Invoke-D365ModuleFullCompile -Module AAANonBinary, it just compiled the module as expected.
However, I was able to reproduce the issue when running Invoke-D365ProcessModule -ExecuteCompile -Module AAANonBinary. In that case I got the error message about ListModulesInDependencyOrder. Note that if AAANonBinary had been compiled before, this would still compile AAANonBinary, despite the error message. It only fails if AAANonBinary is only available as source code, but not as binaries.

PS C:\Repositories\Github\FHInway\d365fo.tools> Invoke-D365ProcessModule -ExecuteCompile -Module AAANonBinary
Exception calling "ListModulesInDependencyOrder" with "0" argument(s): "Module reference 'AAABinary' in model 'AAANonBinary' cannot be located. This is an indication that model store is in a 
broken state."
At C:\Repositories\Github\FHInway\d365fo.tools\d365fo.tools\functions\get-d365module.ps1:164 char:13
+             $diskModules = $metadataProviderViaDisk.ModelManifest.Lis ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : InvalidOperationException
 
[17:51:40][Invoke-D365ProcessModule] Modules to process: 

As a first step, I added a try/catch to the ListModulesInDependencyOrder call in Get-D365Module to fall back to the previous ListModules call. This should get us the best of both worlds for most scenarios. See pr #762 . With that change, the output is

PS C:\Repositories\Github\FHInway\d365fo.tools> Invoke-D365ProcessModule -ExecuteCompile -Module AAANonBinary
WARNING: [18:02:53][Get-D365Module] Failed to retrieve disk modules in dependency order. Falling back to ListModules().
[18:02:53][Invoke-D365ProcessModule] Modules to process: 
[18:02:53][Invoke-D365ProcessModule]  - AAANonBinary 

There is a scenario where this might still cause issues. Assume the following dependency order:
ApplicationPlatform < AAABinary < AAANonBinary1 < AAANonBinary2.
Also assume that AAANonBinary1 and AAANonBinary2 have not been compiled yet, so only their source code is available. In this case, Get-D365Module would first call ListModulesInDependencyOrder, run into an exception and fall back to ListModules. Since we don't know in which order ListModules returns the modules, it might return AAANonBinary2 first and AAANonBinary1 second. This would result in a compile error, because the dependency AAANonBinary1 is not compiled when it compiles AAANonBinary2.

[TrudAX]

Thanks @FH-Inway . Catch loop should at least return behaviour to the original state. can you release these changes? I can't tell how important is your AAANonBinary1,AAANonBinary2 sample, I need to test it, but just an idea - maybe to resolve such problem add an optional parameter, where the user may specify what model to choose first?

[Splaxi]

0.7.2 is on its way. Take it for a spin, and lets nail this son of a gun...

[FH-Inway]

I think this issue can be closed now.