Investigate/mitigate the spurious clangd failures on the internal HLSL ADO pipeline

[bogner]

The microsoft internal HLSL ADO pipeline has been getting spurious test failures in the clangd unit tests. These don't seem to be happening on any public bots and they don't seem to be reproducible on individuals' local machine setups.

We need to carve out some time to investigate and figure out what to do. This could involve trying to reproduce the problem and debug it, fiddling with settings in the bot to see if we can get more information, reaching out to external folks to see if anyone is aware of similar issues, or attempting to disable or automatically flag the failures as inactionable.

The failures are recognizable as a failure in the clangd unit tests that doesn't tend to have a proper error message. They are not always in the same test. Sometimes accompanied by un-symbolized backtraces like so:

2024-10-21T06:44:24.7771696Z    #0 0x00007ffe1fd40ff0 (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x70ff0)
2024-10-21T06:44:24.7771860Z   
2024-10-21T06:44:24.7772075Z    #1 0x00007ffe1fcd8bc3 (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x8bc3)
2024-10-21T06:44:24.7772237Z   
2024-10-21T06:44:24.7772404Z    #2 0x00007ffe1fcdf977 (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0xf977)
2024-10-21T06:44:24.7772915Z   
2024-10-21T06:44:24.7773085Z    #3 0x00007ffe1fcd2d84 (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x2d84)
2024-10-21T06:44:24.7773248Z   
2024-10-21T06:44:24.7773413Z    #4 0x00007ffe1fcd22dc (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x22dc)
2024-10-21T06:44:24.7773572Z   
2024-10-21T06:44:24.7773739Z    #5 0x00007ffe1fce1657 (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x11657)
2024-10-21T06:44:24.7773901Z   
2024-10-21T06:44:24.7774059Z    #6 0x00007ffe1fce19a3 (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x119a3)
2024-10-21T06:44:24.7774220Z   
2024-10-21T06:44:24.7774387Z    #7 0x00007ffe1fce1a63 (C:\ProgramData\Microsoft\VisualStudio\Setup\x64\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x11a63)
2024-10-21T06:44:24.7774552Z   
2024-10-21T06:44:24.7774827Z    #8 0x00007ff68016be63 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x2ddbe63)
2024-10-21T06:44:24.7774985Z   
2024-10-21T06:44:24.7775229Z    #9 0x00007ff6800787dc (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x2ce87dc)
2024-10-21T06:44:24.7775386Z   
2024-10-21T06:44:24.7775633Z   #10 0x00007ff680015cad (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x2c85cad)
2024-10-21T06:44:24.7775792Z   
2024-10-21T06:44:24.7776028Z   #11 0x00007ff67fffbae3 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x2c6bae3)
2024-10-21T06:44:24.7776193Z   
2024-10-21T06:44:24.7776440Z   #12 0x00007ff67e25db18 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0xecdb18)
2024-10-21T06:44:24.7776603Z   
2024-10-21T06:44:24.7776849Z   #13 0x00007ff67ee0b627 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a7b627)
2024-10-21T06:44:24.7777002Z   
2024-10-21T06:44:24.7777249Z   #14 0x00007ff67ee16397 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a86397)
2024-10-21T06:44:24.7777406Z   
2024-10-21T06:44:24.7777645Z   #15 0x00007ff67ee13fde (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a83fde)
2024-10-21T06:44:24.7777803Z   
2024-10-21T06:44:24.7778039Z   #16 0x00007ff67ee14d33 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a84d33)
2024-10-21T06:44:24.7778201Z   
2024-10-21T06:44:24.7778445Z   #17 0x00007ff67ecd7cc0 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1947cc0)
2024-10-21T06:44:24.7778605Z   
2024-10-21T06:44:24.7778845Z   #18 0x00007ff67edcd72b (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a3d72b)
2024-10-21T06:44:24.7778999Z   
2024-10-21T06:44:24.7779241Z   #19 0x00007ff67edcbd8d (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a3bd8d)
2024-10-21T06:44:24.7779400Z   
2024-10-21T06:44:24.7779642Z   #20 0x00007ff67edd0315 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a40315)
2024-10-21T06:44:24.7779798Z   
2024-10-21T06:44:24.7780033Z   #21 0x00007ff67edcf718 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1a3f718)
2024-10-21T06:44:24.7780190Z   
2024-10-21T06:44:24.7780490Z   #22 0x00007ff67f000569 (D:\a\_work\1\b\build-release\tools\clang\tools\extra\clangd\unittests\Release\ClangdTests.exe+0x1c70569)
2024-10-21T06:44:24.7780647Z   
2024-10-21T06:44:24.7781185Z   #23 0x00007ffe2f1c6b4c (C:\Windows\System32\ucrtbase.dll+0x26b4c)
2024-10-21T06:44:24.7781303Z   
2024-10-21T06:44:24.7781432Z   #24 0x00007ffe30b94cb0 (C:\Windows\System32\KERNEL32.DLL+0x14cb0)
2024-10-21T06:44:24.7781553Z   
2024-10-21T06:44:24.7781683Z   #25 0x00007ffe3167ecdb (C:\Windows\SYSTEM32\ntdll.dll+0x7ecdb)
2024-10-21T06:44:24.7781796Z   
2024-10-21T06:44:24.7781880Z   
2024-10-21T06:44:24.7782036Z   --
2024-10-21T06:44:24.7782137Z   exit: 3221225477