PyTable.C dereferencing elements of empty vector, crashing CLI

[biagas]

Describe the bug

The flatten.py test is crashing the CLI on Windows.
It is happening during the box_selection test, specifically the call to test_box(True):

    TestSection('BoxSelectionNoSharedMemory')
    test_box(True)

Running through VS debugger, it segv's on this line in PyTable.C because 'results' is empty:

visit/src/visitpy/common/PyTable.C

Line 61 in c082687

std::cout << results[0] << ", " << results[1] << ", " << results[2] << std::endl;

I'm sure 'results' being empty indicates a bigger problem.

Adding if-test for results.size before printing out its contents fixes the segv, but these BoxSelectionNoSharedMemory tests fail: multi_rect2d_box_nodes and mult_rect2d_box_zones

Helpful additional information

• Did VisIt crash: CLI crashes
• Did you get wrong results:

Desktop

• OS and version: Windows 10
• VisIt Version: develop
• Server info (if applicable):

[biagas]

@cyrush, can you direct me on how best to discover how/why the flatten query fails on Windows?
I have fixed the segv on my local copy and will create a PR soon for that, but the failure is what tickled the bug.

[cyrush]

@Laganellac, could this case be an issue b/c the results are empty and their aren't guards later in the process? (We move the vector, but it is just fundamentally empty?)

[Laganellac]

Hi @cyrush and @biagas, I can remove the std::cout prints; at this point they probably wouldn't even be useful in the debug log. I must've missed them somehow when making my cleanup pass over the code.

As for the results vector being empty - this indicates to me that the avtFlattenQuery has filled out the XMLResult with valid metadata about the size / shape of the results vector yet when it got to the CLI the results vector was empty. My gut tells me this is coming from this block of code in avtFlattenQuery.C:

visit/src/avt/Queries/Queries/avtFlattenQuery.C

Lines 708 to 735 in f7c9558

	if(PAR_Rank() == 0)
	{
	const double sizeInGigaBytes = (pimpl->outData.size() * sizeof(FloatType)) / (double)1e9;
	debug5 << "avtFlattenQuery XML output:\n" << pimpl->outInfo.ToXML() << std::endl;
	qA->SetXmlResult(pimpl->outInfo.ToXML());
	qA->SetResultsMessage("Success!\nNOTE: This query should only "
	"be used in the CLI via the Flatten() function.");
	if(pimpl->useSharedMemory)
	{
	WriteSharedMemory();
	}
	else if(sizeInGigaBytes > pimpl->maxDataSize)
	{
	std::stringstream s;
	s << "ERROR: Data too large to transport via query attributes ("
	<< sizeInGigaBytes << "GB). You can override the limit by"
	<< " overriding the parameter 'maxDataSize' with a (double)"
	<< " size repesented in Gigabytes. (Current value = "
	<< pimpl->maxDataSize << ").\nNOTE: This query should only "
	<< "be used in the CLI via the Flatten() function.";
	qA->SetResultsMessage(s.str());
	debug1 << s.str() << std::endl;
	}
	else
	{
	SwapIntoQueryAttributes(*qA, pimpl->outData);
	}
	}

I can change this logic to only set the XMLResult after we've verified that the results vector will be populated. However, I'm not convinced this is the whole issue because I would expect the Flatten unit test to also fail on Linux if we were hitting the 'else if' branch in the above code.

Are there debug logs from running the test on Windows that I could take a look at?

[biagas]

@Laganellac here are the engine and viewer log files from a run of the test on Windows where I commented out all but test_box_selection, which is the one that fails. Interestingly enough, if I comment out the shared memory portion of test_box_selection, the non-shared portion succeeds.
engine_viewer_logs.zip

[Laganellac]

Hi @biagas, I've made a small change related to this in PR #17423; however, I don't think this will solve the problem with the test on Windows. Unfortunately I cannot reproduce this bug on my Ubuntu machine and will have to setup a Windows build to try and debug. Based on the log output, everything coming from the engine looks correct - specifically line 479 has the correct input parameters, and lines 864 - 889 have the expected XMLInfo. Same goes later in the file when the test is run again with no shared memory.

When the test fails on Windows does it still crash (now that there are no printouts)? Or does it just fail the text diff for the file contents? On Windows, both the shared memory and non shared memory approaches should result in the exact same behavior (shared memory support is turned off for Windows in Flatten), so maybe there is a bug in my logic that always disables shared memory on windows but I'm not seeing that in the log output.

[biagas]

The test fails on Windows (no longer crashes).
It is the second run of the test_box that fails. If I leave the flatten.py file untouched, then the non-shared-memory test_box call fails.
Here are the results from that run:
flatten_box_no-shmem_results.zip

If I swap the order, is still the second call to test_box that fails, so this time the shared memory version.
Here are the results from that run:
flatten_box_shmem_results.zip

[cyrush]

@biagas -- Revisiting -- I can't quite understand the path forward.

Do we need to get #17423 into 3.3? Or did another PR heal this?

[biagas]

@cyrush, the offending code was removed, and the cli no longer crashes, so I don't think anything needs to be done for 3.3.

However, the tests still fail on Windows, and the reason for the failure has yet to be discovered. #17423 may help, I can give it a try.

Here the first line of values in the expected results:

# vec_nid/c0,vec_nid/c1,vec_nid/c2,nodeIds,nodeDomains
0.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00,0.000000000000000000e+00

and the same results on windows:

# vec_nid/c0,vec_nid/c1,vec_nid/c2,nodeIds,nodeDomains
3.454112000000000000e+06,7.426881860921530476e-43,3.348628000000000000e+06,7.426881860921530476e-43,1.740819699404572229e+25

Looks like standard windows garbage for uninitialized vars.

[biagas]

@cyrus, I added a new ticket describing the test suite problem on Windows (#17767)