Out of memory testing our implementation

[jerstlouis]

Describe the bug

validate Feature Collections Metadata Operation | java.lang.OutOfMemoryError: Java heap space

This hapens with both https://cite.opengeospatial.org/teamengine/ and https://cite.opengeospatial.org/te2/

To Reproduce
Steps to reproduce the behavior:

Test https://maps.ecere.com/ogcapi (3 max collections)

Expected behavior
Not running out of memory.

Additional context
This also leads to many other "Skipped" tests, e.g. in Core:

validate Feature Collections Metadata Operation Response_ Items | org.testng.SkipException: Could not find a response for test point Server URL: https://maps.ecere.com/ogcapi , Path: /collections, Replacements: {}

validate Feature Collection Metadata Operation | java.lang.Throwable: Method FeatureCollection.validateFeatureCollectionMetadataOperation(org.testng.ITestContext, java.util.Map)<br>[pri:0, instance:org.opengis.cite.ogcapifeatures10.conformance.core.collections.FeatureCollection@7278cf49] depends on not successfully finished methods in group "collections"

validate Feature Collections Metadata Operation Response_ Links
org.testng.SkipException: Could not find a response for test point Server URL: https://maps.ecere.com/ogcapi , Path: /collections, Replacements: {}

validate Feature Collections Metadata Operation Response_ Crs Property
org.testng.SkipException: Could not find a response for test point Server URL: https://maps.ecere.com/ogcapi , Path: /collections, Replacements: {}

validate Feature Collections Metadata Operation Response_ Content
org.testng.SkipException: Could not find a response for test point Server URL: https://maps.ecere.com/ogcapi , Path: /collections, Replacements: {}

And in CRS:

verify Collections Collection Crs Identifier Of Storage Crs
java.lang.RuntimeException: java.util.NoSuchElementException: No value present

verify Collections Path Collection Crs Property Contains Default Crs
java.lang.RuntimeException: java.util.NoSuchElementException: No value present

verify Collections Collection Crs Identifier Of Crs Property
java.lang.RuntimeException: java.util.NoSuchElementException: No value present

verify Collections Path Collection Crs Property Contains Default Crs
java.lang.RuntimeException: java.util.NoSuchElementException: No value present

[dstenger]

Thank you for reporting.
We are already working on a solution for this problem. It seems to be environment specific.

[dstenger]

@jerstlouis We did some updates on the two mentioned environments (https://cite.opengeospatial.org/teamengine/ and https://cite.opengeospatial.org/te2/) today.
Now, the test suite should run without encountering OutOfMemoryErrors.
Can you please check if your problems are solved?

[jerstlouis]

@dstenger Thank you for the update.
We ran the same test on both the production and beta CITE environments successfully without encountering Out of Memory issues this time around.

Testing time is extremely long though (about 30 minutes), even testing with only 3 collections, and I don't think it is because of processing or data transfer time on our end. It is also a lot more time than the sum of the duration of the tests, most of which are under 1s except for 4 that take up to ~1min30s. There is also no progress bar of any kind to tell us how far along we are in the set of tests.

I would also like to point out that when the OoM issue was happening, our implementation was reported as failing Part 1: Core but passing Part 2: CRS successfully. After the OoM issue was fixed, we passed Part 1: Core, but failed Part 2: CRS because we had not yet implemented bbox-crs.

Should Part 2: CRS perhaps not be tested if Part 1: Core failed? Or perhaps the CRS tests need to be modified so that it does not succeed if the important tests were skipped?

After implementing support for bbox-crs, our implementation now passes both Part 1: Core and Part 2: CRS (on both production and beta environments).

[dstenger]

We ran the same test on both the production and beta CITE environments successfully without encountering Out of Memory issues this time around.

Very good, thank you for the feedback.

Testing time is extremely long though (about 30 minutes), even testing with only 3 collections, and I don't think it is because of processing or data transfer time on our end. It is also a lot more time than the sum of the duration of the tests, most of which are under 1s except for 4 that take up to ~1min30s. There is also no progress bar of any kind to tell us how far along we are in the set of tests.

The Console opening in a separate window after starting the tests should display the progress of the test suite. However, I realize that the output is limited there. If this is important for you, can you please open a new issue for that?

I would also like to point out that when the OoM issue was happening, our implementation was reported as failing Part 1: Core but passing Part 2: CRS successfully. After the OoM issue was fixed, we passed Part 1: Core, but failed Part 2: CRS because we had not yet implemented bbox-crs.

The behavior of the test suites before and after the OoM issue should be identical. The latest update of this test suite on Beta (te2) was done in July 2021 and on Production in September 2021.

Should Part 2: CRS perhaps not be tested if Part 1: Core failed? Or perhaps the CRS tests need to be modified so that it does not succeed if the important tests were skipped?

Both parts act independently from each other. We already introduced dependencies between tests if necessary. If you see any potential for improvements here, please open a new issue.

After implementing support for bbox-crs, our implementation now passes both Part 1: Core and Part 2: CRS (on both production and beta environments).

Perfect, good to hear.

Can you please go through my comments and open further issues if necessary? I propose to close this issue afterwards as the problem described in this issue was solved.

[jerstlouis]

@dstenger Thank you, I have opened the following issues:

• Improved feedback on progress of tests #182 on the lack of progress feedback
• Improve tests performance #183 on the extremely long delays
• Part 2: CRS should fail if it key tests were skipped #184 on CRS passing when it should not

Closing this issue which has been resolved. Thank you!