No way to get xcresult from running logic tests

[segiddins]

xctestrunner currently assumes that any test without a test host (a logic test) must be run directly via the xctest agent, and not via xcodebuild test-without-building. This means there's no way to get an xctestresult bundle from logic tests, making it much harder to analyze what happens during the run of a log test.

Changing xctest_session.py to not check if test_type != ios_constants.TestType.LOGIC_TEST, and unconditionally run logic tests via xcodebuild seems to work just fine, and is "just" slower:

Direct:
Stats over 10 runs: max = 1.0s, min = 0.9s, avg = 0.9s, dev = 0.0s

Xcodebuild:
Stats over 10 runs: max = 2.9s, min = 2.7s, avg = 2.7s, dev = 0.1s

Xcodebuild, with the sleeps shortened:
Stats over 10 runs: max = 2.5s, min = 2.2s, avg = 2.2s, dev = 0.1s

Would an option to run logic tests via xcodebuild be acceptable?

[segiddins]

@albertdai would you be open to an option to force running logic tests via xctestrun?

[albertdai]

Samuel, I could add an option to run logic test via xctestrun. It means that you need to pass an extra flag to bazel test command to enable that. Is that OK for you?

Why we don't use xctestrun for logic test is because we found the current way is much faster than the xcodebuild xctestrun for logic test.

[segiddins]

It means that you need to pass an extra flag to bazel test command to enable that. Is that OK for you?

Yup, that's fine. I know that xctestrun is slower, but in our case we think the trade-off (better reporting) is worth it

[brentleyjones]

We would also love this feature.

[tinder-maxwellelliott]

Would love to have this as well over forking our reporting logic

[tinder-maxwellelliott]

I started looking into this and I am seeing a much longer slowdown, around 5x slower and I could visibly see a simulator launching for a logic only test

@segiddins did you see a simulator launch in your testing?

[dostrander]

@tinder-maxwellelliott Yeah this is a very similar experience we have