Skip to content

Commit

Permalink
[SPARK-50908][SS][PYTHON] Disable flaky TTL test in `test_pandas_tran…
Browse files Browse the repository at this point in the history
…sform_with_state.py`

### What changes were proposed in this pull request?

Disable the flaky TTL suite for TransformWithStateInPandas.

### Why are the changes needed?

The suite is flaky on CI because for this test case, we want to test on scenarios of one key expires and the other key is still in the state store. The tricky thing is that we don't have equivalent of `StreamManualClock` in Scala and we have to manually set a Thread.sleep() time duration such that the TTL expires after a certain time duration while the other key does not. As the batch duration varies a lot on CI, this suite is flaky. We decided to disable the suite until we find a holistic fix.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested on CI.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49634 from jingz-db/fix-ttl-suite.

Authored-by: jingz-db <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
  • Loading branch information
jingz-db authored and HyukjinKwon committed Jan 24, 2025
1 parent 2b657fe commit 8f66aef
Showing 1 changed file with 2 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -330,9 +330,8 @@ def check_results(batch_df, batch_id):
SimpleTTLStatefulProcessor(), check_results, False, "processingTime"
)

@unittest.skipIf(
"COVERAGE_PROCESS_START" in os.environ, "Flaky with coverage enabled, skipping for now."
)
# TODO SPARK-50908 holistic fix for TTL suite
@unittest.skip("test is flaky and it is only a timing issue, skipping until we can resolve")
def test_value_state_ttl_expiration(self):
def check_results(batch_df, batch_id):
if batch_id == 0:
Expand Down

0 comments on commit 8f66aef

Please sign in to comment.