failing benchmark "maize_2020_belgium"

[soxofaan]

last run at this time: https://github.com/ESA-APEx/apex_algorithms/actions/runs/11704050323/job/32595666231

failure due to job failure

0:11:22 Job 'cdse-j-2411063f288b4b6185abd01f6017ee66': running (progress N/A)
0:12:24 Job 'cdse-j-2411063f288b4b6185abd01f6017ee66': running (progress N/A)
0:13:24 Job 'cdse-j-2411063f288b4b6185abd01f6017ee66': running (progress N/A)
0:14:25 Job 'cdse-j-2411063f288b4b6185abd01f6017ee66': error (progress N/A)
Your batch job 'cdse-j-2411063f288b4b6185abd01f6017ee66' failed. Error logs:

[{'id': '[1730898304841, 766544]', 'time': '2024-11-06T13:05:04.841Z', 'level': 'error', 'message': 'Error communicating with MapOutputTracker'}, {'id': '[1730898307527, 929242]', 'time': '2024-11-06T13:05:07.527Z', 'level': 'error', 'message': 'Error communicating with MapOutputTracker'}, {'id': '[1730898719492, 309340]', 'time': '2024-11-06T13:11:59.492Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898719504, 672887]', 'time': '2024-11-06T13:11:59.504Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898726780, 545688]', 'time': '2024-11-06T13:12:06.780Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898726814, 514047]', 'time': '2024-11-06T13:12:06.814Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898727375, 4552]', 'time': '2024-11-06T13:12:07.375Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898727385, 687542]', 'time': '2024-11-06T13:12:07.385Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898733567, 520874]', 'time': '2024-11-06T13:12:13.567Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898733587, 678688]', 'time': '2024-11-06T13:12:13.587Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898746571, 876237]', 'time': '2024-11-06T13:12:26.571Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898746573, 66535]', 'time': '2024-11-06T13:12:26.573Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898760094, 477824]', 'time': '2024-11-06T13:12:40.094Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898760096, 997566]', 'time': '2024-11-06T13:12:40.096Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898773650, 225682]', 'time': '2024-11-06T13:12:53.650Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898773679, 161617]', 'time': '2024-11-06T13:12:53.679Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898782203, 343805]', 'time': '2024-11-06T13:13:02.203Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898782215, 682075]', 'time': '2024-11-06T13:13:02.215Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898792090, 765327]', 'time': '2024-11-06T13:13:12.090Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898792092, 764814]', 'time': '2024-11-06T13:13:12.092Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898793255, 482369]', 'time': '2024-11-06T13:13:13.255Z', 'level': 'error', 'message': 'Python worker exited unexpectedly (crashed)'}, {'id': '[1730898793287, 852032]', 'time': '2024-11-06T13:13:13.287Z', 'level': 'error', 'message': 'This may have been caused by a prior exception:'}, {'id': '[1730898793457, 74609]', 'time': '2024-11-06T13:13:13.457Z', 'level': 'error', 'message': 'Task 1 in stage 68.0 failed 4 times; aborting job'}, {'id': '[1730898793466, 568148]', 'time': '2024-11-06T13:13:13.466Z', 'level': 'error', 'message': 'Stage error: Job aborted due to stage failure: Task 1 in stage 68.0 failed 4 times, most recent failure: Lost task 1.3 in stage 68.0 (TID 4850) (10.42.21.133 executor 14): org.apache.spark.api.python.PythonException: Traceback (most recent call last):\n File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 830, in main\n process()\n File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 822, in process\n serializer.dump_stream(out_iter, outfile)\n File "/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 146, in dump_stream\n for obj in iterator:\n File "/usr/local/spark/python/lib/pyspark.zip/pyspark/util.py", line 81, in wrapper\n return f(*args, **kwargs)\n File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/utils.py", line 57, in memory_logging_wrapper\n return function(*args, **kwargs)\n File "/opt/openeo/lib/python3.8/site-packages/epsel.py", line 44, in wrapper\n return _FUNCTION_POINTERS[key](*args, **kwargs)\n File "/opt/openeo/lib/python3.8/site-packages/epsel.py", line 37, in first_time\n return f(*args, **kwargs)\n File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/geopysparkdatacube.py", line 569, in tile_function\n result_data = run_udf_code(code=udf_code, data=data)\n File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/udf.py", line 57, in run_udf_code\n return openeo.udf.run_udf_code(code=code, data=data)\n File "/opt/openeo/lib/python3.8/site-packages/openeo/udf/run_code.py", line 232, in run_udf_code\n func(data)\n File "", line 392, in apply_udf_data\n File "", line 367, in _execute\n File "", line 222, in _execute\n File "", line 358, in execute\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/presto/inference.py", line 283, in get_presto_features\n presto_model = Presto.load_pretrained_url(presto_url=presto_url, strict=False)\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/presto/presto.py", line 812, in load_pretrained_url\n presto_model_layers = torch.load(io.BytesIO(response.content), map_location=device)\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/torch/serialization.py", line 815, in load\n return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/torch/serialization.py", line 1033, in _legacy_load\n magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '{'.\n\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:561)\n\tat org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:767)\n\tat org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:749)\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:514)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:513)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)\n\tat scala.collection.Iterator.foreach(Iterator.scala:943)\n\tat scala.collection.Iterator.foreach$(Iterator.scala:943)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1431)\n\tat org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:320)\n\tat org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:734)\n\tat org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:440)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2088)\n\tat org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:274)\n\nDriver stacktrace:'}, {'id': '[1730898796509, 406061]', 'time': '2024-11-06T13:13:16.509Z', 'level': 'error', 'message': 'OpenEO batch job failed: UDF exception while evaluating processing graph. Please check your user defined functions. File "/opt/openeo/lib/python3.8/site-packages/openeo/udf/run_code.py", line 232, in run_udf_code\n func(data)\n File "", line 392, in apply_udf_data\n File "", line 367, in _execute\n File "", line 222, in _execute\n File "", line 358, in execute\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/presto/inference.py", line 283, in get_presto_features\n presto_model = Presto.load_pretrained_url(presto_url=presto_url, strict=False)\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/presto/presto.py", line 812, in load_pretrained_url\n presto_model_layers = torch.load(io.BytesIO(response.content), map_location=device)\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/torch/serialization.py", line 815, in load\n return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n File "/opt/spark/work-dir/dependencies/worldcereal_deps/torch/serialization.py", line 1033, in _legacy_load\n magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '{'.'}]
Full logs can be inspected in an openEO (web) editor or with connection.job('cdse-j-2411063f288b4b6185abd01f6017ee66').logs().