MeerKAT S-band data - error in obsconf worker

[viralp]

I am trying to reduce MeerKAT S-band data but getting the error in obsconf worker -

2024-12-10 15:48:06 CARACal INFO: obsconf: initializing
2024-12-10 15:48:06 CARACal.Stimela.listobs-ms0 INFO: job started at 2024-12-10 15:48:06.463528
# Illegal option --
# /usr/bin/xvfb-run: 184: Usage:: not found
2024-12-10 15:48:07 CARACal.Stimela.listobs-ms0 ERROR: /usr/bin/singularity run --workdir /lustre/aoc/users/vparekh/VLASS_selfcal/MeerKAT_S-band/.stimela_workdir-17338708863148015 --contain returns error code 127
2024-12-10 15:48:07 CARACal.Stimela.listobs-ms0 ERROR: job failed at 2024-12-10 15:48:07.331850 after 0:00:00.868322
2024-12-10 15:48:07 CARACal ERROR: Job 'listobs-ms0:: Get observation information ms=COSMO-S2_3_4.ms' failed: /usr/bin/singularity run --workdir /lustre/aoc/users/vparekh/VLASS_selfcal/MeerKAT_S-band/.stimela_workdir-17338708863148015 --contain returns error code 127 [PipelineException]
2024-12-10 15:48:07 CARACal INFO:   More information can be found in the logfile at output/logs-20241210-154805/log-caracal.txt
2024-12-10 15:48:07 CARACal INFO: exiting with error code 1

log-caracal.txt
log-obsconf-listobs-ms0-20241210-154805.txt
S_band_data.txt

I test that the same script (S_band_data.txt) can work for L-band data (with same CARACAL and stimela version) without any error. I also confirmed that the CASA's listobs task work fine externally for the S-band ms, but just not working with the CARACAL. None of the file (summary json or elevation plot) generated in the obsconf worker for S-band data. Any input here?

[Athanaseus]

Hi @viralp, thank you for reporting this.

It's strange that this works for a different MS file, as it seems to fail to execute the singularity container.
I also noticed the virtualenv is Py3.6 (Version 1.0.7 installed at /users/vparekh/caracal-venv/lib/python3.6/site-packages/caracal ) while the latest caracal version (1.1.4) supports Python >= 3.8.
Can you please reconfirm that this issue remains even after updating to at least Python3.8?

Best regards

[viralp]

Hi Athanaseus,

Yes the problem is same with 1.1.4. I tried with the another ms file (S-band), but the same issue with the obsconf worker. If you want then I can give you the token for downloading the same ms file. Thanks.
log-caracal.txt

[Athanaseus]

Hi Viral,

I’m unable to reproduce this error with the latest caracal and singularity-3.7.0.
However, I noticed it may be related to #958.
Could you provide the version of singularity you are using?

PS: Also consider clearing singularity cache before pulling the images with: singularity cache list && singularity cache clean

[viralp]

Hi Athanaseus,

That's weird. I am using the latest Caracal
2024-12-17 12:32:35 CARACal INFO: Version 1.1.4 installed at /home/viralp/venv3.8/lib64/python3.8/site-packages/caracal

Singularity version 4.1.5
(venv3.8) viralp$ singularity --version
singularity-ce version 4.1.5-1.el8

Stimela version 1.7.10
(venv3.8) viralp$ stimela --version
stimela version 1.7.10