cannot run the test sample (OSError: [Errno 12] Cannot allocate memory; multiprocessing)

[cjhong]

I followed the instructions and successfully installed SNAF in the py=3.7 conda environment.

I tested the software using the example posted in the test directory. I enabled 'test T antigen function'.

My Linux machine specs are
Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
192GB RAM

I got this error message,

---

(py3.7_snaf) @:test$ python ./analysis.py
2025-01-28 18:06:22 starting initialization
Current loaded gtex cohort with shape (12827, 2629)
Adding cohort tcga_control with shape (12902, 705) to the database
now the shape of control db is (12908, 3334)
Adding cohort gtex_skin with shape (12098, 313) to the database
now the shape of control db is (12908, 3647)
2025-01-28 18:06:56 finishing initialization
-------------pass initiate test----------------
reduce valid NeoJunction from 12927 to 840 because they are present in GTEx
reduce valid Neojunction from 840 to 668 because they are present in added control tcga_control
reduce valid Neojunction from 668 to 597 because they are present in added control gtex_skin
junction_count_matrix: (12927, 2)
cores: 4
valid: 597
invalid: 12330
cond_df: (12927, 2)
subset: (597, 2)
translated: list of 597 nj objects
cond_subset_df: (597, 2)
results: list of length None
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 149/149 [03:45<00:00, 1.51s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 149/149 [03:52<00:00, 1.56s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 150/150 [03:53<00:00, 1.55s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 149/149 [03:57<00:00, 1.59s/it]
adding gene symbol
Input sequence provided is already in string format. No operation performed
113 input query terms found dup hits: [('ENSG00000184697', 3), ('ENSG00000204531', 4), ('ENSG00000237978', 4), ('ENSG00000187266', 4), ('E
5 input query terms found no hit: ['ENSG00000243944', 'ENSG00000243944', 'ENSG00000243944', 'ENSG00000243944', 'ENSG00000243944']
adding chromosome coordinates
adding tumor specificity mean raw count
Traceback (most recent call last):
File "./analysis.py", line 29, in
snaf.JunctionCountMatrixQuery.generate_results(path='./result/after_prediction.p',outdir='./result')
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/site-packages/snaf/snaf.py", line 456, in generate_results
enhance_frequency_table(df,True,True,outdir,'frequency_stage{}_verbosity1_uid_gene_symbol_coord_mean_mle.txt'.format(stage))
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/site-packages/snaf/snaf.py", line 1407, in enhance_frequency_table
df = add_tumor_specificity_frequency_table(df,'mean',False)
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/site-packages/snaf/gtex.py", line 339, in add_tumor_specificity_frequency_table
pool = mp.Pool(processes=cores)
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/multiprocessing/context.py", line 119, in Pool
context=self.get_context())
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/multiprocessing/pool.py", line 176, in init
self._repopulate_pool()
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
w.start()
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home//miniforge3/envs/py3.7_snaf/lib/python3.7/site-packages/snaf/dash_app/app.py", line 20, in clear_assets
imgs = os.listdir('assets')
FileNotFoundError: [Errno 2] No such file or directory: 'assets'

[frankligy]

Hi @cjhong,

Sorry for the inconvenience, but just as the error messages suggest, it's likely a memory issue. I understand you said the total RAM can be as high as 192GB, and this testing case should not be taking that much memory as well, you can refer to this issue page which we did a thorough investigation on that (#27).

Maybe testing around by modifying the cores parameters or just further subset the input matrix to an extremely small size and see if it can run through. There are also some commands that you can use to check the free RAM on your system that you can google it, or if there's any weird global restrictions for your python env, for instance, when you say can not allocate 37.8M for an array, that is very strange, in my case, I only see this when over 10GB or something, so see if this can be modified using some numpy function.

Hoping this helps a bit,
Frank

[cjhong]

Thank you! it seems work now