OSError: [Errno 36] File name too long #29

MrBleem · 2023-02-02T12:15:02Z

Hello!
After get the expected result with the S2R data, I tried run TELR on other fly data.
Then I got the OSError:
OSError: [Errno 36] File name too long: '/home/boxu/temp/telr/result/intermediate_files/input_json/chr2L_1887214_1887214_21128_24591_ID=162;FBgn0000638_FB|ID=163;FBgn0000638_FB|ID=164;FBgn0000638_FB|ID=165;FBgn0000638_FB|ID=166;FBgn0000638_FB|ID=167;FBgn0000638_FB|ID=168;FBgn0000638_FB|ID=169;FBgn0000638_FB|ID=170;FBgn0000638_FB|ID=171;FBgn0000638_FB_input.json'

And I noticed that some insertion annotated with one type of TE but different ID in the result, just like:"ID=12;Tirant|ID=13;Tirant". Is it causing the problem？

Dose it means there is mulitple insertion seq from same TE? I'm confused for that.

Could you help me solve this?

Thank you very much!

The text was updated successfully, but these errors were encountered:

unavailable-2374 · 2023-02-05T05:19:48Z

Hello

I got the same problem , File name too long

Perform liftover...
[Errno 36] File name too long: '/public/home/caoshuo/project/pan_TE/MGX/intermediate_files/tmp/PN7_19261399_19261399_15469_21382_TE_00000005_TE_00000478_TE_00000717_TE_00000843_TE_00000888_TE_00001002_TE_00001026_TE_00001042_TE_00001832_TE_00001973_TE_00002053_TE_00002145_TE_00002148_TE_00002206_TE_00002384_TE_00007428_TE_00007866_TE_00008427_TE_00009810_TE_00010608_TE_00011078_TE_00011095_5p.fa'
liftover failed, exiting...

and also I find the log file shows

Error: Sorted input specified, but the file /public/home/caoshuo/project/pan_TE/MGX/intermediate_files/PN.fa.te.bed has the following out of order record

Should I clean similar sequence of rep.fa, or there is something should be changed in the script ?

Thank you!

shunhuahan · 2023-02-07T07:02:59Z

Hi @MrBleem, based on the error message, it appears that you ran TELR with a TE library that includes multiple identical or highly similar sequences under the same TE family. The current TELR assumes a TE library that contains one TE sequence for each family. I would suggest you to clean up the library and remove redundant sequences before running TELR. For D. melanogaster genome sequencing data, we suggest using v10.2 of the D. melanogaster canonical TE sequence library available at https://github.com/bergmanlab/drosophila-transposons/blob/master/releases/D_mel_transposon_sequence_set_v10.2.fa.

Hi @unavailable-2374, thanks for sharing the failed log. Yes, the issue appears to be caused by redundant TE sequences in the library described above. I agree with your intuition to clean redundant/similar sequences and try again.

Sorry for not making that clear in the README. Let me know if the issue still exists after the library clean up.

Best,
Shunhua

unavailable-2374 · 2023-02-07T07:40:36Z

Oh, thank you! I will try it right now.

Best,
Shuocao

MrBleem · 2023-02-07T08:04:54Z

Ok, I will change the TE reference file.
Thank you very much !

Best
Bo

unavailable-2374 · 2023-02-07T16:04:58Z

Hi @MrBleem, based on the error message, it appears that you ran TELR with a TE library that includes multiple identical or highly similar sequences under the same TE family. The current TELR assumes a TE library that contains one TE sequence for each family. I would suggest you to clean up the library and remove redundant sequences before running TELR. For D. melanogaster genome sequencing data, we suggest using v10.2 of the D. melanogaster canonical TE sequence library available at https://github.com/bergmanlab/drosophila-transposons/blob/master/releases/D_mel_transposon_sequence_set_v10.2.fa.

Hi @unavailable-2374, thanks for sharing the failed log. Yes, the issue appears to be caused by redundant TE sequences in the library described above. I agree with your intuition to clean redundant/similar sequences and try again.

Sorry for not making that clear in the README. Let me know if the issue still exists after the library clean up.

Best, Shunhua

Hello

Very unfortunately, I tried to merge the sequences with similarity 85+ in TE_lib with cd-hit and run TELR with the result as a library, but I still get "File name too long" errors.

Best, ShuoCao

shunhuahan · 2023-02-08T04:57:37Z

Thanks for reporting that the error persists @unavailable-2374. I will try to push a code update later this week to have this issue fixed.

unavailable-2374 · 2023-02-08T05:46:01Z

Thanks for reporting that the error persists @unavailable-2374. I will try to push a code update later this week to have this issue fixed.

Thanks!

shunhuahan · 2023-02-12T04:15:20Z

Hi @unavailable-2374 and @MrBleem,

I have merged a fix into the master branch on the "file name too long" error (see #30). Please pull the latest commit from the master branch, reinstall TELR locally, and see if this new version can resolve the issue. Thanks!

cd TELR # this is the TELR git folder
git pull # necessary to use this patched update

conda activate TELR
pip install . # install this patched update

# run TELR
telr -h

Best,
Shunhua

unavailable-2374 · 2023-02-13T02:08:25Z

Hi Shunhua

Thank, I will try it immediately！

Shuo

unavailable-2374 · 2023-02-14T01:02:20Z

git pull

Hi Shunhua

Unfortunately, what if I said that the error still exists?

[caoshuo@login04 pan_TE]$ tail -n 3 aug.out
Perform liftover...
[Errno 36] File name too long: '/public/home/caoshuo/project/pan_TE/MGX/intermediate_files/tmp/PN7_19261399_19261399_12323_18236_TE_00000005_TE_00000478_TE_00000717_TE_00000843_TE_00000888_TE_00001002_TE_00001026_TE_00001042_TE_00001832_TE_00001973_TE_00002053_TE_00002145_TE_00002148_TE_00002206_TE_00002384_TE_00007428_TE_00007866_TE_00008427_TE_00009810_TE_00010608_TE_00011078_TE_00011095_5p.fa'
liftover failed, exiting...
[caoshuo@login04 pan_TE]$ cd ~/tools/TELR/
[caoshuo@login04 TELR]$ git pull 
Already up-to-date.

Also reported in the additional .err file:

Error: Sorted input specified, but the file /public/home/caoshuo/project/pan_TE/MGX/intermediate_files/PN.fa.te.bed has the following out of order record

shunhuahan · 2023-02-14T02:58:25Z

Hi @unavailable-2374,

The error message indicates that the new TELR version was not being used in the new run. Could you double check that you have locally pip installed the new version of TELR after git pull?

cd TELR # this is the TELR git folder
conda activate TELR # active TELR environment
pip install . # install the latest update locally

# run TELR
telr -h

If you previously installed TELR through conda/mamba. You would notice that the telr version is the following from bioconda channel when listing out all the dependencies using conda list.

conda activate TELR
conda list
...
telr                      0.2                pyhdfd78af_2    bioconda
...

After you pip install TELR locally, you would see that the TELR version is now from pip when you check the environment dependencies.

conda activate TELR
conda list
...
telr                      1.0                      pypi_0    pypi
...

Hopefully this helps confirm the new pip version of TELR has been installed successfully.

Let me know if you still encounter issues. Sorry about the trouble.

Best,
Shunhua

unavailable-2374 · 2023-02-15T01:38:18Z

Hi @shunhuahan

Traceback (most recent call last):
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/bin/telr", line 8, in <module>
    sys.exit(main())
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/lib/python3.6/site-packages/telr/telr.py", line 158, in main
    thread=args.thread,
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/lib/python3.6/site-packages/telr/TELR_te.py", line 924, in find_te
    telr_mode=True,
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/lib/python3.6/site-packages/telr/TELR_liftover.py", line 1042, in liftover
    with open(liftover_input, "w") as output:
OSError: [Errno 36] File name too long: '/public/agis/zhouyongfeng_group/caoshuo/data/TE/TELR/Vzc/intermediate_files/input_json/PN10_11553286_11553288_14059_15463_CACTA-nMITE:CACTA-nMITE1923|CACTA-nMITE:CACTA-nMITE1960|CACTA-nMITE:CACTA-nMITE1963|CACTA-nMITE:CACTA-nMITE3229|CACTA-nMITE:CACTA-nMITE846|CACTA-nMITE:CACTA-nMITE946|DIRS:DIRS3534|Gypsy:Gypsy301|Gypsy:Gypsy437|L1:L1225|MITE:MITE1319|MITE:MITE1890|MITE:MITE3676_input.json'

I have double checked this, sorry that the problem still here.

Best,
ShuoCao

shunhuahan · 2023-02-15T02:41:35Z

Hi @unavailable-2374,

I realized the previous update only partially fixes the issue. The error you posted in #29 (comment) and #29 (comment) should be gone already with the previous update at #30. The most recent error message is caused by the intermediately JSON file containing too many TE family names. I have made another pull request to fix this issue (see #32).

Please do git pull, conda activate TELR, pip install ., run TELR again and check if the most recent update works. Sorry for the back and forth on this. Kinda tricky to do the bug fix without having the input files that raise the error for testing.

Best,
Shunhua

cbergman · 2023-05-03T15:44:24Z

Hi @MrBleem and @unavailable-2374

We have updated the TELR recipe at bioconda with a new version (v1.1) that includes the fix to the filename problem you identified. You can install the latest version of TELR via bioconda as follows:

mamba create -n TELR --channel bioconda telr=1.1

Please let us know if this resolves your issue.

Best regards,
Casey

shunhuahan mentioned this issue Feb 12, 2023

Fix file name too long issue #30

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OSError: [Errno 36] File name too long #29

OSError: [Errno 36] File name too long #29

MrBleem commented Feb 2, 2023 •

edited

Loading

unavailable-2374 commented Feb 5, 2023 •

edited by cbergman

Loading

shunhuahan commented Feb 7, 2023

unavailable-2374 commented Feb 7, 2023

MrBleem commented Feb 7, 2023

unavailable-2374 commented Feb 7, 2023

shunhuahan commented Feb 8, 2023

unavailable-2374 commented Feb 8, 2023

shunhuahan commented Feb 12, 2023 •

edited

Loading

unavailable-2374 commented Feb 13, 2023

unavailable-2374 commented Feb 14, 2023 •

edited by cbergman

Loading

shunhuahan commented Feb 14, 2023

unavailable-2374 commented Feb 15, 2023 •

edited by cbergman

Loading

shunhuahan commented Feb 15, 2023

cbergman commented May 3, 2023 •

edited

Loading

OSError: [Errno 36] File name too long #29

OSError: [Errno 36] File name too long #29

Comments

MrBleem commented Feb 2, 2023 • edited Loading

unavailable-2374 commented Feb 5, 2023 • edited by cbergman Loading

shunhuahan commented Feb 7, 2023

unavailable-2374 commented Feb 7, 2023

MrBleem commented Feb 7, 2023

unavailable-2374 commented Feb 7, 2023

shunhuahan commented Feb 8, 2023

unavailable-2374 commented Feb 8, 2023

shunhuahan commented Feb 12, 2023 • edited Loading

unavailable-2374 commented Feb 13, 2023

unavailable-2374 commented Feb 14, 2023 • edited by cbergman Loading

shunhuahan commented Feb 14, 2023

unavailable-2374 commented Feb 15, 2023 • edited by cbergman Loading

shunhuahan commented Feb 15, 2023

cbergman commented May 3, 2023 • edited Loading

MrBleem commented Feb 2, 2023 •

edited

Loading

unavailable-2374 commented Feb 5, 2023 •

edited by cbergman

Loading

shunhuahan commented Feb 12, 2023 •

edited

Loading

unavailable-2374 commented Feb 14, 2023 •

edited by cbergman

Loading

unavailable-2374 commented Feb 15, 2023 •

edited by cbergman

Loading

cbergman commented May 3, 2023 •

edited

Loading