BOT CRASHED ON DOWNLOAD DURING LARGE DATA REQUEST

[talkaal]

A 38 GB (single files each) file torrent was put but it happened to crash after a while.

As well as two feature request also if possible add!

• SEEDS/LEECH amount (could be able to estimate health of torrent)
• Download percentage of Torrents

A minor 🐞Bug

• Thumbnail generation in video before upload is much slower.

Log of the activity ( HEROKU)

2020-09-29T10:26:57.191869+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2020-09-29T10:26:57.191869+00:00 app[worker.1]:
2020-09-29T10:26:57.191869+00:00 app[worker.1]: Traceback (most recent call last):
2020-09-29T10:26:57.191870+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 110, in progress_dl
2020-09-29T10:26:57.191870+00:00 app[worker.1]: await message.edit(
2020-09-29T10:26:57.191871+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/types/messages_and_media/message.py", line 2507, in edit_text
2020-09-29T10:26:57.191872+00:00 app[worker.1]: return await self._client.edit_message_text(
2020-09-29T10:26:57.191873+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/methods/messages/edit_message_text.py", line 78, in edit_message_text
2020-09-29T10:26:57.191873+00:00 app[worker.1]: r = await self.send(
2020-09-29T10:26:57.191874+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/client.py", line 1424, in send
2020-09-29T10:26:57.191874+00:00 app[worker.1]: r = await self.session.send(data, retries, timeout, self.sleep_threshold)
2020-09-29T10:26:57.191874+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 451, in send
2020-09-29T10:26:57.191875+00:00 app[worker.1]: return await self.send(data, retries - 1, timeout)
2020-09-29T10:26:57.191875+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 429, in send
2020-09-29T10:26:57.191876+00:00 app[worker.1]: return await self._send(data, timeout=timeout)
2020-09-29T10:26:57.191876+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 399, in _send
2020-09-29T10:26:57.191877+00:00 app[worker.1]: RPCError.raise_it(result, type(data))
2020-09-29T10:26:57.191877+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/errors/rpc_error.py", line 79, in raise_it
2020-09-29T10:26:57.191878+00:00 app[worker.1]: raise getattr(
2020-09-29T10:26:57.191878+00:00 app[worker.1]: pyrogram.errors.exceptions.bad_request_400.MessageNotModified: [400 MESSAGE_NOT_MODIFIED]: The message was not modified (caused by "messages.EditMessage")

[azamaulanaaa]

everything you ask for are possible, it just me don't add it. I will add it soon.

A minor 🐞Bug

• Thumbnail generation in video before upload is much slower.

for thumbnail is so slow due to ffmpeg needs to seek to the middle of video to generates thumbnail. from many post I read, it is possible to just jump to the middle to snapshot instead of seek but the drawback is sometimes it does not work, in my case mostly does not work. the solution I knew is snapshot the first frame, the drawback it does not represent the video. I still working with it along side optimization for video split.

A 38 GB (single files each) file torrent was put but it happened to crash after a while.

for large file, as far as I know is due to limitation from the heroku itself. I did 5gb+ and most the time the file gone from the service. the app already optimize for binary split ( like hjsplit does) but it just help upto 4-5gb files. I am still working on this issue on my other project torrent-mirror with nodejs since python does not have torrent module natively

[talkaal]

CRASHED AGAIN
THIS TIME AS PER YOU HAVE TOLD 3GB TORRENT I'VE ADDED.

LOGS

2020-09-29T18:48:17.398735+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/message.py", line 578, in get_content_type
2020-09-29T18:48:17.398737+00:00 app[worker.1]: value = self.get('content-type', missing)
2020-09-29T18:48:17.398737+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/message.py", line 471, in get
2020-09-29T18:48:17.398738+00:00 app[worker.1]: return self.policy.header_fetch_parse(k, v)
2020-09-29T18:48:17.398738+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/_policybase.py", line 316, in header_fetch_parse
2020-09-29T18:48:17.398738+00:00 app[worker.1]: return self._sanitize_header(name, value)
2020-09-29T18:48:17.398739+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/_policybase.py", line 287, in _sanitize_header
2020-09-29T18:48:17.398739+00:00 app[worker.1]: if _has_surrogates(value):
2020-09-29T18:48:17.398740+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/utils.py", line 57, in _has_surrogates
2020-09-29T18:48:17.398740+00:00 app[worker.1]: s.encode()
2020-09-29T18:48:17.398741+00:00 app[worker.1]: RecursionError: maximum recursion depth exceeded while calling a Python object
2020-09-29T18:48:17.398741+00:00 app[worker.1]:
2020-09-29T18:48:17.398742+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2020-09-29T18:48:17.398742+00:00 app[worker.1]:
2020-09-29T18:48:17.398743+00:00 app[worker.1]: Traceback (most recent call last):
2020-09-29T18:48:17.398743+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 124, in progress_dl
2020-09-29T18:48:17.398744+00:00 app[worker.1]: return await progress_dl(message, aria2_api, gid, text)
2020-09-29T18:48:17.398745+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 139, in progress_dl
2020-09-29T18:48:17.398745+00:00 app[worker.1]: download.remove(force=True)
2020-09-29T18:48:17.398754+00:00 app[worker.1]: UnboundLocalError: local variable 'download' referenced before assignment

Minor UI Alteration if possible

• Download progress has two "%' symbol

Future feature addition Request (Just a thought)

• The download progress blocks be extended to 20 rather than currently 10.

• Add to Specific telegram group for usage rather than individual bot interaction. Multiple users if using the same bot run on the script, the connection to bot will be more and may result in Speed reduction on individual bot interaction.

• In group this problem might be rectified as group will be a single interactor for request providing files.

• A command to know available server space and another command to clear residual downloads in the server still remaining this can be used to add torrent after previous file request are uploaded

Thank you for your Interaction based quick response and necessary updates.
Hope that I'm not pressurizing you😅

[azamaulanaaa]

hi @talkaal, mind if you share the torrent? so I can replicate your bug

[azamaulanaaa]

• The download progress blocks be extended to 20 rather than currently 10.

let me give you option to customize it from local/default.py, please wait.

• Add to Specific telegram group for usage rather than individual bot interaction. Multiple users if using the same bot run on the script, the connection to bot will be more and may result in Speed reduction on individual bot interaction.
• In group this problem might be rectified as group will be a single interactor for request providing files.
  you can actually add the bot to specific group.

1. set the password
2. add the bot to the group
3. verify the bot at that group using command /pass [yourpass]

• A command to know available server space

I can actually do it but there is a catch. Since most service using shared storage, it might not represent actual storage space for you.

another command to clear residual downloads in the server still remaining this can be used to add torrent after previous file request are uploaded

Pardon me, I do not understand.

Note : please create new issue with label enhancement for requested feature. thank you.

[talkaal]

hi @talkaal, mind if you share the torrent? so I can replicate your bug

Sure

magnet:?xt=urn:btih:15610ff6ee0d0fe11d491c805cf141235f184e32&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker4.itzmx.com%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fdenis.stalker.upeer.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Fbt.xxx-tracker.com%3A2710%2Fannounce&tr=udp%3A%2F%2Fthetracker.org%3A80%2Fannounce&tr=udp%3A%2F%2Fretracker.lanta-net.ru%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.uw0.xyz%3A6969%2Fannounce&dn=%5BSolidTorrents.net%5D+Sense8+Season+2+S02+720p+WEBRip+x265+HEVC+Crazy4ad

[talkaal]

Pardon me, I do not understand.

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

That's what I meant hope you understood?

[talkaal]

let me give you option to customize it from local/default.py, please wait.

Did you change to customizable variable?

[azamaulanaaa]

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

[azamaulanaaa]

@talkaal now, you can customize bar size on ./__init__.py. sorry not on local/default.py

[azamaulanaaa]

hi @talkaal, mind if you share the torrent? so I can replicate your bug

Sure

magnet:?xt=urn:btih:15610ff6ee0d0fe11d491c805cf141235f184e32&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker4.itzmx.com%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fdenis.stalker.upeer.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Fbt.xxx-tracker.com%3A2710%2Fannounce&tr=udp%3A%2F%2Fthetracker.org%3A80%2Fannounce&tr=udp%3A%2F%2Fretracker.lanta-net.ru%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.uw0.xyz%3A6969%2Fannounce&dn=%5BSolidTorrents.net%5D+Sense8+Season+2+S02+720p+WEBRip+x265+HEVC+Crazy4ad

looks like the torrent has no seeder

[azamaulanaaa]

sorry wrong commit reference

[talkaal]

The crash of large torrents after a while might be due to two factors of time I think.
Don't know whether it's helpful.

Maximum process timeout
Maximum time to wait for torrent to start

I hope it'll be a valuable thought.
Thank you for your effort on the code

PLEASE DO CLOSE IN CASE THE THOUGHT IS AN INVALID ONE.

[fernandooliveira2]

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

Any chance to youtube download too? 👍

[azamaulanaaa]

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

Any chance to youtube download too? 👍

sorry out of topic