[Errno 24] Too many open files - Same Setup new Issue

[SSj-Saturn]

Related to: #470

As noted in the previous issue, checked my ulimit and it's unlimited, so don't think it's related to that.

Describe the bug

Running command line, or using the GUI interface with multiple models now crashes out, this wasn't an issue previously and the config hasn't changed other than updating to the latest version of ofscraper.

To Reproduce

1. Run OFscraper
2. Select to download multiple accounts, unsure of quanity or amount, seems hit or miss
3. Eventually after a few minutes it'll crash out with error messaging, putting you back to the main menu.

Expected behavior

Shouldn't crash out or give more precise error messaging (and a way to resolve).

Screenshots/Logs

https://privatebin.io/?13d74935b84ef0b6#BU3ssssM4CCvuchFUoCkBVRW4P5Ntzttc1cjyRuZLDZo

Config

https://privatebin.io/?d887e027c0ace75d#6XFukrr6edebp1LDiX1PiFoVnyiDKEBXqPTcqYcVSheG

System Info

• OS: OSX
• Browser: Chrome
• Version [e.g. 22]
• Binary

Additional context

Add any other context about the problem here.

[Puk0]

Ponle:
"download-sem": 4,
y
"threads": 1

[SSj-Saturn]

Changed that, didn't solve the problem.

[downloadnormal.process_dicts:45] Downloading in single thread mode downloadnormal.py:45 [Errno 24] Too many open files: '/Users/[USERNAME]/.config/ofscraper/config.json' [Errno 24] Too many open files: '/Users/[USERNAME]/.config/ofscraper/config.json' [Errno 24] Too many open files: '/Users/[USERNAME]/.config/ofscraper/config.json' [Errno 24] Too many open files: '/Users/[USERNAME]/.config/ofscraper/config.json' [consumer.consumer:46] PID: 13590 Media:3011114377 Post:729247532 Download Failed because consumer.py:46 [Errno 24] Too many open files: '/Users/your_username/.config/ofscraper/main_profile/locks/rich.lock' [consumer.consumer:46] PID: 13590 Media:3036328169 Post:754995482 Download Failed because consumer.py:46 [Errno 24] Too many open files: '/Users/your_username/.config/ofscraper/main_profile/locks/rich.lock' [close.sendCloseMessage:49] Finished Script

[gri1n]

I got this working by increasing the open file limit but it'd be better if the scraper were fixed so it doesn't use such an absurd number of file descriptors.

[SSj-Saturn]

I got this working by increasing the open file limit but it'd be better if the scraper were fixed so it doesn't use such an absurd number of file descriptors.

Was this with the solve via ulimit?

https://askubuntu.com/questions/1182021/too-many-open-files

As yeah not sure what's changed as this wasn't an issue before now, and I've been running it via the same commands for over a year, not sure what could have changed otherwise.

[gri1n]

Yep, I changed it permanently in /etc/security/limits.conf.

[WyohKnott]

Symptom: You have an error message : Too many open files while running the script with lots of contents/creators.

Solution: There are multiple things to change:

1. The kernel system-wide limits:

In /etc/sysctl.d/60-file-max.conf as root:

fs.file-max = 2097152
fs.inotify.max_user_instances = 8192

Reload with sudo sysctl --system, you should see in the list:

fs.file-max = 2097152
fs.inotify.max_user_instances = 8192

2. User limits

You can set it up in /etc/security/limits.d/60-ulimits.conf as root:

* soft nofile 1048576
* hard nofile 2097152

• Soft limit (ulimit -nS): This is the current limit that a user can hit before being warned or denied to open more files. It can be adjusted by the user up to the hard limit without needing elevated permissions.
• Hard limit (ulimit -nH): This is the maximum number of file descriptors a user can have open, and it can only be increased by root. The hard limit should not exceed fs.file-max.

The * in the beginning of the line means all users have these rules, but you can limit it to a specific user.

The change will apply on next login, and you can check the limits with: ulimit -nS and ulimit -nH.