[FEATURE] Please consider re-enabling simpcity (and other questions)

[baccccccc]

Hello! First of all, I'm very happy to discover your fork, and thank you for taking up the torch!

Second, I would like to ask you to consider re-enabling simpcity. I understand your reasons, and I absolutely do not want to make life harder for anyone. But even if you (or them) throttle it to literally one request per minute (instead of blocking altogether), it will still go long way for me. I'm not in a hurry, just lazy.

Third, do you consider feature requests posted in the original repository? I did submit a bunch back in the days, and I'm happy to repost them here, if that's appropriate. But maybe it's redundant and you could just take a look over there. For instance, here's one that I understand might be quite a significant change but can be potentially helpful in the long run: Jules-WinnfieldX/CyberDropDownloader#864

[jbsparrow]

I appreciate you sponsoring me and taking the time to ask about this. Unfortunately I can't re-enable SimpCity, I have talked with the SimpCity admins and they do not want to allow us to re-enable it even if we implement hard ratelimits. We've been working on systems to reduce the impact that we have on SimpCity but the admins don't want to allow scraping at all.

I can definitely look at feature requests from the original repository but it would be easier if you make them here. I will take a look at making a config option for CSV unsupported URL files.

[baccccccc]

Too bad. Unfortunately, without simpcity support, the app is mostly useless for me. I still appreciate your work on this and will keep sponsoring because I think it's valuable work for the community. But I'll look forward for future progress on simpcity negotiations, and hopefully it will get resolved some day somehow.

[jbsparrow]

We're working on negotiations but for the time being we can't support SimpCity ☹️ We have built features to significantly reduce the impact of scraping but thus far have had no luck getting the go-ahead to release them.

[vajdao]

I just don’t understand how or why they claim their server is overloaded when all the content is stored on third-party clouds, not the forum itself. It’s like we’re just browsing through threads and seeing the links, and the script essentially does the same thing: it “scrolls” through and stores links, then uses those links to download content from clouds that have no direct connection to Simpcity at all. I think the problem lies elsewhere, like when many users don't use the program as intended. For example, one person might start multiple sessions of cyberdrop-dl targeting different forum threads, which would make it appear that the server is overloaded by this chain of parallel requests.

I’m not a coder, but would it be possible to work with Simpcity and hardcode the scraper to communicate with each other (forum and script) so that only one cyberdrop-dl connection/session/instance is allowed from a specific IP at any given time, thus prohibiting parallelization? Just an idea...

They do support a script though: ForumPostDownloader, but it doesn’t work as efficiently as this one.

[baccccccc]

I just don’t understand how or why they claim their server is overloaded

everyone might have their own reasons. One reason is, as you mentioned, some people might be hammering the forum with multiple instances of the app running in parallel. And in this case, there's very little the app can do, since one instance (running in one window on your PC) will be unaware of another instance (running in another instance) so they won't be able to coordinate.

but another reason is that the forum is likely supported by ads. If everyone were using apps and scripts like this, nobody would be seeing those ads, and hence the forum owners won't be able to pay for hosting and other expenses. So, it's understood that they would prefer everyone to just work with the forum using a web browser.

and on top of that, if you're mostly using the app to passively crawl forum contents, you're perhaps less likely to interact with other users and contribute. You won't participate in discussions, ask question or help others. In short, you're not helping the forum in any shape or form, so why would they accommodate you?

And there may be other reasons, too.

I agree that there should be a way to find middle ground. But I understand why the negotiations are tough and the forum owners might be not at all interested in supporting such scripts or crawling apps.

[baccccccc]

I will take a look at making a config option for CSV unsupported URL files.

Not sure if it's related, but I found that you now include the source link (i.e., forum thread) to Unsupported_URLs.txt. IIRC that was not the case before. It is a bit confusing for a few reasons.

1. the file extension is still .txt.
2. the delimiter is ; instead of ,.

do you have plans to iterate on this feature, or that's the final state of affairs? I personally find the current behavior a bit confusing.