Usernames are showing up as highlighted and not downloading - Any suggestion why?

[Aston77]

I've had to move the folder containing the crawler around a couple of times lately while changing hardware. It invariably breaks the directory connections to the different services and I have to manually update them individually. It's a bit of an annoyance but it's fixable once you've figured out what is causing the problem the first time.

This last time though I ran into a new problem...

• Updated the main data path
• Updated the Twitter folder path, cookies, verified authorization, updated token
• Updated the Reddit folder path and cookies
• Imported users (current path)
• Restarted program

Pretty much everything shows up again but a lot of the Twitter & Reddit usernames are highlighted in either blue or red.

As soon as I try to manually download temp usernames from Twitter, their user names go from unhighlighted -> highlighted red.

[AAndyProgram]

1. If you've moved SCrawler to a new location, all you need to do is just open Settings.xml and change the paths.
2. Read about colors here: https://github.com/AAndyProgram/SCrawler/wiki#user-colors
3. About Twitter: Will Twitter scraping be affected by this API announcement? #104 (reply in thread)
4. Fascist Twitter is currently broken. I can't tell you when I'll fix it.
5. I don't know why Reddit profiles are marked in red. The Reddit parser works without problems. Does the Reddit profile (highlighted in red) really exist?

[AAndyProgram]

Just seemingly a random issue here and there (like <10%). I can see the Reddit profile exists on the Reddit site, but it shows as deleted on the crawler 🤷‍♂️

I downloaded Reddit today with no problems. Is there data in the LOG? To update user's status, you need to select this user(s) and force download them. Then, if the user exists on the site, the red highlight disappears.

Reddit/Twitter https://www.androidpolice.com/reddit-kill-third-party-clients-unreasonable-fees/

About Reddit: #130 (comment)

I got a notification out of the blue from IG the other day but it sounds like whatever they are doing over there no one has been able to get any clarification. https://www.reddit.com/r/Instagram/comments/13vkm2h/weve_detected_automated_behavior_on_your_account/

We already talked about IG 3 weeks ago. Still nothing has changed. I'm currently testing requests with a few more additional headers but no luck. It still logs me out once every 1-3 days. Not much different (previously kicked out every day). Refreshing the x-csrf token isn't that hard, so it doesn't really bother me.

[AAndyProgram]

Btw, if you know gallery-dl, can you help me with the command? How to get all tweets including retweets?

[AAndyProgram]

Solution found. I need a couple of days to test the new code.

[AAndyProgram]

Fixed. Please update to the latest release.

[Aston77]

Just seemingly a random issue here and there (like <10%). I can see the Reddit profile exists on the Reddit site, but it shows as deleted on the crawler 🤷‍♂️

I downloaded Reddit today with no problems. Is there data in the LOG? To update user's status, you need to select this user(s) and force download them. Then, if the user exists on the site, the red highlight disappears.

I had a chance to look further into this and whatever the issue is it doesn't generate any apparent log.

The same issue seems to occur regardless of the source as I had the issue with both Reddit & Twitter (even with the newest version). I was able to get most of the Twitter user profiles to repair with the new update, but for whatever reason I had to do them manually one by one because I couldn't get them to repair with Download all site users [FULL].

The ones that didn't repair I tried the following.

1. Manually re-download them (Didn't work)
2. Separated from the collection if they were in one & manually re-download (Didn't work)
3. Delete the User & Re-Add the User whilst keeping the data (Didn't work)
4. Deleted the User AND Data & then Re-Added the user (Issue was resolved)

I did notice that when broken if I toggled them on to Ready to Download and then tried to manually force a download the Ready to Download setting with be toggled off again. I'm not sure what causes that...just an observation.

The above workaround works, just wanted to provide contextual feedback on our previous discussion.

We already talked about IG 3 weeks ago. Still nothing has changed. I'm currently testing requests with a few more additional headers but no luck. It still logs me out once every 1-3 days. Not much different (previously kicked out every day). Refreshing the x-csrf token isn't that hard, so it doesn't really bother me.

Yea, I wasn't pushing the issue on IG, just simply pointing out that all of the services seemed to be making changes at about the same time.

[AAndyProgram]

I had to do them manually one by one because I couldn't get them to repair with Download all site users [FULL].

Did you press this button with a Shift?

I had the issue with both Reddit .....

I don't know what happened to Reddit on your side. You can post the profiles you are trying to download and I'll try to download them from my end. Maybe the problem is in a particular profile. You can also send me or post here users' xml settings files.

I did notice that when broken if I toggled them on to Ready to Download and then tried to manually force a download the Ready to Download setting with be toggled off again.

Hm... You are right. This is a bug. This was added in early versions to prevent deleted users from downloading. After a while I correct the algorithms for collecting users, but I forgot to delete this bypass. Thank you for pointing out the problem in such detail.

SCrawler/SCrawler/API/Base/UserDataBase.vb

Line 1041 in 938042e

If Not UserExists Then ReadyForDownload = False

Currently you can select multiple users, right click, click Change ready for download in the context menu. This action is intended to change this parameter for one or multiple users.

Yea, I wasn't pushing the issue on IG, just simply pointing out that all of the services seemed to be making changes at about the same time.

Maybe. I'm not a conspiracy theorist but it looks like it. 🤣

[AAndyProgram]

To display all deleted users, you can use filters (All sites + Deleted + Ignore excluded labels (if you set Excluded labels earlier) + remove dates limits (if you set them)).

[AAndyProgram]

I did notice that when broken if I toggled them on to Ready to Download and then tried to manually force a download the Ready to Download setting with be toggled off again. I'm not sure what causes that...just an observation.

Fixed. Please update to the latest release.