Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update utils.py #154

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Update utils.py #154

wants to merge 1 commit into from

Conversation

Zabina
Copy link

@Zabina Zabina commented Feb 17, 2023

I made some changes to the utils folder to avoid some frequently occurring errors.
I have to mention that I only made the changes to get_users_follow since I was only using this part. But solutions can also be added to other parts
For one to avoid the error in #105 and #122 StaleElementReferenceException I looped through the parts where the error occurred with the help of selenium.common.exceptions - StaleElementReferenceException "try: .... except StaleElementReferenceException:". Since the error popped up at different points of the code I added the loops at several parts.
On the other hand to avoid data loss, when an Exception occurs I added the lines

                    except Exception as e:
                        print(f"An exception occurred while trying to scroll the page: {e}")
                        continue

            follows_users[user] = follows_elem
        except Exception as e:
            print(f"An exception occurred while crawling {user}: {e}")

    # Call the function inside a try-except block
    try:
        results = get_users_follow(users, headless, env, follow, verbose, wait, limit)
    except Exception as e:
        print(f"An exception occurred: {e}")

since I also had issues with the headless mode I added some other options:

        options.add_argument('--disable-gpu')
        options.add_argument('--disable-dev-shm-usage') 
        options.add_argument('--no-sandbox')

Here I am not sure if it is necessary but it solved my problems.

Just a suggestion and hope It helps others to run the code more smoothly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant