CI Issues

[cullzie]

Creating this discussion to track any issues with CI

[fakeid30]

Good idea. I am debating whether to post the dummy account for CI @aapatre setup here.

[cullzie]

Ye we might need to remove the udemy login part for now. I'll try out a few possible fixes but it might be more effort to maintain than it is worth

[cullzie]

Headless browser seems to be always triggering robot check at the moment.
If I can't find a solution I would suggest we remove this from the CI run for the moment.
Instead just add a sanity check for the scrapers and expand unittests.

[fakeid30]

Another idea is to run the CI suite from GitLab while hosting the code in GitHub (it's doable)

[cullzie]

Ye unittests and some smoke/sanity tests for the scrapers is straightforward in github.
It might be possible to setup selenium grid on gitlab. I haven't looked at the options over there yet

[fakeid30]

@cullzie CI is now running without the --debug flag.

[cullzie]

Ye I added in accept-language header #174 which seemed to fix it locally but it was still having some issues just after I merged it. At least it's back on the right track

[fakeid30]

the issue was happening because udemy locked out the account. Needed to reset the pass and update GH secret.

[fakeid30]

apparently udemy has done something with our dummy account, i was reviewing CI logs and it's failing to login for a LONG time. I'll ask @aapatre to change it again.

[cullzie]

I think we should split up the checks we are running at the moment since the login is no longer working via headless browser and it has always been quite a flaky test.

1. Have smoke test for each site we scrape to determine that it still works.
2. Have unit tests to test code is working as we expect

For udemy logic we'll just need to rely on users creating bugs if/when they see issues

@fakeid30 What do you think?

[fakeid30]

Yep seems to be the best way for now.

[fakeid30]

@cullzie homebrew 2.7.0 has changed and now it's breaking mac installing. We need to fix the mac syntax for CI.

[fakeid30]

Fixed.

[fakeid30]

@cullzie CI still failing.

[cullzie]

@fakeid30 Need to rework the CI jobs a bit. Can probably just set it up to run on ubuntu now.
Might need @aapatre to have a look at the account currently being used. Looks like it is having issues logging in