Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix EvilAngel performerByURL -> Refactor Algolia scraping #2177

Open
wants to merge 70 commits into
base: master
Choose a base branch
from
Open

Fix EvilAngel performerByURL -> Refactor Algolia scraping #2177

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
f4416d5
fix: start refactor to standard Algolia Python client
nrg101 Jan 23, 2025
fd42f14
add performerByName and performerByFragment
nrg101 Jan 24, 2025
f50820d
use guess_nationality; remove unused imports; tidy typings
nrg101 Jan 24, 2025
8de32ba
make the homepage url and headers better
nrg101 Jan 24, 2025
e621444
refactor sceneByURL
nrg101 Jan 24, 2025
9e06b45
refactor sceneByURL
nrg101 Jan 24, 2025
2767b57
refactor sceneByFragment and sceneByQueryFragment
nrg101 Jan 24, 2025
d0ab149
refactor galleryByURL
nrg101 Jan 24, 2025
1423b03
refactor galleryByFragment
nrg101 Jan 24, 2025
b6bcec0
refactor movieByURL
nrg101 Jan 24, 2025
0d0fa04
move AlgoliaAPI.py to its own folder
nrg101 Jan 25, 2025
e434681
ensure requests package is installed
nrg101 Jan 25, 2025
6298dec
make suggested changes
nrg101 Jan 25, 2025
d2112e0
make EvilAngel.py
nrg101 Jan 25, 2025
8884d25
implement postprocess; tidy comments/typings
nrg101 Jan 26, 2025
1a2c9be
reference correct package
nrg101 Jan 27, 2025
702bbb5
make studio determination work for all sub-studios sceneByURL
nrg101 Jan 27, 2025
28ea1f7
fix naive ts->trans mass find-replace
nrg101 Jan 27, 2025
995ffb5
add postprocess_movie
nrg101 Jan 27, 2025
1ef96b5
use clean_text; fix available site iteration order
nrg101 Jan 27, 2025
16d9aaa
add postprocess_gallery
nrg101 Jan 27, 2025
2bd039a
use generic type
nrg101 Jan 27, 2025
280a55f
improve AlgoliaAPI and EvilAngel
nrg101 Jan 28, 2025
ed40f4e
add new AdultTime scraper that uses new AlgoliaAPI
nrg101 Jan 28, 2025
ec13415
address linting problems; improve property getting
nrg101 Jan 29, 2025
33cc2dc
complete Adult Time Pilot sub-studio support
nrg101 Jan 29, 2025
b75abe9
complete all Adult Time sub-studio support
nrg101 Jan 29, 2025
c909aca
use channel name map keys
nrg101 Jan 29, 2025
5fd80c5
use script for custom post-processing
nrg101 Jan 29, 2025
1217fc7
add process_action_tags function
nrg101 Jan 29, 2025
403dbcf
sort multiple scene search hits by match ratio scores
nrg101 Jan 29, 2025
f0277a1
add sorting for multiple performer search hits
nrg101 Jan 29, 2025
dd57bbe
simplify log messages
nrg101 Jan 29, 2025
36c8639
add photoset search hit match ratio sorting
nrg101 Jan 29, 2025
a766e61
make indentation consistent
nrg101 Jan 29, 2025
da4a18c
reduce line count
nrg101 Jan 29, 2025
740ed3d
move analacrobats.com to EvilAngel
nrg101 Jan 29, 2025
9e41337
move JohnnyDarkoXXX to EvilAngel
nrg101 Jan 29, 2025
e128f8d
move pantypops.com to EvilAngel
nrg101 Jan 29, 2025
466cf56
move povblowjobs.com to EvilAngel
nrg101 Jan 29, 2025
09b3554
move strapattackers.com to EvilAngel
nrg101 Jan 29, 2025
5359629
move tittycreampies.com to EvilAngel
nrg101 Jan 29, 2025
5332893
move transsexualangel.com to EvilAngel
nrg101 Jan 29, 2025
1c7f3b1
use trace loglevel for action_tags
nrg101 Jan 30, 2025
3bc6e8d
Revert "use trace loglevel for action_tags"
nrg101 Jan 30, 2025
4c2c415
use trace loglevel for action_tags
nrg101 Jan 30, 2025
7e91418
map Transfixed Muses to Transfixed
nrg101 Jan 30, 2025
ed6b3ba
fix gallery scraping
nrg101 Jan 30, 2025
05ad8a4
change GenderXFilms to use AlgoliaAPI
nrg101 Jan 30, 2025
ff33ad0
add studio logic for Devil's Film
nrg101 Jan 30, 2025
fdc2d5c
fix argument variable name
nrg101 Jan 30, 2025
b3a4456
add video URL for devilsfilm gallery
nrg101 Jan 30, 2025
4ebfe05
add studio name logic for ASMR Fantasy
nrg101 Jan 30, 2025
65bab27
make Transfixed gallery work with photoset URL
nrg101 Jan 30, 2025
ff0bb04
add galleryByURL for oopsie.com
nrg101 Jan 30, 2025
58bce66
fix galleryByURL
nrg101 Jan 30, 2025
df3d0fa
add galleryByFragment to GenderXFilms
nrg101 Jan 30, 2025
4afecd4
use db gallery folder file count in match ratio evaluation
nrg101 Jan 30, 2025
d83860b
fix bug for studio override when no channels prop
nrg101 Jan 30, 2025
b87c964
add zip support for galleryByFragment
nrg101 Jan 30, 2025
a5f6d04
fix function call missing argument
nrg101 Jan 30, 2025
70de3cf
fix function call missing argument
nrg101 Jan 30, 2025
6469343
support Devil's Tgirls better
nrg101 Jan 30, 2025
a27380f
move sites to FantasyMassage (Network)
nrg101 Jan 30, 2025
8905958
move sites
nrg101 Jan 30, 2025
c8fac9a
migrate TabooHeat
nrg101 Jan 30, 2025
d797fea
remove straggler
nrg101 Jan 30, 2025
b616bd5
migrate Gangbang Creampie
nrg101 Jan 30, 2025
f47d03b
migrate sites
nrg101 Jan 30, 2025
36d84ec
add extra logic for TransgressiveXXX studio name
nrg101 Jan 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 0 additions & 31 deletions scrapers/21Naturals/21Naturals.yml
View file
Open in desktop

This file was deleted.

31 changes: 0 additions & 31 deletions scrapers/21Sextreme/21Sextreme.yml
View file
Open in desktop

This file was deleted.

40 changes: 0 additions & 40 deletions scrapers/21Sextury/21Sextury.yml
View file
Open in desktop

This file was deleted.

49 changes: 0 additions & 49 deletions scrapers/3rdDegreeFilms/3rdDegreeFilms.yml
View file
Open in desktop

This file was deleted.

40 changes: 0 additions & 40 deletions scrapers/AddictedToGirls/AddictedToGirls.yml
View file
Open in desktop

This file was deleted.

Loading