Cannot import snscrape as library when installed using pip.

[fthbaskin]

Describe the bug

I have installed snscrape into my venv using pip. The current version is snscrape-0.7.0.20230622 and I am using Python 3.12.4 with pip 24.1.2 version. I use Windows 10 as OS and Powershell as a console.

The problem is when I try to import the module using import snscrape.modules.instagram as snIns I get the following exception. Also, I get a similar exception when I try to use CLI.

  File "try.py", line 10, in <module>
    import snscrape.modules.instagram as snIns
  File ".venv\Lib\site-packages\snscrape\modules\__init__.py", line 17, in <module>
    _import_modules()
  File ".venv\Lib\site-packages\snscrape\modules\__init__.py", line 13, in _import_modules
    module = importer.find_module(moduleName).load_module(moduleName)

It seems like a bizarre bug. I would like to help you to fix this bug.

How to reproduce

The problem is when I try to import the module using import snscrape.modules.instagram as snIns I get the following exception. Also, I get a similar exception when I try to use CLI.

  File "try.py", line 10, in <module>
    import snscrape.modules.instagram as snIns
  File ".venv\Lib\site-packages\snscrape\modules\__init__.py", line 17, in <module>
    _import_modules()
  File ".venv\Lib\site-packages\snscrape\modules\__init__.py", line 13, in _import_modules
    module = importer.find_module(moduleName).load_module(moduleName)

Expected behaviour

import snscrape.modules.instagram as snIns should work correctly and I should be able to use the library.

Screenshots and recordings

No response

Operating system

Windows 10 Enterprise 2009

Python version: output of python3 --version

3.12.4

snscrape version: output of snscrape --version

0.7.0.20230622

Scraper

Instagram

How are you using snscrape?

Module (import snscrape.modules.something in Python code)

Backtrace

  File "try.py", line 10, in <module>
    import snscrape.modules.instagram as snIns
  File ".venv\Lib\site-packages\snscrape\modules\__init__.py", line 17, in <module>
    _import_modules()
  File ".venv\Lib\site-packages\snscrape\modules\__init__.py", line 13, in _import_modules
    module = importer.find_module(moduleName).load_module(moduleName)

Log output

No response

Dump of locals

No response

Additional context

No response

[fthbaskin]

Apparently, commenting out everything in modules/__init__.py solved the issue.

[TheTechRobo]

#782

[CortoRikasa]

Encountered the same problem. Changed modules/init.py to the following:

import pkgutil
import importlib

all = []

def _import_modules():
prefixLen = len(name) + 1
for importer, moduleName, isPkg in pkgutil.iter_modules(path, prefix = f'{name}.'):
assert not isPkg
moduleNameWithoutPrefix = moduleName[prefixLen:]
all.append(moduleNameWithoutPrefix)
module = importlib.import_module(moduleName)
globals()[moduleNameWithoutPrefix] = module

_import_modules()