Scraping the DHT?

[selfhoster1312]

Is this a goal of torrust-index to provide a component for scraping the DHT like btdig does with dhtcrawler2?

If it's not an exact goal of the project, would a new torrust-index-scraper component be welcome? I have no idea where to start with this, but i'd be willing to try and make it if some people can guide me.

My usecase: i'm archiving niche anarchist content which has often been torrented in the past. The torrents are usually not advertised on public indexes like TPB, however you can find them in BTDig. However, the search feature in BTDig is terrible, and it doesn't expose the seeders count so i often have to add a dozen torrents before finding one that is still seeded. I need a DHT index with proper search (eg. exact match for more than one word in a single file, instead of matching across all files) and that can load seed count on demand for a specific query.

Is this usecase compatible with the spirit of your project?

[geo-avn]

This project is, as the repository describes, a torrent tracker framework, not a torrent scraper for grabbing torrents.

The torrust-index repository is a just meta-repo for the torrust-backend and torrust-frontend.

[da2ce7]

For the moment the index only scapes a single tracker over a dedicated api. We have not generalized this api, so that an index-backend could collect the statistics from multiple sources.

The process to implement this feature could be something like this:

1. Define a proper scraping API.
2. Update the Backend to use this API to collect seeder and leecher statistics for indexed torrents.
3. Build a bridge that serves this API from data collected from the DHT.

A nice project.

[josecelano]

Hey @da2ce7, you have to consider also what we call the "torrent info" in the index, which is the torrent metadata. The metadata added in the index for the torrent is:

• title
• description
• category
• tags

@selfhoster1312 what do you mean by "exact match for more than one word in a single file, instead of matching across all files"?

I guess you want to import torrents from DHT into the index, then link more than one tracker and finally search for trackers having a torrent with two words in the title, right? Or maybe you mean to find a torrent which contains a file whose name has those two words?

For both cases, we have to create the metadata required by the Index, but I guess we could:

• Use the torrent file for the title and description
• No category
• No tags

Besides, I think the current search (search box) feature does not include the file names included in the torrent. I think that would be a very useful feature. In fact, it does not even include the torrent description, if I'm not wrong. It only uses the title, category, and tags. @da2ce7 I will check it and create a new issue if that is the case. I think we should at least include file names and torrent descriptions.

[da2ce7]

• Define and Use a Scraping API torrust-index-gui#289