Facebookexternalhit/1.1

[PH5HP]

Hi,

I noticed strange activity for some days now. Several ip adress from the USA are downloading systematicly files from my server. The agen is in every case Facebookexternalhit/1.1

What is this? It can not be human activity, because it goes on and on. The whole day. See my screenshot.

Kind regards,
Henk - PH5HP.nl - fileserver.ph5hp.nl

[rejetto]

ask facebook :D
try the stop-spiders plugin

[PH5HP]

Hi,

I was investigating this already on Google. Te 'stop-spiders' plugin makes seems to make no difference.

I got also a warning in the console regarding the API used by the 'stop-spiders' plugin. This API seems to become outdated in the near future. I restarted my HFS Server and till this moment I have not seen this warning again.

UPDATE

This what the co-pilot of Edge came up:

The URL you provided, 1, corresponds to Facebook’s “External User Agent Text.” Let me explain what this means:

When Facebook users share interesting web content with others on the platform, Facebook temporarily displays certain details related to that content. For example, it might show the title of a webpage or the embed tag of a video. This process involves Facebook’s system retrieving information from the provided link. If you’ve noticed this in your server logs, it’s likely because a Facebook user shared a link from your website with others. If you have any questions or concerns about links or content shared by Facebook users, you can contact Facebook at [email protected]1.

Additionally, the user agent string “facebookexternalhit/1.1” belongs to FacebookExternalHit, a library used for performing HTTP requests. It often acts as a web crawler or bot, fetching content from external websites on behalf of Facebook2. If you encounter this user agent in your logs, it’s related to Facebook’s content retrieval process.

Kind regards,
Henk - PH5HP.nl - fileserver.ph5hp.nl

[rejetto]

the warning will go away with next release.

I was investigating this already on Google. Te 'stop-spiders' plugin makes seems to make no difference.

i expect the "spider" program to fetch the "robots.txt" only at start, so if it's already started it's normal to have no effect

[PH5HP]

Hi,

I did read something about that Facebook crawler and it seems that robots.txt does not help to exclude it. I banned the 2 ip ranges like:

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

So, it's not this https://developers.facebook.com/docs/sharing/bot/

[PH5HP]

Hi,

Itlooks the same but I am not sure about that. This is what I found when searching on the agent from the logs:

Question regarding the stop-spider plugin: Where do I put the Robots.txt ?

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

you don't put it, that's the whole point of the plugin.
if you would, you would not need the plugin.
i made the plugin just because it was a feature offered by hfs 2, and it's easy to make.

anyway, these 2 facebook bots seems to be different, as they have different goals.

[PH5HP]

Hi,

Ok. I understand your point. :-) It would be easier that somehow more info about plugins were made available directly in the plugin section of HFS. Some kind of help section for every plugin. :-)

The annoying thing was that every link was accessed and showed up as downloaded in the log. I do not know if everything was actually downloaded.

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

"fully downloaded" means that, to the last byte.

I would be easier that somehow more info about plugins were made available directly in the plugin section of HFS.

i'd be glad to get your suggestion on what description of my plugin would be clearer for you.

Some kind of help section for every plugin. :-)

"help section" sounds like a lot of content.
the idea is that when a lot of informative content should be provided by the plugin, it will be available through the link on its name.
it's just one click away.
you had a different idea?

[PH5HP]

Hi,

Not really. :-) But I am interested how the 'stop-spider' plugin reacts op a 'robots.txt' request. But I coud look into the souce of the plugin. However I am still not an expert of this javascript. :-)

The crawling stopped after banning those ip-ranges. I wonder if I had better narrowed it down to a smaller range of ip adresses.

UPDATE

Never mind. This plugin is so simple even I can understand what it is doing :-)

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

it's easier than reading javascript :D
just look at https://fileserver.ph5hp.nl/robots.txt

it's probably the most simple plugin out there

[PH5HP]

Hi,

Yeah. TNX. I thought that it had to be you when I saw an Italian accessing my server. :-)

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[PH5HP]

Hi Rejetto,

Maybe an idea to block ip adresses with a specific agent?

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

just wrote this server_code but didn't test it, so it will probably not work :)

exports.init = api => {
  const { block } = api.require('./block')
  return {
    middleware(ctx) {
      if (ctx.get('user-agent').includes('Facebook')) {
         block.set(x => [...x, { ip: ctx.ip, comment: 'facebook' }])
         ctx.socket.destroy()
      }
    }
  }
}

[PH5HP]

Hi,

I put the code above in the OPTIONS section ->Server code. Is this right? It does not seem to work after disabling the blocked ip adresses

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[PH5HP]

Hi,

Suggestion. Adding enable/disable checkbox to the blocked ip list. Easier to turn them on/of

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

you mean like this?

[rejetto]

It does not seem to work after disabling the blocked ip adresses

it worked well for me, but the very first request was served anyway.
i made a change in the script above, so that also first request is forbidden.

i used this to make my test:
curl -v 192.168.0.5 -H "User-agent: Facebook123"

[PH5HP]

Hi,

Yes that one and I know it excist there, but would be easier if it could be done in the list itself. Here:

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

you set all the expirations by hand for each line?

[PH5HP]

Hi,

Yes... Most of them by clicking on the button in the log... A lot of Chinese and now some of Facebook. The Chinese stopped now and I think I have Facebook now covered too. The Facebook ones are not shown in the screenshot, but I blocked some ip ranges there.

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

since the blocked ip supports wildcards, one can specify multiple ips like 1.1.1.1|2.2.2.2|3.3.3.3

would you prefer the block from the log to just add the ip to a single line/rule to not have so many?
of course with a single line you would have only 1 expiration.

[PH5HP]

Hi rejetto,

I know abou the wildcards. I am using that, but I do not want to exclude a lot of ranges. So I used a range when the FB crawler used 2 or more ip adresses in the same range. Like:

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

not talking about facebook now, i specified "block from log" because i saw you have many of those entries.

[PH5HP]

Hi,

The other entries were from Chinese origin who tried to flood the server with quests. All different ip adress ranges.

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

yes, but it's no need to be in ranges, the wildcards worka also by separating single ip with "|" character.
that's why i asked if you'd like them to be in a single line.

[PH5HP]

Hi,

Yes, that would give me a beter oversight of banned ip adresses

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

ok, but as i highlighted before, you would get just one expiration date.
would that be a deal breaker?

[PH5HP]

Hi,

No, not at all.

Kind regards,
Henk - ph5hp.nl - fileserver.ph5hp.nl

[rejetto]

ok, next version will do like that.
I also had to make other adjustments to better handle long text on that field

[rejetto]

ok, from version 0.54 it will append on existing rule