Display and Account for Unknown Browsers aka User Agents

[jordantrizz]

I understand there is a method to add custom browsers aka User Agents using the --browsers-file as per #560

1. Is there a way to in goaccess to not have an unknown category show all user agents?

2. There should be better documentation as some User Agents are massive. How does the matching occur? Does the user agent need to be exact in the --browsers-file? Or is there partial matching?

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

3. What are all the categories available? I know it sounds silly, but it would be good to have this in the sample file for those that are just discovering goaccess.

[allinurl]

1. I'm not entirely sure I understand your question, but to categorize unknowns as crawlers, you can utilize the --unknowns-as-crawlers option.

2. When dealing with custom user agents, GoAccess will try to find the first occurrence of the substring provided in your custom list. For crawlers, since legitimate ones often contain a URL, GoAccess will attempt to match them based on that.

3. Here are the default categories. However, it is possible to generate your own custom categories by editing the browsers.list file. Make sure to separate the values and categories using tabs. For example:

   XYZ			Scanners
   

Heads up: The size of the list is directly proportional to the run time.

[jordantrizz]

Thanks for the reply!

1. Sorry I didn't phrase the question properly. There is a User Agent of "WP Rocket/Preload" that doesn't show up under browsers when expanded to show all User Agents. Unless I use --browsers-file option. Would it be possible to list all Browsers/User Agents and not have Unkown?

It would be impossible to keep up with all the User Agents being used.

[allinurl]

I like the idea of getting the specific name, but I'm not sure if showing the whole user agent string is the right approach. It could end up cluttering the panel with long user agents. I'm curious, if you use --unknowns-log=file.log, do you get a lot of "real" bots or mostly fake user agent strings when opening file.log?

[jordantrizz]

Let me try --unknowns-log as I didn't know about this command, and should have RTFM. I'll post the results.

To give you an idea of how I'm using GoAccess. Mostly I'm looking for attacks, either crawlers or automated attacks, that would spike resources. There are a ton of custom user agents out there in this regard, so it helps to know all of the user agents so that I can see if a specific custom user agent is the cause of a majority of the traffic.

[jordantrizz]

Here's the output.

[BR]   -
[BR]   0
[BR]   ACI Site Scanner
[BR]   axios/1.4.0
[BR]   Barracuda Sentinel (EE)
[BR]   Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)
[BR]   EasyBib+AutoCite+(http://autocite-info.citation-api.com/)
[BR]   lua-resty-http/0.10 (Lua) ngx_lua/10019
[BR]   ShortPixel.com optimization API
[BR]   Site24x7
[BR]   Uptime-Kuma/1.15.1
[BR]   ZoteroTranslationServer/WMF (mailto:noc@wikimedia.org)
[OS]   -
[OS]   0
[OS]   ACI Site Scanner
[OS]   axios/1.4.0
[OS]   Barracuda Sentinel (EE)
[OS]   Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)
[OS]   EasyBib AutoCite (http://autocite-info.citation-api.com/)
[OS]   GuzzleHttp/6.3.3 curl/7.58.0 PHP/7.1.33-52 ubuntu18.04.1 deb.sury.org 1
[OS]   lua-resty-http/0.10 (Lua) ngx_lua/10019
[OS]   Mozilla/5.0 (compatible; archive.org_bot  http://archive.org/details/archive.org_bot)
[OS]   Mozilla/5.0 (compatible; inoreader.com; 1 subscribers)
[OS]   ShortPixel.com optimization API
[OS]   Site24x7
[OS]   Uptime-Kuma/1.15.1
[OS]   ZoteroTranslationServer/WMF (mailto:noc@wikimedia.org)

[jordantrizz]

Also created this pull request #2520