Skip to content

Downloading IP Lists

Jump to bottom Edit New page
Costa Tsaousis edited this page Nov 8, 2015 · 62 revisions

You need to have FireHOL's update-ipsets installed for downloading IP Lists directly from their maintainers. Follow Installing update-ipsets to install it.

Using update-ipsets

Once the latest development version of FireHOL is installed on your computer, you can run update-ipsets to download all the IP lists you need.

For example to download dshield you should run:

sudo update-ipsets enable dshield

You can enable multiple IP lists at once:

sudo update-ipsets enable dshield blocklist_de spamhaus_drop spamhaus_edrop

Then, to download all the enabled lists, run:

sudo update-ipsets

You can run this command repeatedly. It will not harm. update-ipsets is smart enough to download IP lists only when they are expected to be updated.

For example, it knows that dshield is updated every 10 minutes, so it will not download it again unless there have been 10 minutes since the date/time of the last download (it does not care when you downloaded it, but what date/time the downloaded file had when downloaded). It will attempt to download it again, only if 10 minutes have passed since the modification date/time of the downloaded file.

Periodic Updates - The Cron Job

This is how I run update-ipsets via cron:

# sudo crontab -e   or   sudo nano /etc/crontab

*/9 * * * * root update-ipsets >/dev/null 2>&1

I run it every 9 minutes. You can choose 8, 11, 12, etc, it does not matter. Avoid using 5, 10, 15, 20, etc. If all of us choose, let's say 10, the maintainers' sites will get all the requests concurrently. I suggest to pick a random number between 5 and 20.

update-ipsets keeps its files in /etc/firehol/ipsets. In this directory you will have, for each IP list:

  • NAME.source - the file downloaded from the IP list maintainer
  • NAME.ipset (or .netset depending on the IP list type) - the final processed and normalized file

Download all IP Lists

To download all the IP lists known by update-ipsets, use this:

sudo update-ipsets --enable-all

Then, sit back and watch it download all IP lists (it will take some time).

Keep in mind that a few lists (like hphosts_*) publish hostnames, not IPs. These lists are converted to IPs using your name server. update-ipsets is optimized to do parallel DNS queries to minimize the time needed for this conversion, still though it will need some time to finish.

You can set the number of parallel DNS queries using this:

export PARALLEL_DNS_QUERIES=50
sudo update-ipsets

The default is PARALLEL_DNS_QUERIES=10.

You can also put options in its configuration file. The default configuration is /etc/firehol/update-ipsets.conf. In our example, the configuration file should contain:

PARALLEL_DNS_QUERIES=50

Updating the ipsets in kernel

update-ipsets will do this by itself, automatically!

If it finds an ipset named with the name of an IP list, it will update it automatically when the IP List is updated.

update-ipsets goes through the following procedure for each IP list:

  1. download the IP list from the maintainer
the downloaded file is saved in `/etc/firehol/ipsets` as `NAME.source` - it has the modification date/time of the same file at the maintainer's site
  1. process it to extract / normalize its content
the result is saved in `/etc/firehol/ipsets` as `NAME.ipset` or `NAME.netset` depending on its contents

  1. if an ipset with the same NAME already exists in kernel, then do all the following:

    1. create a temporary ipset in kernel

    2. load the contents of NAME.ipset or NAME.netset to the temporary ipset

    3. swap the temporary and the production ipsets in kernel

    this swap is an atomic operation that is handled by the kernel itself - either it succeeds (and the new ipset is used) or it fails (and the old ipset is left untouched)

If at any step there is a failure, the operation is aborted (all temporary resources used are released) and a message is sent to syslog.

The whole operation has been designed so that there will be no disruption at all at your firewall.

Keep in mind that the above are not FireHOL specific. update-ipsets will update the ipsets in kernel even if they are created by hand or another netfilter based firewall solution. The only condition is to have the same NAME and of course ipset type.

Since update-ipsets uses iprange, it also optimizes hash:net ipsets for optimal kernel performance. For more information about this optimization check iprange: optimizing ipsets for iptables.

update-ipsets configuration

The default configuration file is /etc/firehol/update-ipsets.conf.

You can override this path using the option -f, like this:

sudo update-ipsets -f /path/to/other/configuration/file

update-ipsets supports these options:

BASE_DIR

BASE_DIR states the directory ipset files should be stored. The default is /etc/firehol/ipsets.

RUN_PARENT_DIR

RUN_PARENT_DIR states the directory to save the lock file. The lock file prevents running multiple update-ipsets concurrently. The default is /var/run.

TMP_DIR

TMP_DIR states the directory to save temporary files. A new directory named update-ipsets-XXXXXXX will be created in TMP_DIR during each run. The default is /tmp.

HISTORY_DIR

HISTORY_DIR is used for storing the aggregation log for IP lists that need one. The default is /etc/firehol/ipsets/history.

ERRORS_DIR

ERRORS_DIR is used for storing downloaded IP list files that failed to be processed. The default is /etc/firehol/ipsets/errors. If you set it to empty, this feature will be disabled.

MAX_CONNECT_TIME

MAX_CONNECT_TIME gives the time in seconds to connect to maintainer's site. The default is 10.

MAX_DOWNLOAD_TIME

MAX_DOWNLOAD_TIME gives the time in seconds to wait for a download to complete. The default is 300 (5 minutes).

IGNORE_REPEATING_DOWNLOAD_ERRORS

IGNORE_REPEATING_DOWNLOAD_ERRORS is the number consecutive download errors to be ignored and favoured. After these many consecutive download errors, the refresh frequency of the IP list will be incremented.

Let's take an example:

IP list X has a refresh frequency of 20 minutes. The web server of the maintainer has issues and force us to timeout, slowing down the whole process.

We have set IGNORE_REPEATING_DOWNLOAD_ERRORS=5.

This is what will happen:

failed try next try in minutes
1 10 = 20 / 2
2 10 = 20 / 2
3 10 = 20 / 2
4 10 = 20 / 2
5 10 = 20 / 2
6 20 = 20 * 1
7 40 = 20 * 2
8 60 = 20 * 3
9 80 = 20 * 4
10 100 = 20 * 5

So, while the fail count is below IGNORE_REPEATING_DOWNLOAD_ERRORS, update-ipsets favours the ipset, hopping the maintainer will fix the problem soon. If the fail count goes above IGNORE_REPEATING_DOWNLOAD_ERRORS then update-ipsets delays the update every time more, to avoid stalling the whole process due to poorly maintained download sites.

PARALLEL_DNS_QUERIES

PARALLEL_DNS_QUERIES gives the number of threads update-ipsets will spawn for resolving hostnames found in downloaded IP lists. The default is 10.

(keep in mind this is a feature of iprange, but the parameter is used by update-ipsets to configure iprange properly).

Keep in mind that increasing this too much may hit a quota on the name server, while lowering it too much may delay the whole process.

IPSET_REDUCE_FACTOR and IPSET_REDUCE_ENTRIES

IPSET_REDUCE_FACTOR and IPSET_REDUCE_ENTRIES are used by update-ipsets to optimize hash:net ipsets when they are loaded into the kernel. These values are not used when generating the .netset files on disk. They are only used when ipsets are updated in kernel.

The defaults are: IPSET_REDUCE_FACTOR=20 and IPSET_REDUCE_ENTRIES=65536.

Check iprange: optimizing ipsets for iptables for more information and a demonstration of how this works.

Add a custom footer
Add a custom sidebar
Clone this wiki locally