Extending update ipsets

You can extend update-ipsets to download you IP lists.

There are 3 directories that are consulted for additional IP lists definitions:

1. /etc/firehol/ipsets.d
2. /usr/share/firehol/ipsets.d
3. ~/.update-ipsets/ipsets.d

In any of the above directories, you can place *.conf files, each describing one or more IP lists.

Here is what each of these files can contain:

IP list definitions

IP lists are defined with the update command. This is the template:

update name mins aggregation ipv keep \
    url \
    processor \
    category \
    info \
    maintainer maintainer_url \
    other options

name

Is the name of the IP list.

The name uniquely identifies the IP list and controls the filenames that will be generated.

mins

mins is the update frequency of the list, in minutes. update-ipsets will not attempt to download again an IP list if mins minutes have not passed since the modification time of the last downloaded file.

The modification time of the file at the maintainer's site is copied to the local file when it is downloaded. For example, if you download IP list named A, the file A.source that will be generated will have the modification time of the file as it is stored at the maintainer's site.

aggregation

aggregation is a list of durations in minutes that the ip list should be aggregated.

You can disable aggregation, by setting this parameter to 0.

Example values: "$[24*60] $[7*24*60] $[30*24*60]" to enable daily, weekly and monthly aggregations.

ipv

It can be ipv4 or ipv6.

Only ipv4 is supported at the moment.

keep

It can be:

• ip to keep only the single IPs found in the source file. If the source file includes also subnets, these will be filtered out.

  The generated file will have the suffix .ipset.

• net to keep only subnets found in the source file, ignoring single IPs.

  The generated file will the suffix .netset

• both to keep both single IPs and subnets found in the source file.

  The generated file will the suffix .netset

url

It is the URL to download the IP list from.

update-ipsets supports custom downloaders. The default supports downloading HTTP and HTTPS urls.

processor

The processor is a BASH function or external command that should input the source file and output the IPs or subnets it contains.

The processor is used the same way pipes work. Something like this:

cat source_file | processor > ips_and_subnets

# or

processor <source_file > ips_and_subnets

There are many processors already present in update-ipsets which you can use:

processor	description
trim	remove leading, trailing and multiple spaces and tabs from the file, remove empty lines
remove_comments	remove anything after a # and also trim
remove_comments_semi_colon	remove anything after a ; and also trim
gz_remove_comments	like remove_comments for gzip compressed files
snort_alert_rules_to_ipv4	extract IPs and subnets from snort alert rules
pix_deny_rules_to_ipv4	extract IPs (not subnets) from Cisco PIX access list deny rules
extract_ipv4_from_any_file	extract IPs (not subnets) from any text file
hostname_resolver	convert a list of hostnames to IPs by doing parallel DNS queries

Internally, there is also the infrastructure for an XML parser. You can check the parse_rss_rosinstrument, parse_rss_proxy, etc processors in the update-ipsets source.