Firewalld-0.9.11-1 update drastically slowed ipset importing on AlmaLinux 8 #186
Comments
|
This looks like it's a known issue--you can find some upstream references here:
While I was able to replicate the high resource utilization of [root@el8 ~]# rpm -q firewalld
firewalld-0.9.3-13.el8.noarch
[root@el8 ~]# time firewall-cmd --reload
success
real 1m17.009s
user 0m0.372s
sys 0m0.057s
[root@el8 ~]# rpm -q firewalld
firewalld-0.9.11-1.el8_8.noarch
[root@el8 ~]# time firewall-cmd --reload
success
real 2m53.163s
user 0m0.361s
sys 0m0.041s
I was curious if the latest version from upstream fixed this, as it looks like they're on version [root@el8 ~]# rpm -q firewalld
firewalld-0.9.11-4.el8.noarch
[root@el8 ~]# time firewall-cmd --reload
success
real 3m1.829s
user 0m0.337s
sys 0m0.059sWhen not using the [root@el8 ~]# mv /etc/firewalld/ipsets/beacon.xml{,.bak}
[root@el8 ~]# time firewall-cmd --reload
success
real 0m1.264s
user 0m0.349s
sys 0m0.049sThere was a suggestion in the bugzilla above which suggests it could be What are the specs of the machine you're running this on? Are you also seeing |
|
Hi Cody, Thank you for your observations and for taking the time to test. I should mention that we did long ago encounter the separate memory issues due to the Can I just check if there were any typos or mis-phrasing in your opening statements? Because whilst you said "I was not able to replicate the huge performance change between versions:" your results seem to demonstrate it slowing significantly (it taking 1 minute and 36 seconds longer to complete than before), which seems to reinforce what we are also seeing (a substantial performance decrease going from I have performed fresh tests this morning on a AlmaLinux 8.9 VM with 2 CPU cores & 4GB of RAM running By chance I just stumbled across this whilst looking for AlmaLinux rpm's the date of 3 months ago and last three filenames caught my eye: Which has led me to wonder if some developers have also witnessed and have been testing a performance issue/decrease with ipsets over the past few months? I couldn't as easily replicate the side-by-side effects this time around, the downgrade process I was using previously has changed due to the old package being pulled from the repo now so cannot yum downgrade. I will have to do some further testing, but this problem appeared to be clear as anything when myself and a colleague looked at it the other week, we could flip between two firewalld versions using the same ipset file, one would take numerous minutes to reload, the other just a few seconds. In recent testing I have found the hardware my VM can change things significantly time wise, I moved my test VM around between physical nodes, on one node it would take like 1.5 minutes to reload, on another just 17 seconds. Even still, I believe a significant performance decrease in the code has been introduced which is breaking network connections during the reload and introducing disruption. Thanks for your time and help all. |
|
BTW currently CentOS Stream 8 is on 0.9.11-4 version and changelog includes ipset fixes: @ashleycawley Could you check it by installing packages from CS8 to your system? |
|
Hi Andrew, thank you for that. I have tried downgrading my test VM to the RPMs on that page, however I'm still seeing the same performance decrease I believe. I'm using yum install and then specifying the URL to each RPM which appears to be downgrading or upgrading them appropriately during testing, but we're not seeing the same clear-cut results flicking between versions that we were seeing when we were testing previously on the 21st November, back then we were using the yum downgrade command. Checking yum history we have confirmed it was only downgrading packages:
Which are the ones I've been focusing my testing on. But it feels like perhaps something else has changed (another package?) since the 21st of Nov, either that or my testing / downgrading methods are flawed. I have been trying different / clean VM's too. If you have any advice on how I should be swapping out these packages and anything else to try in between that would be greatly appreciated, thank you. |
|
Thank you for your time and patience all, we've done a lot more testing today and I wanted to update you on my findings so far: we think we have narrowed it down possibly to a previous nftables update that only becomes apparent after a restart of firewalld. In this example below I purposely start with a clean but outdated AlmaLinux8 image which contains old packages which have not yet been updated, take a look at this: The restart of firewalld after the upgrade of nftables is important, if firewalld is not restarted then you don't witness the issue. I'm wondering if an earlier nftables update (July?) introduced a possible issue / conflict with firewalld (peformance wise) which didn't immediately become apparent because the update itself wasn't enough to see the issue, as a restart of firewalld was also required and the presence of a large ipset. To clarify the timeline we started to see this issue across a fleet of hundreds of systems around 21st-22nd September, I wonder if the nftables was updated months prior to that, but it perhaps took the act of another package being updated (firewalld or other) before it restarted the firewalld service and then the issue was apparent. Any thoughts or advice appreciated as always 🙏🏻 |
I was wondering if I should rephrase it. I ran the tests about 5 times each with wildly inconsistent numbers across versions--some being faster then the newer version, some being slower. I don't think it's the regression in performance you're seeing. Sorry for the confusion.
That's why I tested the latest build from upstream--to test those patches :). They didn't fix anything, unfortunately.
This was already tested above.
This is interesting. I'll see if I can get those results and see if I can find anything useful :) |
|
Yeah I just did a fresh test on another system, downloaded & ran a copy of AlmaLinux-8.8-x86_64-minimal.iso and then out of the box (no updates): On a low spec VM (2 CPU, 2GB RAM) it went from 4.5s before nftables update to almost 6 minutes after. |
|
Could you please try to test the same in AlmaLinux 8.10 beta? |
|
Thanks for the update @andrewlukoshko I will be happy to setup the same test scenario with 8.10 beta and will report back within the next few hours if I can. |
|
OK, tested on the same hardware and spec as the earlier post, with AlmaLinux 8.10 beta, out of the box with the 12k ipset in place (same one as before) a firewall-cmd --reload this time took 1min 14secs. Unlike my previous post where this process used to take a mere 4 seconds and then got worse (almost 6 minutes) after an update to nftables, there was no update available to nftables on this release so no update was applied to that package during this round of testing. So arguably you could say it has improved from the last state of play; dropping from 6min to 1min, however, its still no where near as fast or as effecient as it originally was for quite some time where it would take just 4 seconds. Thank you so much for your time and help on tracking this issue down. If I can help with any further testing then please let me know. |
|
I know, this is Alma Linux, but I've the same issue with Rocky-8 (latest updates) --> here it takes up to 19 minutes to reload. Since Alma and Rocky are almost identical, I thought I may give my feedback here too. |
OS: AlmaLinux 8
In late Sept 2023 an update for firewalld was released (firewalld-0.9.11-1) and we believe since that update we have seen a massive decrease in performance with regards reloading the firewall when large ipsets are in use (ie. a ipset that contains 12,000 IP addresses).
I'd be intrigued to see if anyone else has also been impacted by this bug or if anyone can help to confirm if they see the same?
Our use case: we have hundreds of AlmaLinux machines which have a custom ipset which contains 12,000 IP addresses which periodically rotate, the ipset always contains that quantity (12k). On the previous version of firewalld-0.9.3-13 when we would reload the firewall with
firewall-cmd --reloadit would complete in around 6 seconds - at that point firewalld would pass the ipset to the backend firewall.Our firewalld setup is using nftables behind the scenes not iptables.
After upgrading from firewalld-0.9.3-13 to firewalld-0.9.11-1 the same reloading operation will take a matter of minutes, it can vary from 2 - 5 minutes and causes disruption to network connections which is noticeable.
If we downgrade the firewalld package it reverts to taking a mere 6 seconds to reload. Bearing in mind firewalld is ingesting our .xml ipset which contains 12k ip addresses and passing that to nftables during that time, then reloading.
If the firewall backend is swapped out (switching from nftables to iptables in the firewalld.conf) then the problem goes away. So it appears to be a bug in the latest version of firewalld + nftables when ingesting large ipsets - it has drastically slowed down the act of reloading and can cause disruption.
From our testing it is taking anywhere from 25-55x longer to reload, so quite the performance decrease. This has been tested on a variety of AlmaLinux boxes with varying specifications.
Is there anyone else out there handling ipsets which are 10k+ in combination with firewalld + nftables? If so, do you notice firewalld taking significantly longer to reload nowadays?
If it helps anyone to validate to see if you get the same results as us then I will share the ipset XML file below that contains 12k of IP addresses:
I'm hoping someone may be able to corroborate that when reloading firewalld with this ipset in place with the package firewalld-0.9.11-1 (post Sept 2023) it'll take ages, yet the pre Sept package firewalld-0.9.3-13 will reload in seconds.
The XML contents below could be placed in a file at:
/etc/firewalld/ipsets/beacon.xmlIt wouldn't let me attach a .xml so I've published it here:
https://drive.google.com/file/d/1JGrTS60hRTpCffnBi-dAGu04HLgXR8g0/view?usp=sharing
Then:
Then:
If anyone is curious what these IP addresses are - they are "bad" IPs which we have witnessed trying to brute-force their way into our systems.
The text was updated successfully, but these errors were encountered: