Add support for collecting DHCP metrics from Kea Control Agent #2937
Conversation
|
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2937 +/- ##
==========================================
+ Coverage 56.58% 56.65% +0.07%
==========================================
Files 602 604 +2
Lines 43729 43890 +161
Branches 48 48
==========================================
+ Hits 24744 24868 +124
- Misses 18973 19010 +37
Partials 12 12 ☔ View full report in Codecov by Sentry. |
jorund1
force-pushed
the
kea-ctrl-agent-metrics
branch
from
d157f0b to
0640c0c
Compare
There was a problem hiding this comment.
Thank you, @jorund1 !
I have not read and understood the code in full detail, but I have some generic feedback.
Since this is your first contribution, I like to mention that I am a proponent of the "step-down rule". Full quote:
We want the code to read like a top-down narrative. We want every function to be followed by those at the next level of abstraction so that we can read the program, descending one level of abstraction at a time as we read down the list of functions. I call this The Stepdown Rule.
To say this differently, we want to be able to read the program as though it were a set of TO paragraphs, each of which is describing the current level of abstraction and referencing subsequent TO paragraphs at the next level down.
To include the setups and teardowns, we include setups, then we include the test page content, and then we include the teardowns.
To include the setups, we include the suite setup if this is a suite, then we include the regular setup.
To include the suite setup, we search the parent hierarchy for the "SuiteSetUp" page and add an include statement with the path of that page.
To search the parent...
It turns out to be very difficult for programmers to learn to follow this rule and write functions that stay at a single level of abstraction. But learning this trick is also very important. It is the key to keeping functions short and making sure they do "one thing." Making the code read like a top-down set of TO paragraphs is an effective technique for keeping the abstraction level consistent.
In general, in kea_metrics_test.py, I would like to see the actual tests first in the code file, fixtures and helpers further towards the bottom. The only exception is when the programming language makes this ordering impossible, which can happend in Python, since it's not a pre-compiled language.
Other than that, I see you have written some code in order to validate JSON config data from Kea. At this point, we might consider pulling in Pydantic into NAV also, I think a lot of this code would disappear with it. We use Pydantic > 2 for validation of JSON data in several of the other projects our team maintains. It's not core to what you're really working on, but I at least urge you to have look at the library :)
jorund1
force-pushed
the
kea-ctrl-agent-metrics
branch
2 times, most recently
from
4c8f756 to
78607bf
Compare
Why: The main metrics one wishes to obtain from a dhcp server of a particular type (Kea DHCP, ISC DHCP, udhcpd, etc.) are the same accross the board, and thus the methods that process these metrics (e.g. sending them to a graphite server, creating a canonical graphite path for a specific type of metric, etc) are better off being defined once in a superclass.
KeaDhcpMetricSource is an implementation of DhcpMetricSource which collects the four metrics defined in the DhcpMetricKeys enum (number of total, used, free and touched addresses) for each vlan of a Kea DHCP server.
Why: I'll need to check more carefully how to obtain the amount of free addresses in a subnet, it proboably must be calculated since Kea doesn't seem to supply it.
Need to see how to deal with shared networks that might also be defined as well. Should be exactly the same as for subnets, since a shared network really is just a uniquely named list of subnets.
why:
This is build-up for an up-coming commit that implements caching
of self.kea_dhcp_config in KeaDhcpMetricSource; everytime we fetch
a new kea_dhcp_config with fetch_dhcp_config(), we would like to
store it as well - hence the name change of the function.
The up-coming commit will then make use of Kea Control Agent's
`config-hash-get` command (included in Kea versions >= 2.4.0) to
check if we need to update the cached config or not whenever
set_and_fetch_dhcp_config() is called.
What:
In addition to updating function names etc., I've also updated the
mocking of the requests.post requests in the test script so that
we can give different respsonses based on the Kea Control Agent
command the kea_dhcp_data script sends to the Kea Control Agent
server with its request.post calls
For now, I'll just call this new function for unwrap(), since all of
the uses of send_query() expects a list of one response and the first
thing that is done is always to unwrap the singleton response
list. unwrap() could be made to have generic typing, but for
simplicity it uses KeaResponse instead of a generic TypeVar('T') for
now.
Also fixes a typo on line 342 where an extra comma had sneaked itself into the code
What:
Before, we only included subnets defined in the subnet[4,6]
section of the Kea DHCP config obtained through the "config-get"
query in the KeaDhcpConfig.subnets list. Now we also include the
subnets defined in the shared-networks section of the Kea DHCP
config, and thus we include all subnets than could possibly be
configured for a Kea DHCP server, which means that we can now
fetch metrics from all defined subnets.
Why:
Datetime instances are the most precise, and can easily be
converted to unix timestamps is need be. Datetime instances also
makes it easy to work with timezone differences, which we sadly
seem to have to care about since the Kea Control Agent doesn't
provide timezone data along with its timestamps.
Why:
If we don't obtain the config, we do not have enough information
to start fetching subnet metrics; in this case there is no way to move forward
and there's no hope of obtaining some useful data by continuing
the call.
what:
prior to this commit, the key for a specific metric (i.e. the name
of that metric used by NAV) had the same naming convention as
dhcpd-pools(1), e.g. "cur" was the name used for the "amount of
addresses currently assigned to dhcp clients on this subnet" and
"max" was the name used for the "total amount of addresses
controlled by this subnet". DhcpMetricKey.CUR and
DhcpMetricKey.MAX, however, is not very descriptive, so I changed
the key names to be DhcpMetricKey.TOTAL and
DhcpMetricKey.ASSIGNED. DhcpMetricKey.TOUCH was removed all
together because it seems to me like this is not a common metric
to be reported by dhcp servers (dhcpd-pools(1) uses "touch" to
mean the number of assigned addresses that has timed out but that
is not yet marked as re-assignable by the dhcp-server).
what:
Modify the docstrings so that they all follow the same pattern;
i.e. they all begin by describing what they return.
why:
In the future, one might want to include sensitive information,
such as passwords or tokens in requests, and a response from Kea
might contain secrets, especially with regards to "config-get"
responses, where a config might contain passwords.
what:
before this change, our mocking functionality only allowed for
setting the string of the response object that is returned by
requests.post() and requests.Session().post(), by using
responsequeue.add("<command-name>", "<returned-string>").
responsequeue.autofill("dhcp<4 or 6>", config_to_return, statistics_to_return)
This commit modifies adds an extra parameter to both of these
functions, the `attrs` parameter:
responsequeue.add("<command-name>", "<returned-string>", attrs={}).
responsequeue.autofill("dhcp<4 or 6>", config_to_return, statistics_to_return, attrs={})
if `attrs` = {"myattr": "myval"}, then the requests.Response()
object returned by any call to requests.post() or
requests.Session().post() will have the attribute "myattr":
attrs = {"myattr", "myval"}
responsequeue.add("<command-name>", "<returned-string>", attrs)
response = requests.post(...)
assert response.myattr == "myval"
(this is what the graphite exporting function expects)
jorund1
force-pushed
the
kea-ctrl-agent-metrics
branch
from
78607bf to
76da903
Compare
|
It's been way too long since I looked at this, but I surmise that it really just adds a generic framework for fetching DHCP metrics, with a Kea-specific implementation. There seems to be no way to drive this new functionality (i.e. a command line program), and no documentation on how to use it. Those things would be of utmost importance for this to make it into a NAV release. I think I would rather have a live code-review of the whole thing so I can understand it better, and since you're going on vacation today, I don't think we'll be able to commit to that until January at the earliest. This means this will miss the deadline for the 5.12 release, but we'll likely make 5.13 by March next year, which will be it's next chance :) |
Implements #2931
Uses python 3.9 typehints
Defines the
KeaDhcpMetricSourceclass and its superclassDhcpMetricSource, having methodsfetch_metricsandfetch_metrics_to_graphitethat can be used to fetch metrics from a Kea DHCP server controlled by a Kea Control Agent and send these metrics to a graphite server. Example usage:The other defined classes and functions are helpers and is not really meant to be used by other parts of NAV.
Todo:
KeaDhcpConfig.from_jsonfor storing the configuration hash supplied in the json obtained fromconfig-getqueries.config-hash-getqueries inKeaDhcpMetricSource.fetch_and_set_dhcp_configandKeaDhcpMetricSource.fetch_dhcp_config_hashstatistic-getqueries inKeaDhcpMetricSource.fetch_metrics