4.x major/breaking new features list #295

mikaku · 2020-11-03T14:00:44Z

This issue will serve as a list of all the major/breaking new features that could be nice to include in a future 4.x version:

Create an individual configuration file for each module in conf.d/:
- This would reduce a lot the size of the main configuration file which would only have the global options.
- This would make less tedious upgrading the configuration file on every new version.
Move the directory reports/ out from the www/ (Move web assets to /usr/share/monitorix #294):
- The FHS defines /var/lib as the location for "Variable state information".
Include the use of Template-Toolkit:
- This would remove all the static HTML code.
Change the default network port (8080) for HTTP server built-in:
- Suggested by @lyknode based on this bug.
- A different default network port would reduce a conflict with other software.

The text was updated successfully, but these errors were encountered:

IzzySoft · 2020-11-03T14:34:39Z

Create an individual configuration file for each module in conf.d/:

Consider this should not clash with the user configuration which is currently placed there to override the default config. The current Debian package also already places its 00-debian.conf file here. So I'd suggest NOT to use /etc/monitorix/conf.d for this – but maybe following the "Apache way" would be a good idea: Place all module configs in /etc/monitorix/mods_avail and symlink those to be used to /etc/monitorix/mods_enabled. Both listed goals would still be reached.

This still would need a decision on how user overrides then should take place:

as it is now, in arbitrary config files located in conf.d
still in conf.d but using the very same file names – which then are only picked up if their couter-part is found in mods_enabled (this I'd favor)
by adding it with the correct name to mods_enabled instead of the symlink (not a good idea for new additions to a module, which then would not be reflected until updated manually)

As pointed out, I'd go with the second bullet point for module configs, and keep the first for "global stuff" as it's now.

Package each module separately:

Uh, really? I didn't notice the Monitorix package already became that big that "disk space footage" would even be noticable. The goal of only using the modules needed would be covered by the point above: No corresponding config in /etc/monitorix/mods_enabled ⇒ the module is not active. Less headache for both of us (consider I'd have to create a bunch of Debian packages on each release instead of just a single one, which I'd consider quite a bit overkill).

No notes on moving the reports/ dir from my end. As for templates: Yes! Remember we discussed that several years ago already, when speaking about CSS? That would certainly be welcome. Users then could even contribute their own template sets – and we'd have everything from "simple & functional" to "candy & dandy with Gigabytes of Javascript loaded from 100+ 3rd party locations" 🙊 😱 Uh-oh, better strike the latter 😆

mikaku · 2020-11-03T14:59:12Z

Consider this should not clash with the user configuration which is currently placed there to override the default config.

Yes, OK. I think we could follow the "Apache way" (in RHEL) to create a second configuration directory named conf.modules.d/ where would reside all the default module configuration files (system.conf, kern.conf, fs.conf, net.conf, etc.).

Monitorix would load the configuration files in the following order:

/etc/monitorix/monitorix.conf
All files in /etc/monitorix/conf.modules.d/
All files in /etc/monitorix/conf.d/

So each user could still have the current conf.d/ for his own custom configurations.

Less headache for both of us (consider I'd have to create a bunch of Debian packages on each release instead of just a single one, which I'd consider quite a bit overkill).

I didn't consider the idea of releasing new versions of modules separately, that would be indeed very difficult to stay on the track of the current version. I just had in mind the .spec file of a possible future SRPM which creates automatically all the individual packages with only one build, on every new release.

Don't know how package building works on Debian and other distributions, but yeah, perhaps it creates an excessive amount of work.

So, let's remove that feature. 😄.

... to "candy & dandy with Gigabytes of Javascript loaded from 100+ 3rd party locations"...

On my! 🤦

IzzySoft · 2020-11-03T15:55:26Z

[…] So each user could still have the current conf.d/ for his own custom configurations.

Yes, that sounds solid, thanks!

I didn't consider the idea of releasing new versions of modules separately

Ah, OK, then all is fine 😄

Don't know how package building works on Debian

TBH, neither have I. My workflow is based on RedHat *.spec – all else is done by a wrapper tool converting that to Debian-style on-the-fly and spitting out *.deb files. So theoretically I could download all those SRPMs, unpack them, build the corresponding *.deb, clean up… Might work, but needs some time to set up and test.

So, let's remove that feature. 😄.

Yes, that solution I can live with 😆

On my!

Well, you know how it goes today. Question is less what is needed and should be done – but rather what is fancy and technically possible, whether it makes sense or not. Just look at how Github breaks compatibility with most non-Chrome browsers nowadays for no obvious reason, just bells-and-whistles. Or those websites just showing the string "this app requires Javascript" when visited with JS disabled (hey, I didn't want to start an app – I wanted to read a simple page, dude!). And please don't get me started on my NodeJS rant… 🙊 🤐 🤪

Saentist · 2020-11-03T16:08:01Z

It's more easy to have
Main.conf /updatable by installer/
and
user.conf /changes made by user/
main.conf and user.conf is same file.
But if user make changes to user file,
just differences to be used from compared files,
with priority of user.conf

It will be very simple for user to add only section of with it need changes.

lyknode · 2020-11-03T16:42:17Z

What I would suggest for the configuration files is to implement an include directive.

That way, package maintainer and users will be able to accommodate the configuration hierarchy to their convenance.

This would work that way:

Monitorix only read /etc/monitorix/monitorix.conf
For each include directive, load the directory or the file. On already defined key, replace the previous configuration

This would allow, at convenance:

Overrides configuration with conf.modules.d/ and conf.d/ (for RedHat familly)
Selected configuration with mods-available/ and mods-enabled/ (for Debian familly)
Single file override with main.conf and user.conf (as suggested by @Saentist)

IzzySoft · 2020-11-03T16:57:47Z

main.conf and user.conf is same file.

That would cause a mess. First, as a user, at least I'd prefer to have a config with only the stuff I need to adjust – no need for 5k extra lines that just make me dig through stuff I don't need. Second, what happens should the structures of even only sections in the config file change? I'd have to start over, hopefully remembering what options I adjusted. Not UX friendly.

include directive

Not a bad approach. By default, there could be a conf.d/general.conf.sample (note the trailing .sample) shipping along having the initial includes for the modules.

Side-note: no idea whether mods-available/ is Debian specific. I just took that from Apache 😉 Apart from that, it was the "general approach" I wanted to outline.

overrides

Yes, basically. But then be careful where you include what:

read /etc/monitorix/monitorix.conf and process includes if there are any (these should be generic ones only)
read *modules* (pre-shipped default config for the modules), again interpreting includes
read user-specific config files from conf.d/

As a user can simply drop configs to be processed into conf.d I don't see what for includes would be needed here – but if there are any, well, as long as noone is forced to and it's not too much headache for Jordi to implement, I won't argue.

Note that this way you can use a monolitic user.conf if you insist – but you are not forced to. Makes it much easier to (temporarily) enable/disable things if you just need to copy files and restart the daemon. IMHO it's also much easier to maintain.

Saentist · 2020-11-04T15:56:42Z

main.conf and user.conf is same file.

That would cause a mess. First, as a user, at least I'd prefer to have a config with only the stuff I need to adjust – no need for 5k extra lines that just make me dig through stuff I don't need. Second, what happens should the structures of even only sections in the config file change? I'd have to start over, hopefully remembering what options I adjusted. Not UX friendly.

It will not
from MAIN.CONF you need to copy only needed section
to USER.CONF
And because there is a difference it will be used from there
if section is missing in USER.CONF this mean it's with default config.

if you copy for example
Nvidia, mysql, apcups, lm-sensors
then only they will be shown

there is such scenario in other apps already

mikaku · 2020-11-04T16:20:50Z

Main.conf /updatable by installer/

This puts a little risk. The installer will need to upgrade automatically the main.conf on every release and also the comparison of both files on every restart.

main.conf and user.conf is same file.

Two files with the same contents ... I can imagine that people will end up touching only the main.conf file.

This would work that way:

Monitorix only read /etc/monitorix/monitorix.conf

For each include directive, load the directory or the file. On already defined key, replace the previous configuration

If I understand correctly, we will have (the bottom part of) monitorix.conf plenty of lines to include the official modules plus the user's own configurations files, something like this:

include /etc/monitorix/conf.d/system.pm
include /etc/monitorix/conf.d/kern.pm
include /etc/monitorix/conf.d/proc.pm
...
...
[and near the end]
include /etc/monitorix/conf.d/00-debian.conf
include /etc/monitorix/conf.d/mysql.conf

Then on every new release, the new modules won't be reflected in the configuration file unless the user insert the lines manually:

include /etc/monitorix/conf.d/new_module1.pm`
include /etc/monitorix/conf.d/new_module2.pm`

Also, the user must be careful enough to insert the lines of the new modules above of the lines of his own configuration files, otherwise the override won't has effect.

At the end we will end up, again, with and outdated monitorix.conf, with new modules not included, which is one of the main reasons of this change.

Monitorix would load the configuration files in the following order:

/etc/monitorix/monitorix.conf (small, only global options)

All files in /etc/monitorix/conf.modules.d/

All files in /etc/monitorix/conf.d/

I think this is still a good starting point, because:

The main configuration file is easily upgradable as it has few lines.
New modules will be installed automatically in conf.modules.d/, no need user intervention.
User's own configuration files reside in the same place conf.d/.

Still, the current way of enable graphs will require the intervention of user in the main configuration file:

# Graphs (de)activation
# -----------------------------------------------------------------------------
<graph_enable>
        system          = y
        kern            = y
...

Again this is a problem with new releases that come with new modules, as it requires to edit the main configuration file and include manually the new modules. This is tedious and at the end most users don't do that.

So in order to solve this and avoid completely the intervention of the user, we should take in consideration the way suggested by @IzzySoft of enabling/disabling the modules using symbolic links.

This would have the following advantages:

No need file editing by user to enable/disable modules (less error-prone editing).
No need to upgrade the main configuration file to reflect the new modules.
Script-friendly to enable/disable modules (ideal for firewalls, panels, etc.).

Thoughts?

mikaku · 2020-11-04T16:23:42Z

from MAIN.CONF you need to copy only needed section

@Saentist not bad idea, but it not solves the problem of upgrade the main.conf with new modules.

Saentist · 2020-11-04T16:32:46Z

In this scenario is used in ministra/stalker_porta for more then 10 years
https://wiki.infomir.eu/eng/ministra-tv-platform/ministra-installation-guide/configuration-file
there is some old opensource version in github
https://github.com/iptvhakr/stalker_portal

Do not make any changes to the config.ini file. Make changes to the custom.ini file (you need to create it first in the directory /var/www/stalker_portal/server/ ). This will make upgrading to newer versions more convenient.

mikaku · 2020-11-04T16:37:18Z

On IRC @lyknode made me see that his approach could be just simpler with the following pair of lines:

include ./modules-conf.d/
include ./conf.d/

I think that this idea plus the ability to enable/disable modules using symbolic links would be the way to go.

mikaku · 2020-11-04T16:46:59Z

Do not make any changes to the config.ini file. Make changes to the custom.ini file ...

But what if a new release comes with new modules in the config.ini? these modules won't be reflected in custom.ini.

I think that the best way to avoid user from touching the incorrect configuration file is drive him to the correct directory.

That is, if the user tries to edit monitorix.conf and see that he cannot enable, nor configure any module then he won't touch this file and will go directly to modules-conf.d/ or `conf.d/ directory.

Saentist · 2020-11-04T16:53:59Z

Do not make any changes to the config.ini file. Make changes to the custom.ini file ...

But what if a new release comes with new modules in the config.ini? these modules won't be reflected in custom.ini.

It will not be enabled at first
need to make change to this section

system.conf

<graph_enable>
	system		= y
	kern		= y
	proc		= y
	hptemp		= n
	lmsens		= y
	gensens         = y
	ipmi            = n
	ambsens         = n
	nvidia		= n
	disk		= y
	fs		= y
	zfs             = n
	du              = n
	net		= y
	netstat		= y
	tc		= n	
	libvirt		= n
	process		= n
	serv		= y
	mail		= n
	port		= y
	user		= y
	ftp		= n
	apache		= y
	nginx		= n
	lighttpd	= n
	mysql		= y
	mongodb         = n
	varnish         = n
	pagespeed       = n
	squid		= n
	nfss		= n
	nfsc		= n
	bind		= n
	ntp		= n
	chrony          = n
	fail2ban	= n
	icecast		= n
	raspberrypi	= n
	phpapc		= n
	memcached	= n
	apcupsd		= y
	nut             = n
	wowza		= n
	int		= y
	verlihub        = n
</graph_enable>

example in user conf.conf

<graph_enable>
	system		= n
</graph_enable>

this will disable just system graph without any other data needed from section

second if want to change parameters need to add and coresponding section to user.conf to have custom changes, else will be used default.

mikaku · 2020-11-04T16:56:13Z

It will not be enabled at first
need to make change to this section

second if want to change parameters need to add and coresponding section to user.conf to have custom changes, else will be used default.

Exactly, that's what we are trying to avoid.

Saentist · 2020-11-04T16:57:39Z

It will not be enabled at first
need to make change to this section

second if want to change parameters need to add and coresponding section to user.conf to have custom changes, else will be used default.

Exactly, that's what we are trying to avoid.

then make web configuration constructor ;)

p.s. edited precious post to be more clear.

IzzySoft · 2020-11-04T17:20:10Z

@Saentist

It will not. from MAIN.CONF you need to copy only needed section to USER.CONF

May I also quote what you initially wrote?

main.conf and user.conf is same file.

Apart from that, it would mean you'd have to check and update the user config each time modules were added/removed, as the full block would be needed there.

this scenario is used in ministra/stalker_porta for more then 10 years

Lew Nikolayevich Tolstoi had a nice answer to that. Quoting and translating it from what I remember: "Don't let yourself being impressed by someone saying »My friend, we're doing it this way for 40 years already!« You can do something wrong for 40 years." 😄

then make web configuration constructor

Oh my… please not. That'd totally destroy what Jordi wrote above (and what I had in mind a little further up as well):

Script-friendly to enable/disable modules (ideal for firewalls, panels, etc.).

I'm with @mikaku here: we should keep things as simple and straight-forward as possible for the user. Having to tamper with large files, doing a lot of copy-paste, being forced to remember orders of importance – all that leads to mistakes and frustration, on both sides (or rather on all 3 sides, considering distribution-specific stuff the packager has to take care for).

Saentist · 2020-11-04T17:33:40Z

@IzzySoft
user.conf is blank file where you put modified section and modified lines from inside section

in user.conf must not be possible to add sections with lines with not exist in main.conf
except commented lines

IzzySoft · 2020-11-04T18:31:46Z

Which instructions do you think a user would prefer:

To enable a module, simply symlink its config file from A to B.

or

To enable a module:

find its section in the main config

copy-and-paste the entire block to your user config (make sure you keep the syntax right, i.e. copy from the opening tag to the closing tag, to not end up with a broken config)

on each update of the monitorix package, watch out for what has changed and adjust your user config accordingly.

And please also keep the not-so-tech-savvy-users in mind. Enabling/disabling modules via symlinks is a breeze. Having to remove/add/(un)comment blocks in the user config for that is a nightmare.

Saentist · 2020-11-05T10:10:45Z

To enable a module:

find its section in the main config

copy-and-paste the entire block to your user config (make sure you keep the syntax right, i.e. copy from the opening tag to the closing tag, to not end up with a broken config)

its not so hard to copy config
VALUE = VARIABLE

Value is always static
= is what it is ;)
Variable is what is application start need to use
Extra tabs/spaces need to be ignored
Opening and closing TAG is important to exist.

mikaku mentioned this issue Nov 3, 2020

Move web assets to /usr/share/monitorix #294

Closed

mikaku changed the title ~~4.x new features list~~ 4.x major/breaking new features list Nov 3, 2020

Saentist mentioned this issue Mar 23, 2022

BUG monitorix.conf overwrite #414

Closed

WarmChocolateCake mentioned this issue Sep 8, 2022

BTRFS support for monitoring #373

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4.x major/breaking new features list #295

4.x major/breaking new features list #295

mikaku commented Nov 3, 2020 •

edited

Loading

IzzySoft commented Nov 3, 2020 •

edited

Loading

mikaku commented Nov 3, 2020

IzzySoft commented Nov 3, 2020

Saentist commented Nov 3, 2020 •

edited

Loading

lyknode commented Nov 3, 2020

IzzySoft commented Nov 3, 2020

Saentist commented Nov 4, 2020 •

edited

Loading

mikaku commented Nov 4, 2020

mikaku commented Nov 4, 2020

Saentist commented Nov 4, 2020 •

edited

Loading

mikaku commented Nov 4, 2020

mikaku commented Nov 4, 2020 •

edited

Loading

Saentist commented Nov 4, 2020 •

edited

Loading

mikaku commented Nov 4, 2020

Saentist commented Nov 4, 2020 •

edited

Loading

IzzySoft commented Nov 4, 2020

Saentist commented Nov 4, 2020

IzzySoft commented Nov 4, 2020

Saentist commented Nov 5, 2020 •

edited

Loading

4.x major/breaking new features list #295

4.x major/breaking new features list #295

Comments

mikaku commented Nov 3, 2020 • edited Loading

IzzySoft commented Nov 3, 2020 • edited Loading

mikaku commented Nov 3, 2020

IzzySoft commented Nov 3, 2020

Saentist commented Nov 3, 2020 • edited Loading

lyknode commented Nov 3, 2020

IzzySoft commented Nov 3, 2020

Saentist commented Nov 4, 2020 • edited Loading

mikaku commented Nov 4, 2020

mikaku commented Nov 4, 2020

Saentist commented Nov 4, 2020 • edited Loading

mikaku commented Nov 4, 2020

mikaku commented Nov 4, 2020 • edited Loading

Saentist commented Nov 4, 2020 • edited Loading

mikaku commented Nov 4, 2020

Saentist commented Nov 4, 2020 • edited Loading

IzzySoft commented Nov 4, 2020

Saentist commented Nov 4, 2020

IzzySoft commented Nov 4, 2020

Saentist commented Nov 5, 2020 • edited Loading

mikaku commented Nov 3, 2020 •

edited

Loading

IzzySoft commented Nov 3, 2020 •

edited

Loading

Saentist commented Nov 3, 2020 •

edited

Loading

Saentist commented Nov 4, 2020 •

edited

Loading

Saentist commented Nov 4, 2020 •

edited

Loading

mikaku commented Nov 4, 2020 •

edited

Loading

Saentist commented Nov 4, 2020 •

edited

Loading

Saentist commented Nov 4, 2020 •

edited

Loading

Saentist commented Nov 5, 2020 •

edited

Loading