Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to disable certificate verification during connecting to CS, update docs, and fix anonymizer failing tests #112

Merged
merged 4 commits into from
May 17, 2024
Merged

Ability to disable certificate verification during connecting to CS, update docs, and fix anonymizer failing tests #112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3a11037
feat: Add ability to disable certificate verification during connecti…
melbeltagy May 13, 2024
41af075
docs: Update README and collector module's documentation
melbeltagy May 13, 2024
77c4738
docs: Update collector module's documentation
melbeltagy May 14, 2024
3ec4841
fix: Stop using dictionary unpacking for single members to fix failin…
melbeltagy May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,14 @@ Instructions on setting up the MongoDB can be found in the [Database Module](./d

The modules should be set up in the following order:

1. [Collector](./docs/collector_module.md) (before others)
2. [Corrector](./docs/corrector_module.md) (after Collector, before others)
3. [Reports](./docs/reports_module.md) (optional, after previous)
4. [Opendata](./docs/opendata_module.md) (optional, after previous)
5. [Networking](./docs/networking_module.md) (optional, after Opendata)
6. [Opendata Collector module](./docs/opendata_collector_module.md) (optional, after Opendata)
- [Database module](./docs/database_module.md)
- [Collector module](./docs/collector_module.md)
- [Corrector module](./docs/corrector_module.md)
- [Reports module](./docs/reports_module.md)
- [Anonymizer module](./docs/anonymizer_module.md)
- [Opendata module](./docs/opendata_module.md)
- [Networking/Visualizer module](./docs/networking_module.md)
- [Opendata Collector module](./docs/opendata_collector_module.md)

## Programming language

Expand Down
6 changes: 3 additions & 3 deletions anonymizer_module/metrics_statistics/statistics_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ def collect_statistics(settings: dict, logger: Logger, output_only: bool = False

statistics: StatisticalData = {
**requests_counts,
**{'member_count': json.dumps(member_counts)},
**{'service_count': len(services)},
**{'service_request_count': json.dumps(services_counts)}
'member_count': json.dumps(member_counts),
'service_count': len(services),
'service_request_count': json.dumps(services_counts)
}
if output_only:
logger.info('Metrics statistical data:\n\n%s', pformat(statistics, indent=2, width=2))
Expand Down
8 changes: 7 additions & 1 deletion collector_module/etc/settings.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ xroad:
protocol: http://
host: <FILL>
timeout: 10
# path to client's certificate
tls-client-certificate:
# path to client's private key
tls-client-key:
# path to server's certificate, or False to disable server certificate verification
tls-server-certificate:

# Security server used to contact
security-server:
Expand All @@ -77,7 +83,7 @@ xroad:
tls-client-certificate:
# path to client's private key
tls-client-key:
# path to server's certificate
# path to server's certificate, or False to disable server certificate verification
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently I believe that and empty value also disables this. Since older installations wouldn't have the parameter, can me also have the same behavior in these cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please correct me if I'm wrong, but as per my understanding for this section in the documentation, an empty value would still verify the SSL certificate (since it's the default behavior).

The reason why I added the new feature of disabling certificate verification while connecting to central server (not security server), is because I had SSL enabled for the central server locally, but collector module was failing to verify the certificate and hence failing to start.

Perhaps I am missing something or had my collector misconfigured and that there is way (without my modifications to collector_module/opmon_collector/central_server_client.py) that I can disable the certificate verification during SSL connection?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, sorry, for some reason I thought this related to enabling / disabling mTLS. In this case I think it is fine.

And it is a good catch about the Central Server, it previously did not support HTTPS, only HTTP. That's why the X-Road Metrics code hasn't accounted for the HTTPS connection to the global configuration yet.

tls-server-certificate:

# X-Road service configuration used to fetch operational monitoring requests.
Expand Down
16 changes: 12 additions & 4 deletions collector_module/opmon_collector/central_server_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,14 @@

class CentralServerClient:
def __init__(self, xroad_settings, logger_m):
self.url = f"{xroad_settings['central-server']['protocol']}{xroad_settings['central-server']['host']}"
self.timeout = xroad_settings['central-server']['timeout']
central_server_settings = xroad_settings['central-server']
self.url = f"{central_server_settings['protocol']}{central_server_settings['host']}"
self.timeout = central_server_settings['timeout']
self.server_cert = central_server_settings.get('tls-server-certificate')
self.client_cert = (
central_server_settings.get('tls-client-certificate'),
central_server_settings.get('tls-client-key')
)
self.logger_m = logger_m

def get_security_servers(self):
Expand All @@ -43,13 +49,15 @@ def get_security_servers(self):
def _get_shared_params(self):
internal_conf_url = f'{self.url}/internalconf'

global_conf = requests.get(internal_conf_url, timeout=self.timeout)
global_conf = requests.get(internal_conf_url, timeout=self.timeout, cert=self.client_cert,
verify=self.server_cert)
global_conf.raise_for_status()
# NB! re.search global configuration regex might be changed
# according version naming or other future naming conventions
data = global_conf.content.decode('utf-8')
s = re.search(r'Content-location: (/V\d+/\d+/shared-params.xml)', data)
shared_params = requests.get(f'{self.url}{s.group(1)}', timeout=self.timeout)
shared_params = requests.get(f'{self.url}{s.group(1)}', timeout=self.timeout,
cert=self.client_cert, verify=self.server_cert)
shared_params.raise_for_status()
return shared_params

Expand Down
47 changes: 27 additions & 20 deletions docs/collector_module.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

| [![X-ROAD](img/xroad-metrics-100.png)](https://x-road.global/) | ![European Union / European Regional Development Fund / Investing in your future](img/eu_rdf_100_en.png "Documents that are tagged with EU/SF logos must keep the logos until 1.11.2022. If it has not stated otherwise in the documentation. If new documentation is created using EU/SF resources the logos must be tagged appropriately so that the deadline for logos could be found.") |
| :-------------------------------------------------- | -------------------------: |
| [![X-ROAD](img/xroad-metrics-100.png)](https://x-road.global/) | ![European Union / European Regional Development Fund / Investing in your future](img/eu_rdf_100_en.png "Documents that are tagged with EU/SF logos must keep the logos until 1.11.2022. If it has not stated otherwise in the documentation. If new documentation is created using EU/SF resources the logos must be tagged appropriately so that the deadline for logos could be found.") |
|:---------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|

# X-Road Metrics - Collector Module

Expand Down Expand Up @@ -68,13 +68,13 @@ sudo apt-get install xroad-metrics-collector
```

The installation package automatically installs following items:
* xroad-metrics-collector command to run the collector manually
* `xroad-metrics-collector` command to run the collector manually
* Linux user named _xroad-metrics_ and group _xroad-metrics_
* settings file _/etc/xroad-metrics/collector/settings.yaml_
* cronjob in _/etc/cron.d/xroad-metrics-collector-cron_ to run collector automatically every three hours
* log folders to _/var/log/xroad-metrics/collector/_

Only _xroad-metrics_ user can access the settings files and run xroad-metrics-collector command.
Only _xroad-metrics_ user can access the settings files and run `xroad-metrics-collector` command.

To use collector you need to fill in your X-Road and MongoDB configuration into the settings file.
Refer to section [Collector Configuration](#collector-configuration)
Expand All @@ -92,23 +92,29 @@ To use collector you need to fill in your X-Road and MongoDB configuration into
```bash
sudo vi /etc/xroad-metrics/collector/settings.yaml
```
> [!TIP]
> For a complete list of available settings, please refer to this [settings.yaml](../collector_module/etc/settings.yaml) template file.

Settings that the user must fill in:
* X-Road instance name
* Central- and Security Server hosts
* Central and Security Server hosts
* X-Road client used to collect the monitoring data
* username and password for the collector module MongoDB user

To run collector for multiple X-Road instances, a settings profile for each instance can be created. For example to have profiles DEV, TEST and PROD create three copies of `setting.yaml`
file named `settings_DEV.yaml`, `settings_TEST.yaml` and `settings_PROD.yaml`.
Then fill the profile specific settings to each file and use the --profile
flag when running xroad-metrics-collector. For example to run using the TEST profile:
```
xroad-metrics-collector --profile TEST collect
```

`xroad-metrics-collector` command searches the settings file first in current working direcrtory, then in
_/etc/xroad-metrics/collector/_
#### Configurations for multiple X-Road instances

To run collector for multiple X-Road instances, a settings profile for each instance can be created.
1. To have profiles `DEV`, `TEST` and `PROD`, create three copies of `setting.yaml`
file named `settings_DEV.yaml`, `settings_TEST.yaml` and `settings_PROD.yaml` respectively.
2. Fill the profile specific settings to each file.
3. Use the `--profile` flag when running `xroad-metrics-collector`.
For example, to run using the `TEST` profile:
```shell
xroad-metrics-collector --profile TEST collect
```
> [!IMPORTANT]
> `xroad-metrics-collector` command searches the settings file first in current working directory, then in
`/etc/xroad-metrics/collector/`

### Using client certificate (mTLS) to connect to security server

Expand All @@ -128,10 +134,11 @@ security-server:
tls-client-key: /path/to/client.key # path to client's private key
tls-server-certificate: /path/to/server.crt # path to server's certificate
```
Notes:
Client's certificate has to be sent to security server administrator.
Server certificate has to be sent by server's administrator and save in client's location.
`tls-server-certificate` can be set to `False` to disable server certificate verification.

> [!Note]
> - Client's certificate has to be sent to security server administrator.
> - Server certificate has to be sent by server's administrator and save in client's location.
> - `tls-server-certificate` can be set to `False` to disable server certificate verification.

### Manual usage

Expand Down Expand Up @@ -230,7 +237,7 @@ In case of "activity": "collector_end", the "msg" includes values separated by c

The **collector module** log handler is compatible with the logrotate utility. To configure log rotation for the example setup above, create the file:

```
```bash
sudo vi /etc/logrotate.d/xroad-metrics-collector
```

Expand Down
Loading