Merge branch 'develop'

wvandeun · Sep 20, 2021 · b08761e · b08761e
2 parents 62e969e + 76d3d9b
commit b08761e
Show file tree

Hide file tree

Showing 43 changed files with 14,836 additions and 3,661 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,16 @@
 # Changelog
 
+## v0.3.0 - (2021-09-20)
+
+### Enhancements
+
+* [#27](https://github.com/wvandeun/nornir_netbox/pull/27) add `use_platform_napalm_driver` configuration option, which sets the Host platform to the NetBox configured Napalm driver @johanek
+* [#14](https://github.com/wvandeun/nornir_netbox/pull/14/commits) add support for SimpleInventory defaults and groups
+
+### Bug Fixes
+
+* [#30](https://github.com/wvandeun/nornir_netbox/pull/30) fix a documentation issue in `usage.md` @MajesticFalcon
+
 ## 0.2.0 - November 24 2020
 
 * implements support for virtual-machines in NetBoxInventory2 [@patrickdaj](https://github.com/patrickdaj)

diff --git a/README.md b/README.md
@@ -107,6 +107,45 @@ Arguments:
         (defaults to False)
     use_platform_slug: Use the NetBox platform slug for the platform attribute of a Host
         (defaults to False)
+    use_platform_napalm_driver: Use the Netbox platform napalm driver setting for the platform attribute of a Host
+        (defaults to False)
+    group_file: path to file with groups definition. If it doesn't exist it will be skipped
+    defaults_file: path to file with defaults definition. If it doesn't exist it will be skipped
+```
+
+Only one of use_platform_slug and use_platform_napalm_driver can be set to true.
+
+### Using SimpleInventory groups and defaults file
+
+This feature allows you to use SimpleInventory groups and defaults file with NetBoxInventory2.
+NetBoxInventory2 will automatically create groups for specific attributes of NetBox devices or virtual-machines.
+The following attributes will be used to create groups:
+- site
+- platform
+- device_role
+- device_type
+- manufacturer (extracted from device_type)
+
+From these attributes we will use the slug property to create the group. Groups will be created using the following format `attribute__slug`.
+For example, assume a device that is part of a site `Site 1` for which the slug is `site-1`. NetBoxInventory2 will create a group with the name `site__site-1` and add the device to the group.
+
+These groups can then be used in a groups file and you can add data to them, similar to how the SimpleInventory plugin works.
+More information on on the defaults and groups file can be found in [Nornir's documentation](https://nornir.readthedocs.io/en/latest/tutorial/inventory.html).
+
+```python
+from nornir import InitNornir
+
+nr = InitNornir(
+    inventory={
+        "plugin":"NBInventory",
+        "options": {
+            "nb_url": "https://netbox.local",
+            "nb_token": "123_NETBOX_API_TOKEN_456",
+            "group_file": "inventory/groups.yml",
+            "defaults_file": "inventory/defaults.yml"
+        }
+    }
+)
 ```
 
 # Useful Links

diff --git a/docs/index.md b/docs/index.md
@@ -18,7 +18,16 @@ Nornir Host objects will be created from the data in your NetBox database:
 * the platform will be set to the platform defined for the device in NetBox
 * all other attributes of a device/virtual machine object in NetBox will be stored under the data attribute of the Nornir Host 
 
-Here is an example on to quickly setup NetBoxInventory2 to retrieve inventory data from NetBox:
+Nornir Group objects will be created from specific device and virtual machine attributes in your NetBox database:
+* groups will be created for the following attributes of devices and virtual machines:
+  - site
+  - platform
+  - device_role (role for virtual machines)
+  - device_type
+  - manufacturer
+* the name of the group will be formed by the combination of the name of the property and the slug of it's value, separated by a double underscore `attribute__slug` (for example a device with has its platform defined as `Cisco IOS`, for which the slug is `ios`. NetBoxInventory2 will create a group `platform__ios` and add the device to the group.)
+* devices and virtual machines will be added to their respective groups
+* defaults and groups files can then be used to define properties for devices or virtual machines (similar to to how Nornir's SimpleInventory plugin works)
 
 ### Installation
 

diff --git a/docs/usage.md b/docs/usage.md
@@ -1,9 +1,9 @@
-# Using NetBox as an inventory source 
+# Using NetBox as an inventory source
 
 Before we can use NetBox as an inventory source, we need to know the following 2 properties:
 
 * The URL of your NetBox instance
-* A NetBox [API token](https://netbox.readthedocs.io/en/stable/rest-api/authentication/#tokens) 
+* A NetBox [API token](https://netbox.readthedocs.io/en/stable/rest-api/authentication/#tokens)
 
 You can setup Nornir to leverage NetBoxInvetory2 as the inventory source in 2 ways:
 
@@ -67,7 +67,7 @@ NetBox instance URL.
 | default              | http://localhost:8000 |
 | required             | True                  |
 | environment variable | NB\_URL               |
- 
+
 ### NetBox API token
 
 NetBox API token.
@@ -93,7 +93,7 @@ Alternatively accepts a path the CA bundle file to use for certificate validatio
 
 ### Flatten custom fields
 
-This option allows you to "flatten" custom fields. By default a custom fields for a NetBox device or VM, will be stored in the custom_fields attribute of the data attribute of a Nornir host. 
+This option allows you to "flatten" custom fields. By default a custom fields for a NetBox device or VM, will be stored in the custom_fields attribute of the data attribute of a Nornir host.
 
 Enabling `flatten_custom_fields` will modify that behaviour, so that each custom field is stored direclty as an attibute of the dat attribute of a Host, which makes working with custom fields a little bit easier.
 
@@ -128,7 +128,7 @@ Example with `flatten_custom_fields` enabled
 
 ### Filter parameters
 
-Filter parameters allow you to filter the inventory data returned by thet NetBox API. 
+Filter parameters allow you to filter the inventory data returned by thet NetBox API.
 
 The NetBox API allows you to filter the returned data by attaching one or more query parameters to the request url. More information can be found in [NetBox's documentation](https://netbox.readthedocs.io/en/stable/rest-api/filtering/).
 
@@ -146,8 +146,8 @@ nr = InitNornir(
     inventory={
         "plugin": NetBoxInventory2,
         "options": {
-            "nb_url": "http://netbox.local:8000"
-            "nb_token": "1234567890"
+            "nb_url": "http://netbox.local:8000",
+            "nb_token": "1234567890",
             "filter_parameters": {"site": "site1"}
         }
     }
@@ -160,8 +160,8 @@ nr = InitNornir(
     inventory={
         "plugin": NetBoxInventory2,
         "options": {
-            "nb_url": "http://netbox.local:8000"
-            "nb_token": "1234567890"
+            "nb_url": "http://netbox.local:8000",
+            "nb_token": "1234567890",
             "filter_parameters": {
                 "site": "site1",
                 "platform": "cisco_ios",
@@ -177,8 +177,8 @@ nr = InitNornir(
     inventory={
         "plugin": NetBoxInventory2,
         "options": {
-            "nb_url": "http://netbox.local:8000"
-            "nb_token": "1234567890"
+            "nb_url": "http://netbox.local:8000",
+            "nb_token": "1234567890",
             "filter_parameters": {
                 "site": ["site1", "site2"],
                 "platform": "cisco_ios",
@@ -190,18 +190,32 @@ nr = InitNornir(
 
 ### Use platform slugs
 
-NetBox device/vm's have a platform attribute that indicates the type of operating system that is running on the device. This attribut is directly mappend to the Nornir Host's platform attribute, so that connection plugins understand which driver needs to be used to connect to the device.
+NetBox device/vm's have a platform attribute that indicates the type of operating system that is running on the device. This attribute is directly mappend to the Nornir Host's platform attribute, so that connection plugins understand which driver needs to be used to connect to the device.
 
 By default the name attribute of a NetBox platform is mapped to the Nornir Host's platform. Use platform slugs allows you to use the slug of the NetBox platform instead.
 
-Wether or not you need to enable this option depends on how you defined your platforms in NetBox.
+Whether or not you need to enable this option depends on how you defined your platforms in NetBox. Only one of use_platform_slugs and use_platform_napalm_driver can be set as true.
 
 | name        | use\_platform\_slugs                                                            |
 |-------------|---------------------------------------------------------------------------------|
 | type        | bool                                                                            |
 | default     | False                                                                           |
 | required    | False                                                                           |
 
+
+### Use platform NAPALM driver
+
+Use platform NAPLAM driver works like use platform slugs, but uses the NAPLAM driver attibuted from the NetBox platform instead.
+
+Whether or not you need to enable this option depends on how you defined your platforms in NetBox. Only one of use_platform_slugs and use_platform_napalm_driver can be set as true.
+
+| name        | use\_platform\_napalm\_driver                                                            |
+|-------------|---------------------------------------------------------------------------------|
+| type        | bool                                                                            |
+| default     | False                                                                           |
+| required    | False                                                                           |
+
+
 ### Include Virtual Machines
 
 Enable this option to also create Nornir Hosts for virtual machines stored in the NetBox database.
@@ -211,3 +225,26 @@ Enable this option to also create Nornir Hosts for virtual machines stored in th
 | type     | bool         |
 | default  | False        |
 | required | False        |
+
+
+### Defaults file
+
+Path to file with the defaults definition. If the file doesn't exist, it will be skipped.
+More information on on the defaults file can be found in [Nornir's documentation](https://nornir.readthedocs.io/en/latest/tutorial/inventory.html).
+
+| name     | defaults\_file |
+|----------|----------------|
+| type     | str            |
+| default  | "defaults.yaml |
+| required | False          |
+
+### Group file
+
+Path to file with the groups definition. If the file doesn't exist, it will be skipped.
+More information on on the groups file can be found in [Nornir's documentation](https://nornir.readthedocs.io/en/latest/tutorial/inventory.html).
+
+| name     | group\_file  |
+|----------|--------------|
+| type     | str          |
+| default  | "groups.yaml |
+| required | False        |
diff --git a/nornir_netbox/plugins/inventory/netbox.py b/nornir_netbox/plugins/inventory/netbox.py
@@ -1,16 +1,25 @@
 import os
 import warnings
-from typing import Any, Dict, List, Optional, Union, Type
+from typing import Any
+from typing import Dict
+from typing import List
+from typing import Optional
+from typing import Union
+from typing import Type
+from pathlib import Path
 
 from nornir.core.inventory import ConnectionOptions
 from nornir.core.inventory import Defaults
+from nornir.core.inventory import Group
 from nornir.core.inventory import Groups
 from nornir.core.inventory import Host
 from nornir.core.inventory import HostOrGroup
 from nornir.core.inventory import Hosts
 from nornir.core.inventory import Inventory
+from nornir.core.inventory import ParentGroups
 
 import requests
+import ruamel.yaml
 
 
 def _get_connection_options(data: Dict[str, Any]) -> Dict[str, ConnectionOptions]:
@@ -27,6 +36,18 @@ def _get_connection_options(data: Dict[str, Any]) -> Dict[str, ConnectionOptions
     return cp
 
 
+def _get_defaults(data: Dict[str, Any]) -> Defaults:
+    return Defaults(
+        hostname=data.get("hostname"),
+        port=data.get("port"),
+        username=data.get("username"),
+        password=data.get("password"),
+        platform=data.get("platform"),
+        data=data.get("data"),
+        connection_options=_get_connection_options(data.get("connection_options", {})),
+    )
+
+
 def _get_inventory_element(
     typ: Type[HostOrGroup], data: Dict[str, Any], name: str, defaults: Defaults
 ) -> HostOrGroup:
@@ -187,6 +208,11 @@ class NetBoxInventory2:
             (defaults to False)
         use_platform_slug: Use the NetBox platform slug for the platform attribute of a Host
             (defaults to False)
+        use_platform_napalm_driver: Use the Netbox platform napalm driver setting for the
+            platform attribute of a Host
+            (defaults to False)
+        group_file: path to file with groups definition. If it doesn't exist it will be skipped
+        defaults_file: path to file with defaults definition. If it doesn't exist it will be skipped
     """
 
     def __init__(
@@ -198,6 +224,9 @@ def __init__(
         filter_parameters: Optional[Dict[str, Any]] = None,
         include_vms: bool = False,
         use_platform_slug: bool = False,
+        use_platform_napalm_driver: bool = False,
+        group_file: str = "groups.yaml",
+        defaults_file: str = "defaults.yaml",
         **kwargs: Any,
     ) -> None:
         filter_parameters = filter_parameters or {}
@@ -211,12 +240,54 @@ def __init__(
         self.filter_parameters = filter_parameters
         self.include_vms = include_vms
         self.use_platform_slug = use_platform_slug
+        self.use_platform_napalm_driver = use_platform_napalm_driver
 
         self.session = requests.Session()
         self.session.headers.update({"Authorization": f"Token {nb_token}"})
         self.session.verify = ssl_verify
+        self.group_file = Path(group_file).expanduser()
+        self.defaults_file = Path(defaults_file).expanduser()
+
+        if self.use_platform_slug and self.use_platform_napalm_driver:
+            raise ValueError(
+                "Only one of use_platform_slug and use_platform_napalm_driver can be set to true"
+            )
+
+    @staticmethod
+    def _extract_device_groups(device: Dict[str, Any]) -> List[str]:
+        extract_group_attributes = [
+            {"name": "site", "path": ["site", "slug"]},
+            {"name": "platform", "path": ["platform"]},  # older netbox versions
+            {"name": "platform", "path": ["platform", "slug"]},
+            {"name": "device_role", "path": ["device_role", "slug"]},
+            {"name": "device_role", "path": ["role", "slug"]},  # vm's
+            {"name": "manufacturer", "path": ["device_type", "manufacturer", "slug"]},
+            {"name": "device_type", "path": ["device_type", "slug"]},
+        ]
+
+        groups = []
+        for group in extract_group_attributes:
+            data = device
+            for hop in group.get("path", []):
+                v = data.get(hop)
+                if isinstance(v, dict):
+                    data = v
+                elif isinstance(v, str) and group.get("path", [])[-1] == hop:
+                    groups.append(f"{group.get('name', '')}__{v}")
+                else:
+                    # Unable to extract group
+                    continue
+        return groups
 
     def load(self) -> Inventory:
+        yml = ruamel.yaml.YAML(typ="safe")
+
+        platforms: List[Dict[str, Any]] = []
+
+        if self.use_platform_napalm_driver:
+            platforms = self._get_resources(
+                url=f"{self.nb_url}/api/dcim/platforms/?limit=0", params={}
+            )
 
         nb_devices: List[Dict[str, Any]] = []
 
@@ -237,6 +308,23 @@ def load(self) -> Inventory:
         groups = Groups()
         defaults = Defaults()
 
+        if self.defaults_file.exists():
+            with self.defaults_file.open("r") as f:
+                defaults_dict = yml.load(f) or {}
+            defaults = _get_defaults(defaults_dict)
+        else:
+            defaults = Defaults()
+
+        if self.group_file.exists():
+            with self.group_file.open("r") as f:
+                groups_dict = yml.load(f) or {}
+
+            for n, g in groups_dict.items():
+                groups[n] = _get_inventory_element(Group, g, n, defaults)
+
+            for g in groups.values():
+                g.groups = ParentGroups([groups[g] for g in g.groups])
+
         for device in nb_devices:
             serialized_device: Dict[Any, Any] = {}
             serialized_device["data"] = device
@@ -256,6 +344,14 @@ def load(self) -> Inventory:
 
             if isinstance(device["platform"], dict) and self.use_platform_slug:
                 platform = device["platform"].get("slug")
+            elif (
+                isinstance(device["platform"], dict) and self.use_platform_napalm_driver
+            ):
+                platform = [
+                    platform
+                    for platform in platforms
+                    if device["platform"]["slug"] == platform["slug"]
+                ][0]["napalm_driver"]
             elif isinstance(device["platform"], dict):
                 platform = device["platform"].get("name")
             else:
@@ -271,6 +367,14 @@ def load(self) -> Inventory:
                 Host, serialized_device, name, defaults
             )
 
+            groups_extracted = self._extract_device_groups(device)
+
+            for group in groups_extracted:
+                if group not in groups.keys():
+                    groups[group] = _get_inventory_element(Group, {}, group, defaults)
+
+            hosts[name].groups = ParentGroups([groups[g] for g in groups_extracted])
+
         return Inventory(hosts=hosts, groups=groups, defaults=defaults)
 
     def _get_resources(self, url: str, params: Dict[str, Any]) -> List[Dict[str, Any]]: