Policy Datasources for leveraging built-in services #696

akhomyakov · 2021-12-02T21:12:55Z

Hi folks,
Looking at the nsxt_ns_groups and nsxt_ns_services data resources (currently in Beta) they appear like exactly what is needed to make use of built in services and so on.
What I can't figure out is how to refer to items imported. Is there a way to output the content of what is imported?

<snip for brevity>
data "nsxt_ns_groups" "NS-GROUPS" {
}
data "nsxt_ns_services" "NS-SERVICES" {
}
resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE" {
  display_name = "SHARED-INFRASTRUCTURE"
  description  = "Service access policy shared by all tenants"
  category     = "Infrastructure"
  locked       = false
  stateful     = true
  tcp_strict   = false

  rule {
    display_name       = "ICMP"
    destination_groups = [data.nsxt_ns_groups.NS-GROUPS.items["INTDMZ"]]
    action             = "ALLOW"
    services           = [data.nsxt_ns_services.NS-SERVICES.items["ICMP-ALL"]]
    logged             = true
  }
<snip for brevity>

<snip for brevity>
╷
│ Error: Invalid index
│
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 12, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   12:     destination_groups = [data.nsxt_ns_groups.NS-GROUPS.items["INTDMZ"]]
│     ├────────────────
│     │ data.nsxt_ns_groups.NS-GROUPS.items is map of string with 9 elements
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 14, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   14:     services           = [data.nsxt_ns_services.NS-SERVICES.items["ICMP-ALL"]]
│     ├────────────────
│     │ data.nsxt_ns_services.NS-SERVICES.items is map of string with 373 elements
│
│ The given key does not identify an element in this collection value.
╵
<snip for brevity>

Just using the API doesn't seem to yeild the right results

// GET https://{{nsxmanager}}/policy/api/v1/infra/domains/default/groups
{
    "results": [
        {
            "expression": [
                {
                    "ip_addresses": [
                        "10.223.35.0/24"
                    ],
                    "resource_type": "IPAddressExpression",
                    "id": "b67d26da-ae46-47b0-9ed9-b954c4447864",
                    "path": "/infra/domains/default/groups/a8839a31-dbbf-4c61-ada1-32963711ef94/ip-address-expressions/b67d26da-ae46-47b0-9ed9-b954c4447864",
                    "relative_path": "b67d26da-ae46-47b0-9ed9-b954c4447864",
                    "parent_path": "/infra/domains/default/groups/a8839a31-dbbf-4c61-ada1-32963711ef94",
                    "marked_for_delete": false,
                    "overridden": false,
                    "_protection": "NOT_PROTECTED"
                }
            ],
            "extended_expression": [],
            "reference": false,
            "resource_type": "Group",
            "id": "a8839a31-dbbf-4c61-ada1-32963711ef94",
            "display_name": "INTDMZ",
            "description": "Terraform provisioned Group",
            "tags": [],
            "path": "/infra/domains/default/groups/a8839a31-dbbf-4c61-ada1-32963711ef94",
            "relative_path": "a8839a31-dbbf-4c61-ada1-32963711ef94",
            "parent_path": "/infra/domains/default",
            "unique_id": "d214f969-60f7-44c5-acd6-8b436639f606",
            "marked_for_delete": false,
            "overridden": false,
            "_create_user": "<snip>",
            "_create_time": 1637272533808,
            "_last_modified_user": "[email protected]",
            "_last_modified_time": 1637272533809,
            "_system_owned": false,
            "_protection": "NOT_PROTECTED",
            "_revision": 0
        },
<snip for brevity>

// GET https://{{nsxmanager}}/policy/api/v1/infra/services
<snip for brevity>
        {
            "is_default": true,
            "service_entries": [
                {
                    "protocol": "ICMPv6",
                    "resource_type": "ICMPTypeServiceEntry",
                    "id": "ICMPv6-ALL",
                    "display_name": "ICMPv6-ALL",
                    "path": "/infra/services/ICMP-ALL/service-entries/ICMPv6-ALL",
                    "relative_path": "ICMPv6-ALL",
                    "parent_path": "/infra/services/ICMP-ALL",
                    "marked_for_delete": false,
                    "overridden": false,
                    "_create_user": "system",
                    "_create_time": 1555511297346,
                    "_last_modified_user": "system",
                    "_last_modified_time": 1555511297346,
                    "_system_owned": true,
                    "_protection": "NOT_PROTECTED",
                    "_revision": 0
                },
                {
                    "protocol": "ICMPv4",
                    "resource_type": "ICMPTypeServiceEntry",
                    "id": "ICMPv4-ALL",
                    "display_name": "ICMPv4-ALL",
                    "path": "/infra/services/ICMP-ALL/service-entries/ICMPv4-ALL",
                    "relative_path": "ICMPv4-ALL",
                    "parent_path": "/infra/services/ICMP-ALL",
                    "marked_for_delete": false,
                    "overridden": false,
                    "_create_user": "system",
                    "_create_time": 1555511297345,
                    "_last_modified_user": "system",
                    "_last_modified_time": 1555511297345,
                    "_system_owned": true,
                    "_protection": "NOT_PROTECTED",
                    "_revision": 0
                }
            ],
            "service_type": "NON_ETHER",
            "resource_type": "Service",
            "id": "ICMP-ALL",
            "display_name": "ICMP ALL",
            "description": "ICMP ALL",
            "path": "/infra/services/ICMP-ALL",
            "relative_path": "ICMP-ALL",
            "parent_path": "/infra",
            "marked_for_delete": false,
            "overridden": false,
            "_create_user": "system",
            "_create_time": 1555511297344,
            "_last_modified_user": "system",
            "_last_modified_time": 1620756933397,
            "_system_owned": true,
            "_protection": "NOT_PROTECTED",
            "_revision": 1
        },
<snip for brevity>

The text was updated successfully, but these errors were encountered:

akhomyakov · 2021-12-02T21:37:22Z

Actually, seems like terraform's "output" resource help here

output "ns_groups" {
  value = data.nsxt_ns_groups.NS-GROUPS.items
}

output "ns_services" {
  value = data.nsxt_ns_services.NS-SERVICES.items
}

but the names don't match what is seen in the GUI or even in the API

// API output shown in the OP.
// TF output
Changes to Outputs:
  + ns_groups   = {
      + "Edge_NSGroup"                                    = "7b98e78e-79ae-40b8-aaca-7b959ee47382"
      + "ServiceInsertion_NSGroup"                        = "e2e8ec10-bdaa-4a35-99d2-0d7727380bb0"
      + "default.INTDMZ"                                  = "d214f969-60f7-44c5-acd6-8b436639f606"
      <snip for brevity>
    }
  + ns_services = {
      <snip for brevity>
      + "ICMP Destination Unreachable"                                 = "63d122cf-47d6-44e8-bb58-0453d9bb6493"
      + "ICMP Echo Reply"                                              = "c54b2d86-6327-41ff-a3fc-c67171b6ba63"
      + "ICMP Echo Request"                                            = "5531a880-61aa-42cc-ba4b-13b9ea611e2f"
      + "ICMP Redirect"                                                = "82e6a462-431b-4e2d-8e5e-492a3461913b"
      + "ICMP Router Advertisement"                                    = "2255c641-fc57-449d-b973-e2da77353fe6"
      + "ICMP Router Solicitation"                                     = "93d11f70-6a67-46b9-bcb8-4761d0ed0148"
      + "ICMP Source Quench"                                           = "4f2e4cf1-09ff-4ba2-9f50-065bbe2f6b77"
      + "ICMP Time Exceeded"                                           = "76f12d8c-e9ea-4d08-bb6c-267be759756e"
      + "ICMP-entry"                                                   = "dde6a2fc-fe52-472e-8b10-44573dcbc1bb"
      + "ICMPv4-ALL"                                                   = "82e6a462-431b-4e2d-8e5e-492a3461913c"
      + "ICMPv6-ALL"                                                   = "82e6a462-431b-4e2d-8e5e-492a3461913d"

So is ICMP-ALL object visible in API and GUI returned by TF as "ICMP-entry"? Seems odd. And conversely, "ICMP-entry" is not a valid name that can be found in the GUI

<snip for brevity>
╷
│ Error: expected rule.0.destination_groups.0 to contain a valid IP, Range, CIDR, or Group Path. Got: d214f969-60f7-44c5-acd6-8b436639f606
│
│   with nsxt_policy_security_policy.SHARED-INFRASTRUCTURE,
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 12, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   12:     destination_groups = [data.nsxt_ns_groups.NS-GROUPS.items["default.INTDMZ"]]
│
╵
╷
│ Error: Invalid policy path: dde6a2fc-fe52-472e-8b10-44573dcbc1bb
│
│   with nsxt_policy_security_policy.SHARED-INFRASTRUCTURE,
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 14, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   14:     services           = [data.nsxt_ns_services.NS-SERVICES.items["ICMP-entry"]]
│
╵
<snip for brevity>

annakhm · 2021-12-03T18:51:23Z

Hi @akhomyakov, unfortunately these data sources are based on old-style MP API, not policy API, so they won't do what you expect. These were added for a specific customer that cannot use policy API due to very specific feature set they are using.
Are you looking to improve performance? Otherwise you can just use a singular data sources nsxt_policy_service, nsxt_policy_group for same purpose?

akhomyakov · 2021-12-06T16:35:02Z

I see. This suggests that these two will never come out of beta. I am looking for a way to import all existing service objects and groups rather than import them one by one. In other words, NSXT comes with a long list of prebuilt services to be used in the security policy like ICMPv4 and so on. Is there a sane way to import them all in one go or it's a one by one type of endeavor only?

annakhm · 2022-04-08T17:59:03Z

For now there are no parallel data sources for policy APIs, but seems like people are asking for it. So these are on the roadmap. For now though prebuilt data sources need to be imported one by one.

github-actions · 2023-04-04T17:40:25Z

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

akhomyakov · 2023-04-04T18:22:14Z

bump

These data sources allows the retrieval of the entire tables from NSX to save multiple calls to the backend. Fixes: vmware#696 Signed-off-by: Kobi Samoray <[email protected]>

salv-orlando · 2024-12-13T07:57:12Z

Moving this issue to 3.9.0 due to low priority and the fact that we are close to 3.8.0 release

annakhm added good first issue new-data-source Data Source labels Apr 8, 2022

github-actions bot added the stale Stale label Apr 4, 2023

github-actions bot removed the stale Stale label Apr 5, 2023

salv-orlando assigned GraysonWu Nov 3, 2023

salv-orlando changed the title ~~nsxt_ns_groups and nsxt_ns_services content~~ Policy Datasources for leveraging built-in services Nov 3, 2023

salv-orlando unassigned GraysonWu Aug 9, 2024

salv-orlando added this to the v3.8.0 milestone Aug 9, 2024

salv-orlando self-assigned this Aug 9, 2024

tenthirtyam added good-first-issue Good First Issue and removed good first issue labels Oct 9, 2024

salv-orlando assigned ksamoray and unassigned salv-orlando Nov 28, 2024

ksamoray linked a pull request Dec 3, 2024 that will close this issue

[WIP] Implement map data sources for policy services and groups #1476

Open

salv-orlando modified the milestones: v3.8.0, v3.9.0 Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Policy Datasources for leveraging built-in services #696

Policy Datasources for leveraging built-in services #696

akhomyakov commented Dec 2, 2021 •

edited

Loading

akhomyakov commented Dec 2, 2021 •

edited

Loading

annakhm commented Dec 3, 2021

akhomyakov commented Dec 6, 2021

annakhm commented Apr 8, 2022

github-actions bot commented Apr 4, 2023

akhomyakov commented Apr 4, 2023

salv-orlando commented Dec 13, 2024

Policy Datasources for leveraging built-in services #696

Policy Datasources for leveraging built-in services #696

Comments

akhomyakov commented Dec 2, 2021 • edited Loading

akhomyakov commented Dec 2, 2021 • edited Loading

annakhm commented Dec 3, 2021

akhomyakov commented Dec 6, 2021

annakhm commented Apr 8, 2022

github-actions bot commented Apr 4, 2023

akhomyakov commented Apr 4, 2023

salv-orlando commented Dec 13, 2024

akhomyakov commented Dec 2, 2021 •

edited

Loading

akhomyakov commented Dec 2, 2021 •

edited

Loading