Consider Unit Testable Schema Validation

[bflad]

terraform-plugin-go version

v0.5.0

Use cases

Terraform Plugin SDK has long included the (*helper/schema.Provider).InternalValidate() method, which:

... should be called in a unit test for any provider to verify before release that a provider is properly configured for use with this library.

Terraform CLI version 1.1.0 and later includes some additional schema validation upfront, instead of potentially erroring out later in workflows: hashicorp/terraform#28124

While providers implementing this library are generally encouraged to use the available acceptance testing framework (currently in Terraform Plugin SDK helper/resource), that type of testing is not guaranteed to be used, nor is it guaranteed to trigger some of the Terraform CLI-based validation.

Having a schema validation functionality within terraform-plugin-go should allow provider developers to receive feedback on schema issues more quickly and potentially without the extra burden of acceptance testing (which itself may require credentials, etc). Theoretically, downstream SDKs can also benefit by being able to call into this functionality.

Attempted solutions

Manually implementing schema validation unit tests.

Acceptance testing using Terraform Plugin SDK helper/resource.

Proposals

Schema Method

The tfprotov5.Schema and tfprotov6.Schema types can implement a new method, e.g.

func (s Schema) Validate(ctx context.Context) error { /* ... */ }

Which calls into further Validate methods on SchemaBlock and SchemaAttribute.

terraform-plugin-go providers could then implement a unit test for each schema based on this method:

func TestExampleSchema(t *testing.T) {
  t.Parallel()

  err := exampleSchema.Validate(context.Background())

  if err != nil {
    t.Error("unable to validate schema: %s", err)
  }
}

Providers could also introduce this check prior to returning GetProviderSchemaResponse. Downstream SDKs could either implement wrapping unit testing or similar checks during GetProviderSchemaResponse handling or tfprotov5/tfprotov6 type conversion

GetProviderSchemaResponse Method

Expanding the above, provide a method on top of the tfprotov5.GetProviderSchemaResponse and tfprotov6.GetProviderSchemaResponse types. e.g.

func (r GetProviderSchemaResponse) Validate(ctx context.Context) error {
  err := r.Provider.Validate(ctx)

  if err != nil {
    return fmt.Errorf("unable to validate provider schema: %w", err)
  }

  for name, schema := range r.DataSourceSchemas {
    err := schema.Validate(ctx)

    if err != nil {
      return fmt.Errorf("unable to validate data source %s schema: %w", name, err)
    }
  }

  for name, schema := range r.ResourceSchemas {
    err := schema.Validate(ctx)

    if err != nil {
      return fmt.Errorf("unable to validate resource %s schema: %w", name, err)
    }
  }
}

This could also be introduced later.

ProviderServer Method

Expanding the above, provide a method on top of the tfprotov5.ProviderServer and tfprotov6.ProviderServer types, e.g.

func (s ProviderServer) Validate(ctx context.Context) error {
  resp, err := s.GetProviderSchema(ctx, &GetProviderSchemaRequest{})

  if err != nil {
    return fmt.Errorf("unable to get schema: %w", err)
  }

  err = resp.Validate(ctx)

  if err != nil {
    return fmt.Errorf("unable to validate schema: %w", err)
  }
}

This, along with additional provider server validations, could also be introduced later. This may be the most intuitive place for provider developers wanting to implement unit testing similar to Terraform Plugin SDK, albeit it would be the most broad. Since the RPC request objects are not customizable in this proposal without modifying the function signature, it may not be future proof.

Additional Context

Terraform CLI 1.1.0 implements the following rules:

• Attribute
  • Must be non-nil
  • Must set Optional, Required or Computed
  • Cannot set both Optional and Required
  • Cannot set both Computed and Required
  • Must set Type or NestedType
  • Cannot set both Type and NestedType
  • NestingSet NestedType may not contain attributes of cty.DynamicPseudoType
• Block
  • Must be non-nil
  • Must not have overlapping attribute and nested block names
  • Must have valid name (^[a-z0-9_]+$)
  • MinItems and MaxItems must both be greater than zero
  • MinItems and MaxItems must match in NestingSingle mode
  • MinItems and MaxItems must be set to either 0 or 1 in NestingSingle mode
  • MinItems and MaxItems cannot be used in NestingGroup mode
  • MinItems must be less than or equal to MaxItems in NestingList/NestingSet mode
  • NestingSet blocks may not contain attributes of cty.DynamicPseudoType
  • MinItems and MaxItems must both be 0 in NestingMap mode

References

• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#Provider.InternalValidate
• terraform: validate providers' schemas during NewContext terraform#28124
• https://github.com/hashicorp/terraform/blob/6530055d188d363bf64ff99b56bb00b350de2cfb/internal/configs/configschema/internal_validate.go
• Consider InternalValidate equivalent terraform-plugin-framework#113