Generate "expand" and "flatten" functions for associated external types

[bendbennett]

Background

Given that Go uses static typing, when encountering “objects” with Go code it is very common that the API SDK (or other external code) implementation will use custom Go structure types for those objects. For example, a Terraform schema may need to describe an API with a list nested attribute (types.List of types.Object), whereas the API SDK may implement an array/slice of a custom Go type. The provider developer is responsible for converting to and from the Terraform SDK types to the external types to properly generate API requests (Create, Update, Delete) or handle API responses (Read). This data handling logic can be cumbersome and it is almost exclusively repetitive depending on the API. Previously with terraform-plugin-sdk, this type of provider logic was conventionally put into what developers called “expand” (Terraform to API) and “flatten” (API to Terraform) functions.

Proposal

To ease developer burden, it is proposed that the associated_external_type, which can be optionally specified, be used in the generation of "expand" and "flatten" functions.

The following intermediate representation illustrates the usage of associated_external_type for a single nested block:

{
  "resources": [
    {
      "name": "aws_imagebuilder_image",
      "schema": {
        "blocks": [
          {
            "name": "image_tests_configuration",
            "single_nested": {
              "associated_external_type": {
                "imports": [
                  {
                    "path": "github.com/aws/aws-sdk-go/service/imagebuilder"
                  }
                ],
                "type": "*imagebuilder.ImageTestsConfiguration",
                "mapping": {
                  "timeout_minutes": {
                    "name": "MinutesTimeout",
                    "type": "string"
                  }
                }
              },
              "attributes": [
                {
                  "name": "image_tests_enabled",
                  "bool": {
                    "computed_optional_required": "optional"
                  }
                },
                {
                  "name": "timeout_minutes",
                  "int64": {
                    "computed_optional_required": "computed_optional"
                  }
                }
              ]
            }
          }
        ]
      }
    }
  ],
}

In addition to the schema, models and model helper functions that are generated from the IR, the code generation would also create the following "expand" and "flatten" functions:

func (m ImageTestsConfigurationModel) ToImageTestsConfiguration(ctx context.Context, tfObject types.Object) (*imagebuilder.ImageTestsConfiguration, diag.Diagnostics) {
  var diags diag.Diagnostics

  if tfObject.IsNull() || tfObject.IsUnknown() {
    return nil, diags
  }

  var tfModel ImageTestsConfigurationModel

  diags.Append(tfObject.As(ctx, &tfModel, basetypes.ObjectAsOptions{})...)

  if diags.HasError() {
    return nil, diags
  }

  apiObject := &imagebuilder.ImageTestsConfiguration{
    ImageTestsEnabled: 	tfModel.ImageTestsEnabled.ValueBoolPointer(),
    TimeoutMinutes: 	tfModel.TimeoutMinutes.ValueInt64Pointer(),
  }

  return apiObject, diags
}

func (m ImageTestsConfigurationModel) FromImageTestsConfiguration(ctx context.Context, apiObject *imagebuilder.ImageTestsConfiguration) (types.Object, diag.Diagnostics) {
  var diags diag.Diagnostics
  var tfModel ImageTestsConfigurationModel

  if apiObject == nil {
    return m.objectNull(ctx), diags
  }

  tfModel.ImageTestsEnabled = types.BoolPointerValue(apiObject.ImageTestsEnabled)
  tfModel.TimeoutMinutes = types.Int64PointerValue(apiObject.TimeoutMinutes)

  return m.objectValueFrom(ctx)
}

"Implicit" Mapping

The only fields that are currently defined within the schema for associated_external_type are type and import. Consequently, assumptions have to be made about how fields within an intermediate representation (IR) that have an associated_external_type defined should be handled when "expand" and "flatten" functions are generated.

Primitives

Primitives are defined as the following types:

Schema Attribute Type	Model Type
schema.BoolAttribute	types.Bool, basetypes.BoolValue
schema.Float64Attribute	types.Float64, basetypes.Float64Value
schema.Int64Attribute	types.Int64, basetypes.Int64Value
schema.NumberAttribute	types.Number, basetypes.NumberValue
schema.StringAttribute	types.String, basetypes.StringValue

If the IR contains an associated_external_type for a primitive, the "expand" and "flatten" functions illustrated below will be generated.

Note that this assumes that the associated_external_type.type (e.g., *apisdk.BoolType) is something like the following:

type BoolType *bool

expand

func To<NameOfField>(ctx context.Context, tfType types.<Bool|Float64|Int64|Number|String>) (*<associated_external_type.type>, diag.Diagnostics) {
	var diags diag.Diagnostics

	if tfType.IsNull() || tfType.IsUnknown() {
		return nil, diags
	}

	var m <associated_external_type.type>

	m = tfType.Value<BoolPointer|Float64Pointer|Int64Pointer|BigFloat|StringPointer>()
	
	return &m, diags
}

flatten

func From<NameOfField>(ctx context.Context, apiObject *<associated_external_type.type>) (types.<Bool|Float64|Int64|Number|String>, diag.Diagnostics) {
	var diags diag.Diagnostics

	if apiObject == nil {
		return types.<Bool|Float64|Int64|Number|String>Null(), diags
	}

	m := types.<BoolPointerValue|Float64PointerValue|Int64PointerValue|NumberValue|StringPointerValue>(*apiObject)

	return m, diags
}

The following assumptions have been made when implicitly mapping primitives:

• The associated_external_type.type is a type which can be assigned a pointer of the field value type (i.e., types.Bool | basetypes.BoolValue <=> *bool etc).
• The type specified for the associated_external_type.type in the IR will be a pointer.
  • The usage of a pointer or reference in the "To<...>" and "From<...>" functions is used to emphasise where each is being used.

Collections

Collections are defined as the following types:

Schema Attribute Type	Model Type
schema.ListAttribute	types.List, basetypes.ListValue
schema.MapAttribute	types.Map, basetypes.MapValue
schema.SetAttribute	types.Set, basetypes.SetValue

If the IR contains an associated_external_type for a collection, the "expand" and "flatten" functions illustrated below will be generated.

Note that this assumes that the associated_external_type.type (e.g., *apisdk.BoolSliceType) is something like the following:

type BoolSliceType []*bool

expand

func To<NameOfField>(ctx context.Context, tfType types.<List|Map|Set>) (*<associated_external_type.type>, diag.Diagnostics) {
	var diags diag.Diagnostics

	if tfType.IsNull() || tfType.IsUnknown() {
		return nil, diags
	}

	var m <associated_external_type.type>

	// ElementsAs() converts correctly from types.ListType{ElemType: types.BoolType}
	// to BoolSliceType (i.e., []*bool), for instance.
	diags.Append(tfType.ElementsAs(ctx, &m, false)...)

	if diags.HasError() {
		return nil, diags
	}

	return &m, diags
}

flatten

func From<NameOfField>(ctx context.Context, apiObject *<associated_external_type.type>) (types.<List|Map|Set>, diag.Diagnostics) {
	var diags diag.Diagnostics

	if apiObject == nil {
		// The attr.Type required to call <List|Map|Set>Null() can be obtained from
		// schema.<List|Map|Set>Attribute ElementType.
		return types.<List|Map|Set>Null(types.<?>), diags
	}

	m, d := types.<List|Map|Set>ValueFrom(ctx, types.<?>, apiObject)

	diags.Append(d...)

	if diags.HasError() {
		return types.<List|Map|Set>Null(types.<?>), diags
	}

	return m, diags
}

The following assumptions have been made when implicitly mapping collections:

• The associated_external_type.type is a type which can be passed to ElementsAs(...). For example, if the schema defines the list attribute as schema.ListAttribute{ElementType: types.BoolType} then associated_external_type.type would need to be a type that can be assigned []*bool.

Object

Object is defined as the following type:

Schema Attribute Type	Model Type
schema.ObjectAttribute	types.Object, basetypes.ObjectValue

If the IR contains an associated_external_type for an object, the "expand" and "flatten" functions listed below will be generated.

The "expand" and "flatten" functions are generated on the basis of schema.ObjectAttribute and ApiObject having the following form:

schema.ObjectAttribute{
	AttributeTypes: map[string]attr.Type{
		"bool_field":    types.BoolType,
		"float64_field": types.Float64Type,
		"list_field": types.ListType{
			ElemType: types.StringType,
		},
	},
	/* ... */
},

type ApiObject struct {
	BoolField  *bool
	Int64Field *int64
	ListField  []*string
}

expand

func To<NameOfField>(ctx context.Context, tfObject types.Object) (*<associated_external_type.type>, diag.Diagnostics) {
	var diags diag.Diagnostics

	if tfObject.IsNull() || tfObject.IsUnknown() {
		return nil, diags
	}

	// Iterate over map[string]attr.Type from schema and generate
	// struct as there won't be a corresponding model as they're
	// only generated for list, map, set, single nested attributes/blocks.
	type objStruct struct {
		BoolField    types.Bool    `tfsdk:"bool_field"`
		Float64Field types.Float64 `tfsdk:"float64_field"`
		ListField    types.List    `tfsdk:"list_field"`
	}

	var objS objStruct

	diags.Append(tfObject.As(ctx, &objS, basetypes.ObjectAsOptions{})...)

	if diags.HasError() {
		return nil, diags
	}

	// Generate on basis of corresponding field in AttributeTypes from schema.
	// Need to detect if handling any non-primitive (i.e., list, map, object, or set).
	var listField []*string

	d := objS.ListField.ElementsAs(ctx, &listField, false)

	diags.Append(d...)

	if diags.HasError() {
		return nil, diags
	}

	apiObject := &ApiObject{
		BoolField:    objS.BoolField.ValueBoolPointer(),
		Float64Field: objS.Float64Field.ValueFloat64Pointer(),
		ListField:    listField,
	}

	return apiObject, diags
}

flatten

func From<NameOfField>(ctx context.Context, apiObject *ApiObject) (types.Object, diag.Diagnostics) {
	var diags diag.Diagnostics

	attrTypes := map[string]attr.Type{
		"BoolField":    types.BoolType,
		"Float64Field": types.Float64Type,
		"ListField": types.ListType{
			ElemType: types.StringType,
		},
	}

	if apiObject == nil {
		return types.ObjectNull(attrTypes), diags
	}

	// Generate on basis of corresponding field in AttributeTypes from schema.
	// Need to detect if handling any non-primitive (i.e., list, map, object, or set).
	l, d := types.ListValueFrom(ctx, types.StringType, apiObject.ListField)

	diags.Append(d...)

	if diags.HasError() {
		return types.ObjectNull(attrTypes), diags
	}

	o, d := types.ObjectValue(
		attrTypes,
		map[string]attr.Value{
			"BoolField":    types.BoolPointerValue(apiObject.BoolField),
			"Float64Field": types.Float64PointerValue(apiObject.Float64Field),
			"ListField":    l,
		},
	)

	diags.Append(d...)

	return o, d
}

The following assumptions have been made when implicitly mapping collections:

• The associated_external_type.type contains a one-to-one mapping of field names and types (e.g., if the schema.ObjectAttribute contains within AttributeTypes a "BoolAttribute": types.BoolType the expectation is that the ApiObject will contain a field called BoolAttribute which holds *bool).

Nested Attributes and Blocks

Nested attributes and blocks are defined as the following types:

Schema Attribute Type	Model Type
schema.ListNestedAttribute	types.List, basetypes.ListValue
schema.ListNestedBlock	types.List, basetypes.ListValue
schema.MapNestedAttribute	types.Map, basetypes.MapValue
schema.SetNestedAttribute	types.Set, basetypes.SetValue
schema.SetNestedBlock	types.Set, basetypes.SetValue
schema.SingleNestedAttribute	types.Object, basetypes.ObjectValue
schema.SingleNestedBlock	types.Object, basetypes.ObjectValue

Single Nested Attribute without nested associated_external_type

If the IR contains an associated_external_type defined for a single nested attribute with no associated_external_type(s) defined on any of the attributes within the single nested attribute, then the "expand" and "flatten" functions listed below will be generated.

The "expand" and "flatten" functions are generated on the basis of schema.SingleNestedAttribute, and ApiSingleNestedAttribute having the following form:

schema.SingleNestedAttribute{
	Attributes: map[string]schema.Attribute{
		"bool_attribute": schema.BoolAttribute{
			Optional: true,
		},
		"int64_attribute": schema.Int64Attribute{
			Optional: true,
		},
	},
	/* ... */
},

type ApiSingleNestedAttribute struct {
	BoolAttribute  *bool
	Int64Attribute *int64
}

expand

func ToSingleNestedAttribute(ctx context.Context, tfObject types.Object) (*ApiSingleNestedAttribute, diag.Diagnostics) {
	var diags diag.Diagnostics

	if tfObject.IsNull() || tfObject.IsUnknown() {
		return nil, diags
	}

	var sna SingleNestedAttributeModel

	diags.Append(tfObject.As(ctx, &sna, basetypes.ObjectAsOptions{})...)

	if diags.HasError() {
		return nil, diags
	}

	apiObject := &ApiSingleNestedAttribute{
		BoolAttribute:  sna.BoolAttribute.ValueBoolPointer(),
		Int64Attribute: sna.Int64Attribute.ValueInt64Pointer(),
	}

	return apiObject, diags
}

flatten

func FromSingleNestedAttribute(ctx context.Context, apiObject *ApiSingleNestedAttribute) (types.Object, diag.Diagnostics) {
	var diags diag.Diagnostics
	var sna SingleNestedAttributeModel

	if apiObject == nil {
		// ObjectNull() is pre-generated for nested attributes.
		return sna.ObjectNull(ctx), diags
	}

	sna.BoolAttribute = types.BoolPointerValue(apiObject.BoolAttribute)
	sna.Int64Attribute = types.Int64PointerValue(apiObject.Int64Attribute)

	// ObjectValueFrom() is pre-generated for nested attributes.
	return sna.ObjectValueFrom(ctx, sna)
}

Single Nested Attribute with nested associated_external_type(s)

If the IR contains an associated_external_type defined for a single nested attribute with associated_external_type(s) defined on the attributes within the single nested attribute, then the "expand" and "flatten" functions listed below will be generated.

The "expand" and "flatten" functions are generated on the basis of schema.SingleNestedAttribute, and ApiSingleNestedAttribute having the following form:

schema.SingleNestedAttribute{
	Attributes: map[string]schema.Attribute{
		"bool_attribute": schema.BoolAttribute{
			Optional: true,
		},
		"int64_attribute": schema.Int64Attribute{
			Optional: true,
		},
	},
	/* ... */
},

type ApiSingleNestedAttribute struct {
	BoolAttribute  *ApiBoolAttribute
	Int64Attribute *int64
}

type ApiBoolAttribute *bool

expand

func ToSingleNestedAttribute(ctx context.Context, tfObject types.Object) (*ApiSingleNestedAttribute, diag.Diagnostics) {
	var diags diag.Diagnostics

	if tfObject.IsNull() || tfObject.IsUnknown() {
		return nil, diags
	}

	var sna SingleNestedAttributeModel

	diags.Append(tfObject.As(ctx, &sna, basetypes.ObjectAsOptions{})...)

	if diags.HasError() {
		return nil, diags
	}

	toBoolAttribute, d := ToBoolAttribute(ctx, sna.BoolAttribute)

	diags.Append(d...)

	if diags.HasError() {
		return nil, diags
	}

	apiObject := &ApiSingleNestedAttribute{
		BoolAttribute:  toBoolAttribute,
		Int64Attribute: sna.Int64Attribute.ValueInt64Pointer(),
	}

	return apiObject, diags
}

func ToBoolAttribute(ctx context.Context, tfType types.Bool) (*ApiBoolAttribute, diag.Diagnostics) {
	var diags diag.Diagnostics

	if tfType.IsNull() || tfType.IsUnknown() {
		return nil, diags
	}

	var m ApiBoolAttribute

	m = tfType.ValueBoolPointer()

	return &m, diags
}

flatten

func FromSingleNestedAttribute(ctx context.Context, apiObject *ApiSingleNestedAttribute) (types.Object, diag.Diagnostics) {
	var diags diag.Diagnostics
	var sna SingleNestedAttributeModel

	if apiObject == nil {
		return sna.ObjectNull(ctx), diags
	}

	fromBoolAttribute, d := FromBoolAttribute(ctx, apiObject.BoolAttribute)

	diags.Append(d...)

	if diags.HasError() {
		return sna.ObjectNull(ctx), diags
	}

	sna.BoolAttribute = fromBoolAttribute
	sna.Int64Attribute = types.Int64PointerValue(apiObject.Int64Attribute)

	return sna.ObjectValueFrom(ctx, sna)
}

func FromBoolAttribute(ctx context.Context, apiObject *ApiBoolAttribute) (types.Bool, diag.Diagnostics) {
	var diags diag.Diagnostics

	if apiObject == nil {
		return types.BoolNull(), diags
	}

	b := types.BoolPointerValue(*apiObject)

	return b, diags
}

Further Considerations

• Currently the IR Schema only permits the usage of associated_external_type on data source, provider and resource single nested attributes. This will need to be extended to all attribute and block types.
  • Add associated_external_type to all attributes and blocks
• The associated_external_type as it is currently defined allows specifying type and import. Consequently some assumptions will need to be made about how the fields within the model struct are mapped to/from the fields in the "object" that is used for interaction with the API. An initial assumption would be that there is a direct one-to-one mapping unless/until a "mapping" field is defined for associated_external_type which allows for specifying how this translation from model to API object should happen.
  • Consider adding "mapping" to associated_external_type

[sachinsaxena-okta]

Currently, terraform-plugin-codegen-openapi doesn't provide any field to map schema properties to associated_external_type. Is this something going to be supported in future as part of this issue ?

[austinvalle]

Hey there @sachinsaxena-okta 👋🏻,

The associated_external_type design is still a WIP, so finishing that work would be a pre-requisite for adding any support.

As for specifically the type of support possible in terraform-plugin-codegen-openapi. All of the associated_external_type references are external Go modules and could vary in how the structure of the Go type matches an OpenAPI request/response body. If the design stays as is, one option we have would be to support some type of annotation of the OpenAPI spec to denote an associated_external_type for each JSON schema.

That's not a particularly great option, so we'll want to have a more concrete design before we start expanding support across other projects, like terraform-plugin-codegen-openapi

[sachinsaxena-okta]

Thanks @austinvalle. We generated sdk for api client from openapi spec and thinking to map sdk types using associated_external_type. This will make very easy to call apis using sdk. I will wait for this work to be complete to see how it will be useful.