Hex string to binary conversion function

[akingscote]

Current Terraform Version

Terraform v0.13.2

Use-cases

I wish to be able to convert hexadecmial strings to their binary representations.
I can then wrap than in a base64encode function and use the values in my terraform code.

Attempted Solutions

I tried to use regex and replace functions to complete the conversion but i was unsuccessful.

Proposal

My specific example for Azure Key Vault Certificates.
When you create an Azure Key Vault certificate as per https://www.terraform.io/docs/providers/azurerm/r/key_vault_certificate.html#attributes-reference the certificate_data output contains the public certificate in a Hex string format, however I cant see how to convert that into a usable format.

As per this thread, you can use the azurerm_key_vault_secret data source to download the certificate. The problem with this is that it downloads the entire secret, which includes the private key.

In my example, i want to create a key vault certificate in terraform, grab the public key output and feed it into an azurerm_virtual_network_gateway resource. This requires the a base64 representation of the public certificate only and not the private key, so the data source option does not work.

This request is for a function similar to the xxd tool which will allow me to convert a hex string to binary.
Then i can use the public key output of the certificate creation as an input into my virtual network gateway creation.

The equivelant xxd command for converting a hex string is:

xxd -d -p

Id like to be able to use this new function like this:

base64encode(hextobinary(output.azurerm_key_vault_certificate.kv-cert.certificate_data))

Base64 encoding the hex representation directly does not work.

References

hashicorp/terraform-provider-azurerm#2898

[akingscote]

The way im actually getting around this at the moment is via the following:

module "hex-decode" {
  source  = "matti/resource/shell"
  command = "echo ${var.value} | xxd -r -p | base64 -w 0"

  # runs on every apply
  trigger = timestamp()
}

output "value" {
  depends_on = [module.hex-decode.id]
  value      = module.hex-decode.stdout
  sensitive  = true
}

But its a bit nasty and can be circumvented if there was a function in terraform to convert hex to binary.

[apparentlymart]

Hi @akingscote! Thanks for sharing this use-case.

I will admit to not being familiar with the Azure resource you are describing here, but I suspect that we could not address the use-case exactly as you described here because Terraform strings are always sequences of unicode characters, rather than sequences of bytes, and so a hypothetical function to decode from hex would need to also apply a character encoding (we conventionally require UTF-8) to produce a string. Generally when binary data is encoded as base64 or hex that's because it contains octets that are not valid UTF-8, so if that were the case here then your conversion to base64 via a "binary string" would fail.

Typically we've expected providers to export data in a format that is convenient to assign in to arguments of other resources in the same provider. In some cases, for concepts that span across providers, we also have some general idiomatic conventions we expect providers to follow, such as returning and expecting TLS certificates in the PEM format, so that data can flow between providers without conversions.

If this certificate_data attribute is representing a normal TLS certificate then ideally the provider should export it in the conventional PEM format so that it can interop with other Terraform providers that consume certificates, and likewise the network gateway could accept a PEM-encoded certificate for the same reason. If it's not a TLS certificate in the usual sense, then I'd at least expect the provider to be consistent with itself in how this data is formatted, so that you can pass a result from one resource type into an argument of another without any unusual conversions.

In order to solve this in the Terraform language directly I think we'd need a single function that goes directly from hex to base64, without the intermediate step as a "binary string" (such a concept does not exist in Terraform), which is technically possible but kinda awkward and seemingly specific to this AzureRM use-case alone. 🤔

[akingscote]

@apparentlymart Thanks for the detailed response. I see your points, I was really surprised to find that I had to complete this hacky workaround and it didnt just work out the box. I will raise an issue on the AzureRM provider as well.

I created a Pull Request for the hextobinary here and I will see about updating it to a straight base64 encoded output. Hopefully it will (eventually) get approved as its stand alone.

[apparentlymart]

Hi @akingscote,

While I do appreciate you submitting that hextobinary PR, since the only known use-case for it is to address an odd inconsistency in a specific provider I expect we'll prefer to address that inconsistency in-place to avoid the need for the function in the first place, rather than adding another function, because any new end-user features we add are a maintenance burden even if they are relatively easy to implement.

Let's wait to see how hashicorp/terraform-provider-azurerm#8482 works out before deciding how to resolve this issue.

[EricStG]

This is also an issue with the Google provider

Specifically, detect_md5hash on a google_storage_bucket_object expects a base64 encoded hash instead of a hex string

[crw]

Thank you for your continued interest in this issue.

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

Please see the provider-defined functions documentation to learn how to implement functions in your providers. If you are new to provider development, learn how to create a new provider with the Terraform Plugin Framework. If you have any questions, please visit the Terraform Plugin Development category in our official forum.

We hope this feature unblocks future function development and provides more flexibility for the Terraform community. Thank you for your continued support of Terraform!

[apparentlymart]

Checking in on this some time later, I notice that upstream hashicorp/terraform-provider-azurerm#10275 added a new certificate_data_base64 attribute to the key vault certificate resource type.

Based on what we discussed above, I think that should have solved the problem because that new attribute can now be assigned directly to the azurerm_virtual_network_gateway resource type, which was already expecting certificate data in base64 format.

Therefore I'm going to close this issue. Thanks and sorry for the delay in following up!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.