Azure-IoT-Edge Playground

For Azure IoT Edge - provisioning tests

Architecture:

Each Edge contains valid provisioning Keys, Certificates and a the TPM Edges also a fresh build of Azure IoT SDK.

Tasks / Plans

• ✓ Azure Deployment

• ✓ Vagarant Setup

• ✓ create shared keys

• ✓ register with shared key

• ✓ create certificates

• ✓ register with certificate

• ✓ install TPM dependencies

• ❏ register with TPM (not possible because of interactive tools)

• ❏ Create Metadata Service

• ❏ Install and Configure Squid and proxy settings for Gateway

• ❏ Vagrant optimization by Provision 1 VM and clone then

• ❏ Vagrant using KVM instead of Virtual Box

Known Issues

1 - DPS registration failed because the DPS operation is not assigned

Sometimes the provisioning don’t work with the following error:

[WARN] - Could not provision device
[WARN] - 	caused by: DPS registration failed because the DPS operation is not assigned
[ERR!] - The daemon could not start up successfully: Could not initialize DPS provisioning client
[ERR!] - 	caused by: Could not restore previous provisioning result
[ERR!] - 	caused by: No such file or directory (os error 2)

Then you have to unlock the IoTHub from DPS and register at again via the Azure Portal

2 - Authentication failed

Mostly then the root certificate needs to be validated with a dedicated certificate

Prerequisites

To install this Lab, you need to fulfill some requirements.

This Repo was developed to be used on a Ubuntu 18.04 LTS It should work also on other linux systems, but because of Paths i guess not under Windows.

Requirements

• VirtualBox 5 (6 is not well supported by vagrant yet)

• Vagrant

• Azure CLI https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest

• Terraform https://learn.hashicorp.com/terraform/azure/install_az

• Azure CLI - IOT Extensions (for DPS Enrollment groups)

az extension add --name azure-iot

Authentication

Use 'az login' to connect to your Azure environment or use https://shell.azure.com which already contains az and terraform.

Create a Service Principal

So we have first to cerate an Service Principle Account. If something is unclear follow https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli?view=azure-cli-latest=

az ad sp create-for-rbac --name TerraformDeploy - o json

Note the output PWD and AppID

Now make sure that these account has the correct rights (contributor)

az role assignment list --assignee APP_ID
az role assignment create --assignee APP_ID --role Contributor

Using the Credentials

Export the credentials into an environment variable or add them in the header of our terraform - script (not suggested!).

credentials.sh

export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

a little Helper

There is a maintained file which can be used for dev environments. It’s called "credentials-template.sh". Usually i create a copy of "credentials-template.sh" and name it "credentials-apply.sh". This file is ignored by git and will never checked in. Then i can add my credentials into "credentials-apply.sh" and execute source credentials-apply.sh once, before doing anything with terraform. That prevents on development systems that the credentials are shown in the bash history and that they are mixed up for different projects because of global environment variables.

Create your Playground

The Playground is based on an Azure Cloud Part, the IoT central services and a few Edge Nodes which are running locally on your Virtual Box. The local VMs consuming 3 GB of RAM together.

In addition there are certificates required. We will generate the Certs based on the scripts given from Microsoft for testing purposes. Be aware that this way is not usable for production deployments. See https://github.com/Azure/iotedge/tree/master/tools/CACertificates

Local Edge 1+2 will be connected to the Azure IoT Hub via generated shared Key and Key derivation. Edge 3+4 will use the certificates. If you change the Names of Edge 3+4 you need to change also teh certs.

Be aware that these Certs are valid only for 30 days.

You can do an "all in one" start by executing run-playground.sh

Starting on Azure

source credentials-apply.sh
terraform init  # download all required modules
terraform plan  # check credentials and configuration
terraform apply --parallelism=1 # install or upgrade solution
# or - terrafrom apply --auto-approve --parallelism=1

Starting on Virtualbox

Note: The first start can take up to 40 Minutes because multiple GB of Data and installation sources will be downloaded from the Internet.

vagrant up

Notes

A few notes to the project

DPS symmetric key provisioning configuration

provisioning:
  source: "dps"
  global_endpoint: "https://global.azure-devices-provisioning.net"
  scope_id: "0dsdf001dsdf0dfB"
  attestation:
    method: "symmetric_key"
    registration_id: "iot-edge-key1"
    #must be a derived key including registration id
    symmetric_key: "OcNkknqz7z06jkJfdsfsdDp6aFOfdfdskdspbtfOKmJw="

DPS X509 certificate provisioning

provisioning:
  source: "dps"
  global_endpoint: "https://global.azure-devices-provisioning.net"
  scope_id: "<SCOPE_ID>"
  attestation:
    method: "x509"
    #  registration_id: "<OPTIONAL REGISTRATION ID. LEAVE COMMENTED OUT TO REGISTER WITH CN OF identity_cert>"
    identity_cert: "file:///etc/iotedge/iot-edge-device-identity-iot-edge-cert1-full-chain.cert.pem"
    identity_pk: "file:///etc/iotedge/iot-edge-device-identity-iot-edge-cert1.key.pem"

Device verification certificate

 ./certGen.sh create_verification_certificate B730D538fgdgdfgdfgfd8F5A90D10D0EC5F403B92CF1820

Output: iot-device-verification-code-full-chain.cert.pem Needs to be uploaded to Azure.

Things to think about

Maybe there is an option to use Environment Varbiables instead of modify the config file ..

Sep 02 11:56:51 iot-edge-cert1 iotedged[10637]: 2020-09-02T11:56:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:hsm_provision_edge_id_certificate:1790) Path set in env variable IOTEDGE_DEVICE_IDENTITY_PK is invalid or cannot be accessed: '/etc/iotedge/iot-edge-device-identity-iot-edge-cert1.key.pem'
Sep 02 11:56:51 iot-edge-cert1 iotedged[10637]: 2020-09-02T11:56:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:hsm_provision_edge_id_certificate:1807) To setup the Edge device certificates, set env variables with valid values:
Sep 02 11:56:51 iot-edge-cert1 iotedged[10637]:   IOTEDGE_DEVICE_IDENTITY_CERT
Sep 02 11:56:51 iot-edge-cert1 iotedged[10637]:   IOTEDGE_DEVICE_IDENTITY_PK
Sep 02 11:56:51 iot-edge-cert1 iotedged[10637]: 2020-09-02T11:56:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_crypto.c:hsm_client_crypto_init:47) Could not create store. Error code 2025