Implementation project of 4.0 workshops at IFRS Campus Caxias do Sul.
~Team Iniciativa Arduino IoT
Leia isso em Português
Lea esto en Español
The objective of this project is to simulate an IoT Turnstile connected to the cloud. This would broaden fleet management capabilities as well as allowing for the development of machine learning and data analytics over the public transportating usage and scheduling.
This project uses an Arduino Uno r3 paired with an esp01 WiFi board (based on the esp8266 chip), GPS, and two buttons. The transmission protocol used is Mosquito (MQTT) and our backend is made on Google Cloud Platform using the services IoT Core, and PUB/SUB. For our data retention, we use a Google Cloud Functions that is triggered whenever a new publish PUB/SUB event happens, and sends the received data to Google Firebase.
Disclaimer
This project may incur charges and the authors are not responsible. Do it under your full awareness and read the available material carefully to avoid suprises. Have a Good learning experience.
Overview
The figure below represents our data flux, with the services used.
Materials List
- Arduino Uno R3
- ESP01 8266
- GPS Neo 6M
- LCD 16x2
- Adapter for ESP01 w/ voltage regulator and logic level shifter
- Potentiometer 10kOhm
- Adapter usb-serial ch340 for esp01
- Jumper Cables
- 2 push buttons
- 5v power supply
- Switch (optional)
- Account on Google Cloud Platform
- Node package manager
- Microsoft VSCode
- Gcloud CLI
Table of Contents
Google offers many services in various ways. For more billing information on their free tier, click on this link.
The configuration can be done utilizing the user interface, called console, or a command line interface as a free service called cloud shell. There is also a script at the end of this section, made for those who know how to execute it and make everything automaticaly.
Firstly we need to configure names for our stuff. Copy and paste the variables below on a notepad and fill it with what you find suitable. If you are going to use cloud shell, copy the filled version onto it but don't delete your file, as we are using it troughtout all the development.
export PROJECT_ID= #fill with your project name
export REGION= #choose a region
export TOPIC_ID= #name of your pubsub topic
export SUBSCRIPTION= #name of your pubsub subscription
export REGISTRY= #name of your iot core regitry
export DEVICE_ID= #iot device idWith everything prepared we can go to the next step. Create a project utilizing your PROJECT_ID and be sure to check if billing is activated, otherwise you will be prevente from proceeding when deploying any kind of service. To do that go to google cloud console and search for "billing" in the search bar.
To activate the necessary services, go to the menu at your top-left and find "APIs". Click on "add services and APIs", and then search and activate the following IoT core, Pub/Sub, and Cloud Functions. If you are using shell, paste the following:
gcloud services enable cloudiot.googleapis.com pubsub.googleapis.com cloudfunctions.googleapis.comBefore setting up the services, you need to create a service account to give IoT Core the permissions to publish into a Pub/Sub Topic. Search for "IAM & Admin" and when inside IAM, press "Add". On the first field, insert the following email [email protected] and on the field "Function" search for Cloud PUB/SUB Publish and then hit save. If you are using shell, this will do it:
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:[email protected] \
--role=roles/pubsub.publisherTo proceed, one will need an eliptic curve key pair with 256bits to authenticate the device. Now, it's easier to use cloud shell for that:
openssl ecparam -genkey -name prime256v1 -noout -out ec_private.pem
openssl ec -in ec_private.pem -pubout -out ec_public.pemThis will create two keys and save them in the cloud editor. Open the editor and find your keys. Open it, copy and paste into a temporary file for later. If you are using the shell you don't need to do this.
Important
Don't share your keys with anyone. This represents a security risk to the integrity of your project.
Utilizando os menus
To create a PUB/SUB topic, go to the GCP searchbox look for "Pub/Sub". You can also scroll until you find the service on the left-hand menu. Tap the button "Create topic". In the new window, fill in the "Topic ID" box with whatever you defined as TOPIC_ID and hit create. It might take a short while but a message will say when your topic has been created. Now go to "Subscriptions" and press "Create Subscription". In the new opened window, type in the name that you chose as SUBSCRITION and select the topic that we just created. Accept all default values and press Create.
Utilizando o Shell
If you want to use the google shell, paste the following:
gcloud iot registries create $REGISTRY \
--region=$REGION \
--event-notification-config=topic=$TOPIC_ID \
--enable-mqtt-config --enable-http-configUsing the GUI
To create a IoT Core, use the searchbox to look for "IoT Core" or browse the menu until you find it. When you opend it, your screen will probably be blank. Press "create registry". On the new page, type the registry name into REGISTRY, select the REGION (hint: try to leave all of them close to each other). To end the configuration, select the Cloud Pub/Sub topic that you wish to publish the device messages. After creating the registry, go in devices and click on "Create new Device". Add the DEVICE_ID on the name field. Now, open the advanced options and roll until you find "Authentication". Select "type manually" and for the format select ES256. Below that, copy and paste everthing including the ---BEGIN PUBLIC KEY--- and ---END PUBLIC KEY--- at the end.
Using Shell
To create an IoT Core using shell:
gcloud iot registries create $REGISTRY \
--region=$REGION \
--event-notification-config=topic=$TOPIC_ID \
--enable-mqtt-config --enable-http-configTo create a device:
gcloud iot devices create $DEVICE_ID \
--region=$REGION \
--registry=$REGISTRY \
--public-key="path=./ec_public.pem,type=es256"We made a shell script to make it easy to deploy all of the above infrastructure. Don't forget to modify it to suit your needs. Download it here, and open cloud shell. Now open editor and hit "upload file". To run it:
./GCPScript.shIn this system, the Arduino Uno bpoard is responsible for collecting telemetry data from the sensors and inputs. Whereas, the ESP01 will be responsible for sending the telemetry throught MQTT, connecting to the Wi-Fi Access Point and to GCP.
There are two approaches to work with all the sensors and inputs. We can build our own libraries to make them work or utilize the ones available in the Arduino community. In this project we opt for the latter, as they can certainly do a solid job and reduce our workload.
To program the ESP01 we need to make the IDE recognize it. On the field "manage additional boards", input the following link:
https://arduino.esp8266.com/stable/package_esp8266com_index.json
Hit ok and go to tools > board > Boards Manager. In the new window, search for esp8266 community. You will need to do this every time to flash the esp8266.
To add the libraries go to menu > add libraries > manage libraries. Install the following:
- LiquidCrystal by Arduino
- WifiESP by Bruno Portaluri
- TinyGPS by Mikal Hart
- SerialTransfer by PowerBroker2
- MQTT by Joel Gaewhiler
- Google Cloud IoT Core JWT by Guss Class
- DailyStruggleButton by cygig
During the installation, a pop-up message may ask to install dependent libraries, allow it.
In this project the arduino Uno works agreggating sensorial information. If a button was pressed, the Uno identifies which one was pressed and insert the information into a struct that has the already polled GPS information, and sends it to the ESP01 throught Software Serial. The block Diagram will likely be the following:
The ESP01 program is a little simpler. When the Uno sends a message, the ESP01 creates a .JSON and send it to PUB/SUB. The message format is:
{
"latitude": 0.0000,
"longitude": 0.0000,
"HDOP": 0.0000,
"Altitude": 0.0000,
"Course": 0.0000,
"Speed": 0.0000,
"NumSat": 0,
"Entrance": 0,
"Exit": 0,
}The block diagram can be found below and the code can be found here.
Almost all the Arduino pins are used in this project. The wiring will look like the following:
At the end your project will look like the following:
With the hardware up and running, now we can worry about the backend. So far, we only deployed the infrastructure. There are many ways we can test the IoT Core, but this requires the installation of Eclipse Mosquitto Client and it's out of the scope of this project. For more information this project by @nilhcem, on "connect to http/mqtt bridge" can help you.
Turn everything on. If everything is correctly setup and coded, when you press a button the Uno will display the count, and the ESP01's blue light will blink upon received serial data. To verify Pub/Sub, open the Google Cloud Console and navigate to the Pub/Sub service. Click on the topic ID and search for the $SUBSCRIPTION. Click on it and, to display the messages select "Enable ack messages" and then hit the pull button. Your screen will look like this:
To do the same, but use the cli Google Cloud CLI or the shell:
gcloud pubsub subscriptions pull --auto-ack $SUBSCRIPTION --limit=1If you arrived here, congrats!! Now we can send information anywhere. But, for this project we are going to send it to Firebase as we aim a simpler integration with an App on the future. More info about Firebase. Login into it with the same GCP account. Add a new project and select the name as you created earlier on PROJECT_ID. Confirm the we are going to use blaze, and confirm again. When it asks about Google Analytics, confirm that it is activated. For the Google Analytics, select the standard, hit confirm, and hit create Firebase.
Now we need to create a Firestore database e enable cloud functions. On the left hand menu, select Firestore Database and hit create database. Select the test mode, click next, chose the region closest to your previosly defined REGION.
To build your cloud function we need node package manager, microsoft VSCode and gcloud CLI. Install all of them and then in terminal use the following command:
npm install -g firebase-toolsNow create a new directory and init the firebase funcations:
mkdir myProject
cd myProject
firebase init functionsby default Firebase only allows for js functions... we can use python through GCP Cloud functions
Use the arrow to choose "Use an existing project", then select the Google Cloud project name, and finally the TypeScript language. When it asks about ESLint type "n". When it asks "Do you want to install dependencies?" type "y" and use the following command to open it on vsCode:
. code
Go to the src folder and open the file index.ts. Now you can copy and paste from this link or download and replace the file. Feel free to modify it to ensure the best fit for you. Now use the following commands:
gcloud auth login
# Web browser login
gcloud projects list
# show available projects
gcloud config set project $PROJECT_ID
# set the default working projectNow utilize the following command to deploy the function:
firebase deploy --only functionsOur function can also be written in python. To start, find that you have Cloud Functions in your terminal. To do so, paste the following:
mkdir $PROJECT_ID
# project name
cd $PROJECT_ID
# go into that folder
. code
# open vscode there
A Cloud Function needs two files. Create a main.py and a requirements.txt. Type the follwing in the requirements:
google-cloud-firestore>=2.1.3
Now we can edit main.py. Copy and paste, or replace the file from this link. Modufy it as you please. Then deploy using:
gcloud functions deploy NOME_DA_FUNCAO --runtime python39 --trigger-topic $TOPIC_IDTo verify that firabse is receiving our communications, please do the following:
gcloud pubsub topics publish $TOPIC_ID --message='Catracat02-connected' --attribute='deviceId=Catraca02,subFolder=events'
# or
gcloud pubsub topics publish $TOPIC_ID --message='{"Latitude":-29.19,"Longitude":-51.24,"HDOP":159,"Altitude":724.30,"Course":0.00,"Speed":0.00,"NumSat":6,"Date":2012479491,"Time":131006486,"fixAge":0,"Entrance":1,"Exit":0}' --attribute='deviceId=Catraca02'Errors and events can be found at heatlh and logs.
Here you can find a Streamlit dashboard.