Skip to content

Commit

Permalink
feat: update mirroring contents
Browse files Browse the repository at this point in the history
  • Loading branch information
@fraidev
fraidev committed Jun 3, 2024
1 parent ad004ef commit 5e144bc
Show file tree
Hide file tree
Showing 2 changed files with 97 additions and 74 deletions.
73 changes: 41 additions & 32 deletions docs/tutorials/iot-mirroring-cloud.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Download and install mirroring binary.
Use `curl` to download and install:

```bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
```

Make sure to add `.fluvio/bin` to the `$PATH`as specified in the installation script.
Expand All @@ -87,10 +87,10 @@ Let's provision a new cluster in AWS `eu-central` using the experimental fluvio


```bash
fluvio cloud cluster create --region aws-eu-central-1 --version 0.10.15-dev-2+mirroring-b4f07fc
fluvio cloud cluster create
```

Check cluster status:
Check the result with:


```bash
Expand All @@ -99,6 +99,14 @@ fluvio cluster status

Next, we'll configure the cluster to receive traffic from the edge clusters.

### Register Edge cluster

Let's register the edge cluster `edge1` to inform our home cluster from the cloud to accept connection requests from the remote device:


```bash
fluvio remote register edge1
```

### Create the mirror topic

Expand All @@ -109,15 +117,15 @@ Create a partition assignment file with an array of edge mirros we expect to con

```bash
echo '[
"edge1", "edge2"
"edge1"
]' > assignment_file.json
```

Apply the configuration file to create the topic:


```bash
fluvio topic create edge-topic --mirror-assignment assignment_file.json
fluvio topic create edge-topic --mirror-apply assignment_file.json
```

List partitions to check the assignment:
Expand All @@ -131,34 +139,21 @@ It should display all partitions:

```bash
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 edge1 [] Online 0 B 0 0 0 0 []
edge-topic 1 5001 edge2 [] Online 0 B 0 0 0 0 []
```

We created 2 partitions, but we'll only use one in this tutorial.


### Register Edge cluster

Let's register the edge cluster `edge1` to inform our Cloud cluster to accept connection requests from the remote device:


```bash
fluvio cluster remote-cluster register --type mirror-edge edge1
edge-topic 0 5001 edge1 [] Online 0 B 0 0 0 0 []
```

List remote clusters to check their status:


```bash
fluvio cluster remote-cluster list
fluvio remote list
```

It should show the following:

```bash
RemoteCluster RemoteType Paired Status Last Seen
edge1 mirror-edge - - -
RemoteCluster Status Last Seen
edge1 Waiting -
```


Expand All @@ -174,14 +169,14 @@ mkdir -p ~/local/projects/mirror; cd ~/local/projects/mirror

### Generate metadata for Edge Cluster

Each edge cluster requires a unique metadata file that informs the edge cluster how to connect with the target cluster. Create the config file by running the following command:
Each edge cluster requires a unique metadata file that informs the edge cluster how to connect with the home cluster. Create the config file by running the following command:


```bash
fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge1 --file edge1.json
fluvio cloud remote export edge1 --file edge1.json
```

The Cloud cluster configuration is now complete. Next, we'll create an edge cluster and configure a mirror topic that synchronizes data to the Cloud.
The Cloud cluster configuration is now complete. Next, we'll create an edge cluster that synchronizes data to the Cloud.


## Install Edge Cluster on Local VM
Expand Down Expand Up @@ -214,7 +209,7 @@ Download binaries:


```bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
```

Add to path:
Expand All @@ -235,11 +230,18 @@ fluvio version

### Start Edge Cluster

We'll use the metadata `edge1` to start the edge cluster:
First we will start the cluster:


```bash
fluvio cluster start
```


Then, we'll connect to the Cloud with the metadata `edge1`:

```bash
fluvio cluster start --read-only edge1.json
fluvio home connect --file edge1.json
```

Let's check the partitions:
Expand All @@ -253,7 +255,7 @@ The edge device should show the following partition:

```bash
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 upstream:0 [] Online 0 B 0 0 0 0 []
edge-topic 0 5001 0:public_endpoint [] Online 0 B 0 0 0 0 []
```

---
Expand Down Expand Up @@ -361,12 +363,19 @@ Restart the cluster:


```bash
fluvio cluster upgrade --read-only edge1.json
fluvio cluster upgrade
```

### Consume from edge cluster

On the `edge` terminal, consume from the cluster:
First, on the `edge` terminal, check the status of the home cluster from the Cloud:

```bash
HOME ROUTE STATUS LAST SEEN
home localhost:30003 Online 3s
```

Then, consume from the edge cluster:


```bash
Expand Down Expand Up @@ -394,4 +403,4 @@ Join us on [Discord] if you have questions, or would like to suggest new improve
* ["IoT Mirroring - Raspberry Pi to a Local Cluster"]


["IoT Mirroring - Raspberry Pi to a Local Cluster"]: /tutorials/iot-mirroring-local
["IoT Mirroring - Raspberry Pi to a Local Cluster"]: /docs/tutorials/iot-mirroring-local.mdx
98 changes: 56 additions & 42 deletions docs/tutorials/iot-mirroring-local.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,18 @@ title: "IoT Mirroring - Local"
description: "Reliable IoT monitoring from movable or static Edges sensors."
---

This advanced tutorial reuqires a Raspberry Pi and a local installation of your collector cluster running on Kubernetes.
This advanced tutorial requires a Raspberry Pi and a local installation of your collector cluster running on Kubernetes.
Checkout the basic version at: ["IoT Mirroring - Cloud"][iot-mirroring-cloud].

## Raspberry Pi to Local Cluster

This section will use Raspberry Pi v3 running Ubuntu 32-bit as the edge device and our local machine for the target cluster.
Let's start with installing and configuring the target cluster.
This section will use Raspberry Pi v3 running Ubuntu 32-bit as the edge device and our local machine for the home cluster.
Let's start with installing and configuring the home cluster.


## Install Target Cluster on Local Machine
## Install Home Cluster on Local Machine

Installing the target cluster on Linux or Mac requires Kubernetes. Use the following instructions to set up Kubernetes on your local machine.
Installing the home cluster on Linux or Mac requires Kubernetes. Use the following instructions to set up Kubernetes on your local machine.

* [Install Rancher Desktop for Mac](https://fluvio.io/docs/get-started/mac/#install-rancher-desktop)
* [Install k3d, kubectl and helm for Linux](https://fluvio.io/docs/get-started/linux/#installing-kubernetes-cluster)
Expand Down Expand Up @@ -47,13 +47,13 @@ Use `curl` to download and install:


```bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
```

Make sure to add `.fluvio/bin` to the `$PATH`as specified in the installation script.


### Start target cluster
### Start home cluster

Use the fluvio binary to start the cluster:

Expand All @@ -69,6 +69,23 @@ Check the result with:
fluvio cluster status
```

### Register Edge clusters

Use the `remote-cluster` CLI to register the edge clusters (edge1 and edge2) with the upstream cluster:

**Edge 1**:


```bash
fluvio cluster remote-cluster register edge1
```

**Edge 2**:


```bash
fluvio cluster remote-cluster register edge2
```

### Create the mirror topic

Expand All @@ -87,7 +104,7 @@ Apply the configuration file to create the topic:


```bash
fluvio topic create edge-topic --mirror-assignment assignment_file.json
fluvio topic create edge-topic --mirror-apply assignment_file.json
```

List partitions to check the assignment:
Expand All @@ -105,37 +122,19 @@ It should display all partitions:
edge-topic 1 5001 edge2 [] Online 0 B 0 0 0 0 []
```

### Register Edge clusters

Use the `remote-cluster` CLI to register the edge clusters (edge1 and edge2) with the upstream cluster:

**Edge 1**:


```bash
fluvio cluster remote-cluster register --type mirror-edge edge1
```

**Edge 2**:


```bash
fluvio cluster remote-cluster register --type mirror-edge edge2
```

List remote clusters to check their status:


```bash
fluvio cluster remote-cluster list
fluvio remote list
```

It should show the following:

```bash
RemoteCluster RemoteType Paired Status Last Seen
edge1 mirror-edge - - -
edge2 mirror-edge - - -
RemoteCluster Status Last Seen
edge1 Waiting -
edge2 Waiting -
```

### Generate Metadata for Edge Clusters
Expand All @@ -146,24 +145,24 @@ Generate a metadata file for each cluster:

**Edge 1**:

The target edge device is a Virtual Machine emulating an IoT device:
The home edge device is a Virtual Machine emulating an IoT device:


```bash
fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge1 --upstream host.orb.internal --file edge1.json
fluvio remote export edge1 --public-endpoint host.orb.internal --file edge1.json
```

**Edge 2**:

The target edge device is a Raspberry Pi device. You may skip this if you don't have such a device.
The home edge device is a Raspberry Pi device. You may skip this if you don't have such a device.

<Idea>
The IP address of our machine where the upstream server is running is `192.168.79.252`. Please identify your own IP address and replace it in the command below.
</Idea>


```bash
fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge2 --upstream 192.168.79.252 --file edge2.json
fluvio remote export edge1 --public-endpoint 192.168.79.252 --file edge2.json
```

We'll transfer these files to edge devices in the following sections.
Expand Down Expand Up @@ -203,18 +202,25 @@ On the raspberry pi, run the following command:


```bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
```

Run `fluvio version` to double check.

### Start cluster

We'll use the metadata file to start the edge cluster on the Raspberry Pi:
First we will start the cluster:


```bash
fluvio cluster start --read-only edge2.json
fluvio cluster start
```


Then, we'll use the metadata file on the Raspberry Pi to connect:

```bash
fluvio home connect --file edge2.json
```

Let's check the partitions:
Expand All @@ -227,8 +233,8 @@ fluvio partition list
The edge device should show the following partition::

```bash
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 Source:upstream:5001 [] Online 0 B 11 11 11 0 []
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 home_name:0:public_endpoint [] Online 0 B 11 11 11 0 []
```


Expand Down Expand Up @@ -357,12 +363,20 @@ Restart the cluster:


```bash
fluvio cluster upgrade --read-only edge2.json
fluvio cluster upgrade
```

### Consume from edge cluster

On the `pi` terminal, consume from the cluster:
First, on the `pi` terminal, check the status of the target cluster:

```bash
HOME ROUTE STATUS LAST SEEN
home localhost:30003 Online 3s
```


Then, consume from the edge cluster:


```bash
Expand All @@ -383,4 +397,4 @@ Produce records and observe that the mirror will resume the synchronization.


[Discord]: https://discord.com/invite/bBG2dTz
[iot-mirroring-cloud]: /tutorials/iot-mirroring-cloud
[iot-mirroring-cloud]: /docs/tutorials/iot-mirroring-cloud.mdx

0 comments on commit 5e144bc

Please sign in to comment.