Simple Geolocation Probing #445

Soptq · 2021-09-03T01:47:38Z

This PR tries to implement simple geolocation probing in Phala Network.

Goal:

Implement a contract to collects desensitized geolocation data from workers.
Provide contract query entries.
Using SecretMessageChannel to secure communication between pRuntime and the contract.

Currently this PR is considered to be merged into geo-probing branch for safety, but if you think it is ready, feel free to merge it into the master branch.

…zed geolocation data.

SendCoordinateInfo RPC Sends geolocation data to pruntime to initialize a secret message channel toward geolocation contract. Echo RPC can be used to measure network RTT. Relevant to Phala-Network/phala-blockchain#445

Soptq · 2021-09-03T01:49:06Z

The Protobuf changes is proposed in Phala-Network/prpc-protos#2

Soptq · 2021-09-03T01:49:57Z

ping @kvinwang, will close the previous PR. This PR is currently finalized.

h4x3rotab · 2021-09-03T03:45:30Z

Could you rebase to the latest master branch? Now it's full of history and hard to review.

Soptq · 2021-09-03T03:47:20Z

Could you rebase to the latest master branch? Now it's full of history and hard to review.

Done. Now this PR will be merged into master branch directly.

kvinwang · 2021-09-03T04:06:53Z

crates/phactory/src/prpc_service.rs

+            .send_mq
+            .channel(sender, id_pair);
+        let secret_mq = SecretMessageChannel::new(&ecdh_key,
+                                                  &mq,


Looks like this will break the deterministic of the mq egress?
Is it better that pherry directly report the infomation to the chain?

I previously implemented a version that uses pallets to redirect contract commands, but later I with @shelvenzhou decided to refactor the code to initialize the contract command call directly in pRuntime, mainly to take advantages of SecretMessageChannel.

(For @kvinwang and @shelvenzhou)

Need some general discussion about feeding external data to the consensus system.

Maybe we should allow the sender to feed "loosely consistent" messages to the blockchain. The message is still signed by the sender (e.g. a worker), and has a ingress sequence id associated with it. However the sender no longer send it deterministically. The determinism only start to take effect since the message arrives the blockchain and thus permanently timestamped. Before arriving the blockchain, there's no guarantee on if the message will be processed or now.

A very naive implementation on this specific case can be:

pherry gets the geolocation in some way

pherry gets the latest ingress sequence of WorkerLoose(worker_pubkey)

pherry sends the geolocation data to pruntime by RPC

pruntime encrypts the data, and signs it with the worker identity key, return the pherry

pherry sends the signed payload to the blockchain

This process is not guaranteed to be deterministic. If it fails, we can just redo it a bit later. And if we think one step further, probably we can also let not only pherry, but also any components to initiate a non-deterministic message to the mq. Examples are:

A simple query can results in some data feed back to the blockchain

An on-chain command can trigger an async action (e.g. sending a http request), and once the action is finished, it can sends the result back to the blockchain.

Of course, the above example can be slightly modified to be originated from pruntime itself rather than pherry.

Other thoughts?

h4x3rotab

Besides all the in-line comments, I found the geolocation never get removed. Let's consider adding a TTL to each report, and let the client to refresh the data before it gets expired.

Summary:

TTL of the geolocation records
Admin permission to read the raw data (or remove the ability to read raw data completely)
Non-deterministic message submission

crates/phactory/src/contracts/geolocation.rs

h4x3rotab · 2021-09-04T08:57:43Z

standalone/pherry/README.md

@@ -0,0 +1,3 @@
+Download GeoLite-City.mmdb database at https://drive.google.com/file/d/1UDKHuZ2KQSaDvy34LMKlUPKgToZPoNUk/view?usp=sharing


Need to figure out the license of the database. Then maybe we can redistribute on our end.

Unfortunately we may not. Redistributing of the geolite2 database needs to purchase a "GeoLite2 Commercial Redistribution License".

https://www.maxmind.com/en/geolite2-commercial-redistribution#:~:text=GeoLite2%20Commercial%20Redistribution%20License%20Our%20commercial%20redistribution%20license,GeoLite2%20databases%20%28including%20updates%29%20royalty-free%20for%20one%20year.

No worry. I see if we meet some criteria we are fine:

Auto update (we can do it later now)

Not used to identify individuals or households (we only output aggregated data!)

In any case, we may consider to buy a commercial license in the future.

standalone/pherry/README.md

standalone/pherry/src/main.rs

Soptq · 2021-09-04T16:44:36Z

The geolocation report ttl is added to the pherry. Now workers will update the geolocation record when started, and then it will be updated again every 8 hours.

…zed geolocation data.

…lockchain into dev-geo-probing-v2

h4x3rotab · 2021-09-09T14:10:45Z

I'm ok with this PR. A few follow-ups as discussed:

Move the geocoding from pherry to pRuntime: We have not only pherry but also the next generation version PRB. So by moving the logic to pRuntime, we simplify the overall complexity

This relies on the access to HTTP and file system. We have both, but not enabled by default. Will check with @kvinwang
Enable geolocation submission in pRuntime.

This relies on the design and api of Side Inputs. We have already discussed a few rounds of the design, and now approaching to the first implement.
Smoke test. We can set up a local dev environment with geolocation contract running. Then we register a worker. With the geolocation get into the contract, we can query the contract to get the aggregated result.
Unit tests in the geolocation contract.

SendCoordinateInfo RPC Sends geolocation data to pruntime to initialize a secret message channel toward geolocation contract. Echo RPC can be used to measure network RTT. Relevant to Phala-Network/phala-blockchain#445

implement a secure geolocation contract to collect workers' desensiti…

e9c6a79

…zed geolocation data.

Soptq mentioned this pull request Sep 3, 2021

Add SendCoordinateInfo and Echo RPC Phala-Network/prpc-protos#2

Merged

Soptq mentioned this pull request Sep 3, 2021

WIP: Geolocation Probing #430

Closed

3 tasks

Soptq changed the base branch from geo-probing to master September 3, 2021 03:46

kvinwang reviewed Sep 3, 2021

View reviewed changes

h4x3rotab reviewed Sep 4, 2021

View reviewed changes

h4x3rotab linked an issue Sep 4, 2021 that may be closed by this pull request

Geolocation Probes the simple appraoch #429

Closed

2 tasks

add geolocation report ttl and fix miscs.

14683eb

Soptq and others added 7 commits September 5, 2021 09:55

handle possible err better

2f7f247

implement a secure geolocation contract to collect workers' desensiti…

6e89453

…zed geolocation data.

implement a secure geolocation contract to collect workers' desensiti…

a52acaa

…zed geolocation data.

add geolocation report ttl and fix miscs.

b13fd9d

handle possible err better

254bc41

implement a secure geolocation contract to collect workers' desensiti…

ff49512

…zed geolocation data.

move geo db out to file; refactor error

f46824d

h4x3rotab force-pushed the dev-geo-probing-v2 branch from 4d77e22 to f46824d Compare September 8, 2021 09:48

Soptq and others added 3 commits September 9, 2021 14:05

Update README.md

b59d9e5

Merge branch 'dev-geo-probing-v2' of https://github.com/soptq/phala-b…

d53aeb4

…lockchain into dev-geo-probing-v2

gracefully handle error when geolocation db is not provided.

dc7cc94

Merge remote-tracking branch 'origin/master' into geo

d211fb3

h4x3rotab enabled auto-merge September 9, 2021 21:54

h4x3rotab merged commit 7d8347b into Phala-Network:master Sep 9, 2021

h4x3rotab mentioned this pull request Sep 24, 2021

Geolocation Probes the simple appraoch #429

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simple Geolocation Probing #445

Simple Geolocation Probing #445

Soptq commented Sep 3, 2021 •

edited

Loading

Soptq commented Sep 3, 2021 •

edited

Loading

Soptq commented Sep 3, 2021

h4x3rotab commented Sep 3, 2021

Soptq commented Sep 3, 2021

kvinwang Sep 3, 2021 •

edited

Loading

Soptq Sep 3, 2021

h4x3rotab Sep 4, 2021

h4x3rotab left a comment •

edited

Loading

h4x3rotab Sep 4, 2021

Soptq Sep 4, 2021

h4x3rotab Sep 4, 2021

Soptq commented Sep 4, 2021

h4x3rotab commented Sep 9, 2021

		@@ -0,0 +1,3 @@
		Download GeoLite-City.mmdb database at https://drive.google.com/file/d/1UDKHuZ2KQSaDvy34LMKlUPKgToZPoNUk/view?usp=sharing

Simple Geolocation Probing #445

Simple Geolocation Probing #445

Conversation

Soptq commented Sep 3, 2021 • edited Loading

Soptq commented Sep 3, 2021 • edited Loading

Soptq commented Sep 3, 2021

h4x3rotab commented Sep 3, 2021

Soptq commented Sep 3, 2021

kvinwang Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

Soptq Sep 3, 2021

Choose a reason for hiding this comment

h4x3rotab Sep 4, 2021

Choose a reason for hiding this comment

h4x3rotab left a comment • edited Loading

Choose a reason for hiding this comment

h4x3rotab Sep 4, 2021

Choose a reason for hiding this comment

Soptq Sep 4, 2021

Choose a reason for hiding this comment

h4x3rotab Sep 4, 2021

Choose a reason for hiding this comment

Soptq commented Sep 4, 2021

h4x3rotab commented Sep 9, 2021

Soptq commented Sep 3, 2021 •

edited

Loading

Soptq commented Sep 3, 2021 •

edited

Loading

kvinwang Sep 3, 2021 •

edited

Loading

h4x3rotab left a comment •

edited

Loading