fix workflows, add docs

.github/workflows/test-circuits.yml

@@ -8,6 +8,7 @@ on:
       - dev
     paths:
       - "packages/circuits/**"
+      - erhant/workspaces # TODO: remove
   push:
     branches:
       - master
@@ -33,7 +34,10 @@ jobs:
           cache: "pnpm"
 
       - name: Install
-        run: pnpm install --frozen-lockfile --filter=./packages/circuits
+        run: pnpm --filter ./packages/circuits install
+
+      - name: Build
+        run: pnpm --filter ./packages/circuits run build
 
-      - name: Run tests
-        run: pnpm test --filter=./packages/circuits
+      - name: Test
+        run: pnpm --filter ./packages/circuits run test

.github/workflows/test-client.yml

@@ -8,6 +8,7 @@ on:
       - dev
     paths:
       - "packages/client/**"
+      - erhant/workspaces # TODO: remove
   push:
     branches:
       - master
@@ -33,10 +34,10 @@ jobs:
           cache: "pnpm"
 
       - name: Install
-        run: pnpm install --frozen-lockfile --filter=./packages/client
+        run: pnpm --filter ./packages/client install
 
       - name: Build
-        run: pnpm build --filter=./packages/client
+        run: pnpm --filter ./packages/client run build
 
-      - name: Run tests
-        run: pnpm test --filter=./packages/client
+      - name: Test
+        run: pnpm --filter ./packages/client run test

.github/workflows/test-core.yml

@@ -3,42 +3,47 @@ name: core-test
 on:
   pull_request:
     types:
-      - opened
+      - review_requested
     branches:
-      - master
+      - dev
+    paths:
+      - "packages/client/**"
+      - erhant/workspaces # TODO: remove
   push:
     branches:
       - master
+      - erhant/workspaces # TODO: remove
+    paths:
+      - "packages/client/**"
 
 jobs:
   test:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+
+      - uses: pnpm/action-setup@v4
+        with:
+          version: 8
 
-      - name: Install Node.js
-        uses: actions/setup-node@v3
+      - name: Use Node.js 20
+        uses: actions/setup-node@v4
         with:
           node-version: 20
+          cache: "pnpm"
+
+      - name: Install
+        run: pnpm --filter ./packages/core install
+
+      - name: Build
+        run: pnpm --filter ./packages/core run build
 
       - name: Start Redis
         uses: supercharge/[email protected]
         with:
           redis-version: 7
           redis-port: 6379
 
-      - uses: pnpm/action-setup@v2
-        name: Install pnpm
-        with:
-          version: 8
-          run_install: false
-
-      - name: Install dependencies
-        run: pnpm install
-
-      - name: Build package
-        run: pnpm build
-
-      - name: Run tests
-        run: pnpm test
+      - name: Test
+        run: pnpm --filter ./packages/core run test

.github/workflows/test-prover.yml

@@ -33,7 +33,10 @@ jobs:
           cache: "pnpm"
 
       - name: Install
-        run: pnpm install --frozen-lockfile --filter=./packages/prover
+        run: pnpm --filter ./packages/prover install
 
-      - name: Run tests
-        run: pnpm test --filter=./packages/prover
+      - name: Build
+        run: pnpm --filter ./packages/prover run build
+
+      - name: Test
+        run: pnpm --filter ./packages/prover run test

docs/ZERO-KNOWLEDGE.md

@@ -0,0 +1,94 @@
+# Zero-Knowledge Proofs
+
+HollowDB utilizes [zero-knowledge proofs](https://en.wikipedia.org/wiki/Zero-knowledge_proof) within to provide a zero-knowledge authentication scheme. So, what are they?
+
+Zero-Knowledge Proofs (ZKPs) is a method that allows one to prove that a given statement is true, without revealing any other information other than the statement itself! We usually refer to the proving party as the **Prover**, and the verifying party as the **Verifier**.
+
+Example statements are:
+
+- "I know the solution to some puzzle", which the prover must prove without showing the solution itself.
+- "I know some $x$ such that $f(x) = 0$", which the prover must prove without revealing what $x$ is.
+- "I know the private key that corresponds to some public key", which the prover must prove without revealing the private key.
+
+## Anatomy of a ZKP
+
+Looking at a zero-knowledge protocol as a very high-level diagram, we have the following flow:
+
+```mermaid
+graph
+  TODO
+```
+
+A prover has some secret inputs (a witness) that they would like to keep secret, and they might also have some public inputs. They feed these into an algebraic circuit, which is really like an electric circuit but instead of electricity, it works on non-negative integers (i.e. elements of a finite field) and you can only do addition and multiplication.
+
+As a result, they get the output of this computation, along with a proof. Note that the output is not really necessary too, you could also have a proof without giving any outputs, which is a way of saying "hey I have ran this circuit that you have told me to, and I got no errors". An example of this is a Sudoku solution prover circuit, where a public puzzle is provided and the user feeds their secret solution to the circuit. The circuit then makes sure the solution is valid, and basically compiles without failures if indeed it is valid.
+
+## Hashing
+
+Before we move on, we also need to describe what "hashing" is, which is used extensively in the zero-knowledge proofs of HollowDB. Hashing is simply a function that takes some arbitrary input, and outputs a fixed-length output. We refer to the input as preimage, and the output as digest or hash.
+
+We expect the following properties from a hash function $H$:
+
+- Given some hash $y$ such that $y=H(x)$ it should be really hard to find what $x$ is. This is called **preimage resistance**.
+- Given an input $x_1$, it should be really hard to find another input $x_2$ such that $H(x_1) = H(x_2)$. This is called **second-preimage resistance**.
+- Given two inputs $x_1$ and $x_2$, it should be really unlikely that $H(x_1) = H(x_2)$. This is called **collision resistance**.
+- The output of the hash function should appear "random", i.e. it should be distributed as evenly as possible. In doing so, even just a slight change in the input should completely change the output. This is commonly referred to as **avalanche effect**.
+
+Note that the input size is arbitrary but the output size is fixed, is that a problem for the properties above? Well it certainly could be; however, in practice the output length of these hash functions are pretty big, such as 256-bits or 512-bits. There are $2^{256}$ possible outputs for a 256-bit output, which is a lot more than the number of atoms in the world.
+
+So how does hashing relate to zero-knowledge proofs? Imagine that you wrote the entire hash function as an arithmetic circuit, and you provide the preimage as the secret input. The output will be the digest, and you will have a proof that you know what preimage resulted in this digest. In other words, you can prove the statement "I know some $x$ such that $y = H(x)$ for a publicly known $y$" by simply writing the entire hash function $H$ as a circuit.
+
+There are many different hash functions with varying security levels and output lengths, and the most important thing to note is that not all of them are circuit-friendly. What this means is that, some hash functions (e.g. **SHA256**) are really costly to implement with a circuit. A higher cost means more gates and more constraints, thus requiring a longer proving time and circuit-setup time. Thankfully, there are friendlier hash functions, a well-known one being the **Poseidon** hash.
+
+## Hollow Authorization
+
+HollowDB is a key-value database, and we want users to have control over their data, i.e. only they should be able to change the value at their respective key. This is achieved by the following:
+
+- User knows some secret $s$.
+- They hash this secret to obtain a key $k$ as in $k \gets H(s)$.
+
+The user will PUT to this key, and only they will be able to UPDATE or REMOVE this key! This is done by requiring a ZKP that whoever wants to update some key knows the preimage $s$ of that key. Lets rewrite the diagram above to show what is happening here:
+
+```mermaid
+graph
+  TODO
+```
+
+Lets examine this diagram:
+
+- The client wants to update some `key` that they know the preimage of, with some new `value`.
+- Client generates a zero-knowledge proof to prove that they indeed know the preimage. They send the proof, along with the `key` and the `value` to the smart contract, which is HollowDB.
+- Within the smart contract, the proof is verified and if it is valid, the key is updated. If proof is invalid, transaction is reverted.
+
+The important point here is that Smart Contract does not see `preimage` at all!
+
+### Security Issues
+
+If you think about this method in practice, it has two security issues:
+
+- **Replay Attack**: If an adversary gets hold of your proof, they can use that proof to claim that they know the preimage to your key even though if they dont! This is like someone stealing your credit card, and then doing contactless payments that do not ask for password on low amounts. We would like to prevent this.
+- **Middle-man Attack**: Another issue is that the proof contains nothing related to the value to be written. If an adversary steals your proof before it gets to the smart contract (perhaps a middle-man between you and the smart contract) then they can change the value to be written by simply using your proof.
+
+The solution to these problems are simple: we need to put some constraints within our proof related to the current value and the new value to be written.
+
+- if the proof is only valid for some current value at that key, it will be invalid when that value changes, thus preventing the replay attack.
+- if the proof is only valid for the new value that I am going to write to that key, it will be invalid for any other value, thus preventing the middle-man attack.
+
+However, we have said that arithmetic circuits operate on integers, but our values can be anything; so how do we represent our values as integers? The answer is: hashing! The client will hash both the current value and the new value separately, obtaining two hashes. So now lets see the final diagram that shows how how both attacks are mitigated:
+
+```mermaid
+graph
+  TODO
+```
+
+> [!NOTE]
+>
+> You must ensure that the resulting hash is within the limit of the finite-field used in the circuit. For example, [Circom](https://docs.circom.io/) supports the [Baby JubJub](https://eips.ethereum.org/EIPS/eip-2494) elliptic curve for its arithmetic circuits to be used in Ethereum, and the order of the finite field is:
+>
+> ```
+> 21888242871839275222246405745257275088548364400416034343698204186575808495617
+> ```
+>
+> This means that any value in your circuit must be less than this number. If you have a larger number, they will wrap back around as in modular arithmetic. You can use different curves and thus different orders, but this is important to keep in mind.
+>
+> The number above is around 254 bits, and if we were to input a 256-bit number, things may work without the way we intend them to. This is especially important if we are trying to use a hash as an input. One could use hash functions with smaller output size, such as [ripemd160](https://en.bitcoin.it/wiki/RIPEMD-160), which has a 160-bit output and is definitely within the limits of our arithmetic circuit which is much larger. There are other methods to "fit" a hash within a field element too.

packages/client/README.md

@@ -44,12 +44,16 @@ pnpm add hollowdb-client    # pnpm
 
 ## Usage
 
+> [!CAUTION]
+>
+> HollowDB API service is currently disabled until further notice.
+
 Create a new API key and a database at <https://developer.hollowdb.xyz>. Create a new client by providing your API key and the database name to connect:
 
 ```ts
 client = await HollowClient.new({
-  apiKey: 'your-api-key',
-  db: 'your-database-name',
+  apiKey: "your-api-key",
+  db: "your-database-name",
 });
 ```
 

packages/prover/README.md

@@ -24,12 +24,6 @@
     <a href="./.github/workflows/test.yml" target="_blank">
         <img alt="Workflow: Tests" src="https://github.com/firstbatchxyz/hollowdb-prover/actions/workflows/test.yml/badge.svg?branch=master">
     </a>
-    <a href="https://github.com/firstbatchxyz/hollowdb" target="_blank">
-        <img alt="GitHub: HollowDB" src="https://img.shields.io/badge/github-hollowdb-5C3EFE?logo=github">
-    </a>
-    <a href="https://discord.gg/2wuU9ym6fq" target="_blank">
-        <img alt="Discord" src="https://dcbadge.vercel.app/api/server/2wuU9ym6fq?style=flat">
-    </a>
 </p>
 
 ## Installation
@@ -46,7 +40,134 @@ pnpm add hollowdb-prover    # pnpm
 
 <!-- See documentation at https://docs.hollowdb.xyz/zero-knowledge-proofs/hollowdb-prover. -->
 
-TODO: !!!
+To generate proofs, you will need the zero-knowledge circuit WASM file, and a prover key. Both can be found within the repo, see [here](../circuits/). Notice that there are separate files for each protocol, **Groth16** and **PLONK** respectively.
+
+To create the prover:
+
+```ts
+const prover = new Prover(
+  "./path-to-circuit-wasm",
+  "./path-to-prover-key",
+  "groth16" // or "plonk"
+);
+```
+
+Let us explain the constructor arguments in order:
+
+- `wasmPath` is the relative path to the circuit WASM file. In a web application, this file can be stored under public.
+- `proverKeyPath` is the relative path to the WASM circuit. In a web application, this file can be stored under public.
+- `protocol` is the proof system to be used, that is either `groth16` or `plonk`. HollowDB supports both proof systems, and the verifier can determine which one to use by looking at the verification key.
+
+To generate a proof, simply call `prove` function of the newly created `prover`:
+
+```ts
+const {proof, publicSignals} = prover.prove(PREIMAGE, CURRENT_VALUE, NEXT_VALUE);
+```
+
+The proof object here shall be provided to HollowDB contract, where it will be checked to verify. Note that public signals are also exported, although we do not use them; the contract obtains them in it's own ways.
+
+The value inputs are "hashed-to-group" and then fed into the circuit. See the section below for more information. For the curious, the public signals is a triple with the following elements in order:
+
+- Current value hash
+- Next value hash
+- Key, equal to Poseidon hash of the preimage
+
+### Prove with Hashes
+
+The `prove` function takes as input two objects, and it converts them to be circuit-friendly inputs within the function. If you would like to re-use these hashes, or you simply have access to them, you can generate a proof from them too:
+
+```ts
+const {proof} = prover.proveHashed(PREIMAGE, CUR_VAL_HASH, NEXT_VAL_HASH);
+```
+
+### Proving in NextJS
+
+Note that to use SnarkJS in a NextJS environment you may need to configure some settings w.r.t server-side rendering. We suggest adding the following Webpack option to your NextJS config:
+
+```js
+webpack: (config, { isServer }) => {
+  if (!isServer) {
+    config.resolve.alias = {
+      ...config.resolve.alias,
+      fs: false, // added for SnarkJS
+      readline: false, // added for SnarkJS
+    };
+  }
+  // added to run WASM for SnarkJS
+  config.experiments = { asyncWebAssembly: true };
+  return config;
+},
+```
+
+You might also have to make some configurations in other frameworks if you have server-side rendering enabled.
+
+### Computing the Key without Proofs
+
+When HollowDB is used with proofs in particular, the z is computed by taking the [Poseidon hash](https://www.poseidon-hash.info/) of some secret preimage. The key can be extracted from the z which is in the object that is returned from the z function.
+
+However, if one wants to compute the z without creating a proof (e.g. the user just wants to get a value at their own key) they can do so with `computeKey`.
+
+```ts
+import {computeKey} from "hollowdb-prover";
+
+const key = computeKey(PREIMAGE);
+```
+
+### Hash-to-Group
+
+To "embed" the current value and next value within our proofs, we need to map them to a number. This number must be circuit-friendly, and we provide a `hashToGroup` function for this purpose:
+
+```ts
+import {hashToGroup} from "hollowdb-prover";
+
+const valueHashed: bigint = hashToGroup({foo: "bar", num: 123});
+const valueHashedHex: string = "0x" + valueHashed.toString(16);
+```
+
+## Example
+
+The proof verification is done within the smart-contract side, so as a developer we will mostly be looking at the proof generation that happens on the client side. Below is an example:
+
+```ts
+import {SDK} from "hollowdb";
+import {Prover, computeKey} from "hollowdb-prover";
+import {WarpFactory, JWKInterface} from "warp-contracts";
+import fs from "fs";
+
+// read wallet
+const walletPath = __dirname + "/wallet-name.json";
+const wallet = JSON.parse(fs.readFileSync(walletPath).toString()) as JWKInterface;
+
+// instantiate SDK
+const contractTxId = "<your-contract-txid>";
+const sdk = new SDK(wallet, contractTxId, WarpFactory.forMainnet());
+
+// instantiate the prover
+const wasmCircuitPath = __dirname + "/circuit.wasm";
+const proverKeyPath = __dirname + "/prover_key.zkey";
+const prover = new Prover(wasmCircuitPath, proverKeyPath);
+
+// compute your key from secret
+const secret = BigInt("0xDEADBEEF");
+const key = computeKey(secret);
+
+// generate a proof for UPDATE
+const currentValue = await sdk.get(key);
+const nextValue = "this is a new value!";
+const {proof} = await prover.prove(secret, currentValue, nextValue);
+
+// update
+await sdk.update(key, nextValue, proof);
+```
+
+Let's digest this code step by step:
+
+1. First, we read our Arweave wallet from file, to be used for our transactions. You could also provide the wallet as a JSON object within the code too (but you should be careful not to expose your wallet & accidentally commit them to your repo).
+2. Then, we create the HollowDB SDK object. For this, we provide our wallet, we specify the cache type to be LMDB, and we provide the contract transaction id along with a Warp instance. Basically, we are "connecting" to our contract on the Mainnet.
+3. We now create our Prover object, which is a wrapper around a few SnarkJS functions to generate a proof. We have to provide a path to our WASM circuit and a prover key to create this object. You can obtain them from our repository, and host them on your side. For example, if you are writing a web application, you could host them under the public folder.
+4. We need to compute the key, which is the hash of our preimage. You could generate a dummy proof and read the key from its output, but that is not really efficient. Instead, HollowDB exports a computeKey function for this purpose.
+5. Then, we generate our proof by simply calling prove with the required arguments, that are the inputs we have shown in the above diagram.
+6. Finally, we call sdk.update to update the value at our key, using our zero-knowledge proof!
 
 ## Testing