Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(guide/dozer): move hydration to its own guide 🚗 #3188

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
8b76bdd
feat(store-sync): add util to fetch snapshot from dozer
alvrs Jul 31, 2024
52dd46b
add missing changes
alvrs Jul 31, 2024
121479d
skip test
alvrs Jul 31, 2024
9abfffc
parallelize sql query requests
alvrs Jul 31, 2024
367daec
add KeySchema
alvrs Aug 1, 2024
dc154d2
move protocol-parser changes to store-sync
alvrs Aug 1, 2024
bf885cc
move KeySchema
alvrs Aug 1, 2024
aca18b9
keep zustand types
alvrs Aug 1, 2024
b36d6fe
rename things
alvrs Aug 1, 2024
175fe7c
fix type error
alvrs Aug 1, 2024
a677f32
missed one PartialTable
alvrs Aug 1, 2024
9c398d8
update logic in dozer/getSnapshot
alvrs Aug 1, 2024
f22747a
chore: include main branch when fetching during pre-release action
Kooshaba Aug 5, 2024
0af11fa
docs(state-sync/dozer): first version (#3033)
qbzzt Aug 22, 2024
fa2bbc7
Merge branch 'main' into alvrs/dozer-query
alvrs Sep 3, 2024
25a2a77
remove dozer from file and function name
alvrs Sep 3, 2024
6cb9b3f
rename fetchRecordsSql to fetchRecords
alvrs Sep 3, 2024
f2fbaad
use dozer base url as input to fetchRecords
alvrs Sep 3, 2024
c92258e
move error handling into getSnapshot
alvrs Sep 3, 2024
494b0bb
refactors
alvrs Sep 3, 2024
11c5cdd
stylistic changes
alvrs Sep 3, 2024
9adc159
remove unrelated change
alvrs Sep 3, 2024
403179f
review fixes
alvrs Sep 5, 2024
6488f2c
fix export conflict
alvrs Sep 5, 2024
c71eff1
Merge branch 'main' into alvrs/dozer-query
alvrs Sep 5, 2024
1c77240
docs(filter-sync and dozer): explain that there are two sync types 🚗 …
qbzzt Sep 17, 2024
b819a2c
docs(dozer): add metadata query 🚗 (#3186)
qbzzt Sep 17, 2024
3a6d21e
docs(guide/dozer): move hydration to its own guide
qbzzt Sep 17, 2024
03b7641
types of sync
qbzzt Sep 17, 2024
9056439
add callout for sync types
qbzzt Sep 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/components/common-text/FilterTypes.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { Callout } from "nextra/components";

<Callout type="info">
MUD initial data hydration, and therefore filtering, comes in two flavors: [Dozer](/state-query/dozer) and [generic](/guides/hello-world/filter-sync).
Note that this is for the initial hydration, currently limits on on-going synchronization are limited to [the generic method](/guides/hello-world/filter-sync).

| | Dozer | Generic |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Filtering | Can filter on most SQL functions | Can only filter on tables and the first two key fields (limited by [`eth_getLogs`](https://ethereum.github.io/execution-apis/api-documentation/) filters) |
| Availability | [Redstone](https://redstone.xyz/docs/network-info), [Garnet](https://garnetchain.com/docs/network-info), or elsewhere if you run your own instance | Any EVM chain |
| Security assumptions | The Dozer instance returns accurate information | The endpoint returns accurate information (same assumption as any other blockchain app) |

</Callout>
1 change: 1 addition & 0 deletions docs/pages/guides/hello-world/_meta.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export default {
"add-table": "Add a table",
"filter-sync": "Filter data synchronization",
"dozer": "Add dozer hydration",
"add-system": "Add a system",
"deploy": {
"title": "Deploy to a blockchain",
Expand Down
282 changes: 282 additions & 0 deletions docs/pages/guides/hello-world/dozer.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,282 @@
import { CollapseCode } from "../../../components/CollapseCode";
import FilterTypes from "../../../components/common-text/FilterTypes.mdx";

# Add dozer hydration

<FilterTypes />

In this tutorial you learn how to add dozer hydration to an existing MUD application, such as the ones created by the template.
To avoid running dozer locally, we use a `World` on Garnet at address [`0x95F5d049B014114E2fEeB5d8d994358Ce4FFd06e`](https://explorer.garnetchain.com/address/0x95F5d049B014114E2fEeB5d8d994358Ce4FFd06e) that runs a slightly modified version of [the React template](https://github.com/latticexyz/mud/tree/main/templates/react).
You can see the data schema for the `World` [in the block explorer](https://explorer.garnetchain.com/address/0x95F5d049B014114E2fEeB5d8d994358Ce4FFd06e?tab=mud).

## Create a client to access the `World`

These are the steps to create a client that can access the `World`.

1. Create and run a react template application.

```sh copy
pnpm create mud@latest tasks --template react
cd tasks
pnpm dev
```

1. [Browse to the application](http://localhost:3000/?chainId=17069&worldAddress=0x95f5d049b014114e2feeb5d8d994358ce4ffd06e).
The URL specifies the `chainId` and `worldAddress` for the `World`.

1. In MUD DevTools see your account address and [fund it on Garnet](https://garnetchain.com/faucet).
You may need to get test ETH for your own address, and then transfer it to the account address the application uses.

1. You can now create, complete, and delete tasks.

1. To see the content of the `app__Creator` table, edit `packages/contracts/mud.config.ts` to add the `Creator` table definition.

<CollapseCode>

```typescript filename="mud.config.ts" copy showLineNumbers {15-21}
import { defineWorld } from "@latticexyz/world";

export default defineWorld({
namespace: "app",
tables: {
Tasks: {
schema: {
id: "bytes32",
createdAt: "uint256",
completedAt: "uint256",
description: "string",
},
key: ["id"],
},
Creator: {
schema: {
id: "bytes32",
taskCreator: "address",
},
key: ["id"],
},
},
});
```

</CollapseCode>

### Updating the client to use dozer

The main purpose of dozer is to allow MUD clients to specify the subset of table records that a client needs, instead of synchronizing whole tables.

To update the client, you change `packages/client/src/mud/setupNetwork.ts` to:

<CollapseCode>

```typescript filename="setupNetwork.ts" copy showLineNumbers {17, 80-97, 106-107}
/*
* The MUD client code is built on top of viem
* (https://viem.sh/docs/getting-started.html).
* This line imports the functions we need from it.
*/
import {
createPublicClient,
fallback,
webSocket,
http,
createWalletClient,
Hex,
ClientConfig,
getContract,
} from "viem";

import { DozerSyncFilter, getSnapshot, selectFrom } from "@latticexyz/store-sync/dozer";

import { syncToZustand } from "@latticexyz/store-sync/zustand";
import { getNetworkConfig } from "./getNetworkConfig";
import IWorldAbi from "contracts/out/IWorld.sol/IWorld.abi.json";
import { createBurnerAccount, transportObserver, ContractWrite } from "@latticexyz/common";
import { transactionQueue, writeObserver } from "@latticexyz/common/actions";
import { Subject, share } from "rxjs";

/*
* Import our MUD config, which includes strong types for
* our tables and other config options. We use this to generate
* things like RECS components and get back strong types for them.
*
* See https://mud.dev/templates/typescript/contracts#mudconfigts
* for the source of this information.
*/
import mudConfig from "contracts/mud.config";

export type SetupNetworkResult = Awaited<ReturnType<typeof setupNetwork>>;

export async function setupNetwork() {
const networkConfig = await getNetworkConfig();

/*
* Create a viem public (read only) client
* (https://viem.sh/docs/clients/public.html)
*/
const clientOptions = {
chain: networkConfig.chain,
transport: transportObserver(fallback([webSocket(), http()])),
pollingInterval: 1000,
} as const satisfies ClientConfig;

const publicClient = createPublicClient(clientOptions);

/*
* Create an observable for contract writes that we can
* pass into MUD dev tools for transaction observability.
*/
const write$ = new Subject<ContractWrite>();

/*
* Create a temporary wallet and a viem client for it
* (see https://viem.sh/docs/clients/wallet.html).
*/
const burnerAccount = createBurnerAccount(networkConfig.privateKey as Hex);
const burnerWalletClient = createWalletClient({
...clientOptions,
account: burnerAccount,
})
.extend(transactionQueue())
.extend(writeObserver({ onWrite: (write) => write$.next(write) }));

/*
* Create an object for communicating with the deployed World.
*/
const worldContract = getContract({
address: networkConfig.worldAddress as Hex,
abi: IWorldAbi,
client: { public: publicClient, wallet: burnerWalletClient },
});

const dozerUrl = "https://dozer.mud.garnetchain.com/q";
const yesterday = Date.now() / 1000 - 24 * 60 * 60;
const filters: DozerSyncFilter[] = [
selectFrom({
table: mudConfig.tables.app__Tasks,
where: `"createdAt" > ${yesterday}`,
}),
{ table: mudConfig.tables.app__Creator },
];
const { initialBlockLogs } = await getSnapshot({
dozerUrl,
storeAddress: networkConfig.worldAddress as Hex,
filters,
chainId: networkConfig.chainId,
});
const liveSyncFilters = filters.map((filter) => ({
tableId: filter.table.tableId,
}));

/*
* Sync on-chain state into RECS and keeps our client in sync.
* Uses the MUD indexer if available, otherwise falls back
* to the viem publicClient to make RPC calls to fetch MUD
* events from the chain.
*/
const { tables, useStore, latestBlock$, storedBlockLogs$, waitForTransaction } = await syncToZustand({
initialBlockLogs,
filters: liveSyncFilters,
config: mudConfig,
address: networkConfig.worldAddress as Hex,
publicClient,
startBlock: BigInt(networkConfig.initialBlockNumber),
});

return {
tables,
useStore,
publicClient,
walletClient: burnerWalletClient,
latestBlock$,
storedBlockLogs$,
waitForTransaction,
worldContract,
write$: write$.asObservable().pipe(share()),
};
}
```

</CollapseCode>

<details>

<summary>Explanation</summary>

```typescript
import { DozerSyncFilter, getSnapshot, selectFrom } from "@latticexyz/store-sync/dozer";
```

Import the dozer definitions we need.

```typescript
const dozerUrl = "https://dozer.mud.garnetchain.com/q";
```

The URL for the dozer service.
This is simplified testing code, on a production system this will probably be a lookup table based on the chainId.

```typescript
const yesterday = Date.now() / 1000 - 24 * 60 * 60;
```

In JavaScript (and therefore TypeScript), time is stored as milliseconds since [the beginning of the epoch](https://en.wikipedia.org/wiki/Unix_time).
In UNIX, and therefore in Ethereum, time is stored as seconds since that same point.
This is the timestamp 24 hours ago.

```typescript
const filters: DozerSyncFilter[] = [
```

We create the dozer filter for the tables we're interested in.
This is the _dozer_ filter, so it is only used for the initial hydration of the client.

```typescript
selectFrom({
table: mudConfig.tables.app__Tasks,
where: `"createdAt" > ${yesterday}`,
}),
```

From the `app__Tasks` table we only want entries created in the last 24 hours.
To verify that the filter works as expected you can later change the code to only look for entries older than 24 hours.

```typescript
{ table: mudConfig.tables.app__Creator },
];
```

We also want the entire `app__Counter` table.

```typescript
const { initialBlockLogs } = await getSnapshot({
dozerUrl,
storeAddress: networkConfig.worldAddress as Hex,
filters,
chainId: networkConfig.chainId,
});
```

Get the initial snapshot to hydrate (fill with initial information) the data store.
Note that this snapshot does not have the actual data, but the events that created it.

```typescript
const liveSyncFilters = filters.map((filter) => ({
tableId: filter.table.tableId,
}));
```

The live synchronization filters are used after the initial hydration, and keep up with changes on the blockchain.
These synchronization filters are a lot more limited, [you can read the description of these filters here](/guides/hello-world/filter-sync#filtering).

```typescript
const { ... } = await syncToZustand({
initialBlockLogs,
filters: liveSyncFilters,
...
});
```

Finally, we provide `initialBlockLogs` for the hydration and `filters` for the updates to the synchronization function (either `syncToRecs` or `syncToZustand`).

</details>
15 changes: 15 additions & 0 deletions docs/pages/guides/hello-world/filter-sync.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,25 @@
import { CollapseCode } from "../../../components/CollapseCode";
import FilterTypes from "../../../components/common-text/FilterTypes.mdx";

# Filter data synchronization

In this tutorial you modify `networkSetup.ts` to filter the information you synchronize.
Filtering information this way allows you to reduce the use of network resources and makes loading times faster.

<FilterTypes />

<details>

<summary>Why are only the first two key fields available for filtering?</summary>

Ethereum log entries can have [up to four indexed fields](https://www.evm.codes/?fork=cancun#a4).
However, Solidity only supports [three indexed fields](https://www.alchemy.com/overviews/solidity-events) because the first indexed field is used for the event name and type.
In MUD, [this field](https://github.com/latticexyz/mud/blob/main/packages/store/src/IStoreEvents.sol) specifies whether [a new record is created](https://github.com/latticexyz/mud/blob/main/packages/store/src/IStoreEvents.sol#L26-L32), a record is changed (either [static fields](https://github.com/latticexyz/mud/blob/main/packages/store/src/IStoreEvents.sol#L43) or [dynamic fields](https://github.com/latticexyz/mud/blob/main/packages/store/src/IStoreEvents.sol#L56-L64)), or [a record is deleted](https://github.com/latticexyz/mud/blob/main/packages/store/src/IStoreEvents.sol#L71).
The second indexed fields is always the table's [resource ID](/world/resource-ids).
This leaves two fields for key fields.

</details>

## Setup

To see the effects of filtering we need a table with entries to filter. To get such a table:
Expand Down
1 change: 1 addition & 0 deletions docs/pages/state-query/_meta.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
export default {
dozer: "Dozer",
typescript: "TypeScript",
};
Loading
Loading