Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve distributor oracle perf #406

Merged
merged 32 commits into from
Sep 15, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
0f80be7
Fix iot active devices (#405)
ChewingGlass Sep 12, 2023
96cefc2
improve distributor oracle perf
ChewingGlass Sep 13, 2023
9c2de13
chore(release): publish 0.2.22
ChewingGlass Sep 13, 2023
a04d294
Add canary release option
ChewingGlass Sep 13, 2023
2a1e199
Rm
ChewingGlass Sep 13, 2023
8ab99d3
Fix canary
ChewingGlass Sep 13, 2023
59d8d46
Add comment to canary
ChewingGlass Sep 13, 2023
2b0a510
Fix canary
ChewingGlass Sep 13, 2023
3cdc782
Fix canary
ChewingGlass Sep 13, 2023
20eca0e
Fix canary
ChewingGlass Sep 13, 2023
430613d
Add verified builds and fix tests/other github actions
ChewingGlass Sep 13, 2023
d109531
Rm checkout from anchor build
ChewingGlass Sep 13, 2023
f7b5e02
Don't fail on toml exists
ChewingGlass Sep 13, 2023
37364b2
Merge master workflows
ChewingGlass Sep 13, 2023
95f1f1e
Fix type
ChewingGlass Sep 13, 2023
e9c43b4
Fix overfetch
ChewingGlass Sep 13, 2023
e92a511
Fix bug
ChewingGlass Sep 14, 2023
950648d
Improve build times
ChewingGlass Sep 14, 2023
ffda404
Bulk fetch asset proofs and assets in distributor oracle
ChewingGlass Sep 14, 2023
f244a08
chore(release): publish 0.3.0
ChewingGlass Sep 14, 2023
70ab6b5
Yarn lock change
ChewingGlass Sep 14, 2023
db45bff
More efficiency
ChewingGlass Sep 14, 2023
b0dadd1
More perf
ChewingGlass Sep 14, 2023
c70e7c9
Fix perf more
ChewingGlass Sep 14, 2023
fe0297f
Slightly more performant gMA
ChewingGlass Sep 14, 2023
5571e71
fix
ChewingGlass Sep 14, 2023
013351e
Fix performance from deserializing massive canopy
ChewingGlass Sep 15, 2023
bb96c92
More efficiency
ChewingGlass Sep 15, 2023
201108b
Cache well known merkle tree canopy depths so users don't have to lao…
ChewingGlass Sep 15, 2023
4e88eb5
Bugfix
ChewingGlass Sep 15, 2023
fad6007
Bugfix
ChewingGlass Sep 15, 2023
69e32b6
Update changelog
ChewingGlass Sep 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,24 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.2.22](https://github.com/helium/helium-program-library/compare/v0.2.21...v0.2.22) (2023-09-13)


### Features

* **#373:** replace clockwork ([#388](https://github.com/helium/helium-program-library/issues/388)) ([55033c7](https://github.com/helium/helium-program-library/commit/55033c718df08f41eb2f77d726d59d916ebbd677)), closes [#373](https://github.com/helium/helium-program-library/issues/373)
* **#376:** Replace lazy transactions markers with a bitmap to reclaim rent ([#380](https://github.com/helium/helium-program-library/issues/380)) ([a691257](https://github.com/helium/helium-program-library/commit/a6912570d4e3d89869cd13c5cfc8ce8c4355148e)), closes [#376](https://github.com/helium/helium-program-library/issues/376) [#376](https://github.com/helium/helium-program-library/issues/376)
* **#379:** Automate npm publish and fix devnet conflicting sqds txns ([#386](https://github.com/helium/helium-program-library/issues/386)) ([a4a3780](https://github.com/helium/helium-program-library/commit/a4a37806fefe82ca9a38f04c311f600ad1f7c36c))


### Reverts

* Revert "Add back mobile genesis fix for devnet" ([93e9a2c](https://github.com/helium/helium-program-library/commit/93e9a2c370ba77a49b02efc2590ddf3039465ed6))





## [0.2.15](https://github.com/helium/helium-program-library/compare/v0.2.14...v0.2.15) (2023-07-31)

**Note:** Version bump only for package helium-program-library
Expand Down
6 changes: 4 additions & 2 deletions lerna.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
{
"npmClient": "yarn",
"packages": ["packages/*"],
"packages": [
"packages/*"
],
"useWorkspaces": true,
"version": "0.2.21"
"version": "0.2.22"
}
8 changes: 8 additions & 0 deletions packages/account-fetch-cache-hooks/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.2.22](https://github.com/helium/helium-program-libary/compare/v0.2.21...v0.2.22) (2023-09-13)

**Note:** Version bump only for package @helium/account-fetch-cache-hooks





## [0.2.15](https://github.com/helium/helium-program-libary/compare/v0.2.14...v0.2.15) (2023-07-31)

**Note:** Version bump only for package @helium/account-fetch-cache-hooks
Expand Down
4 changes: 2 additions & 2 deletions packages/account-fetch-cache-hooks/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"registry": "https://registry.npmjs.org/"
},
"license": "Apache-2.0",
"version": "0.2.21",
"version": "0.2.22",
"description": "React hooks and context for account-fetch-cache",
"repository": {
"type": "git",
Expand All @@ -32,7 +32,7 @@
"prebuild": "npm run clean && npm run package"
},
"dependencies": {
"@helium/account-fetch-cache": "^0.2.21",
"@helium/account-fetch-cache": "^0.2.22",
"@solana/web3.js": "^1.78.4",
"react-async-hook": "^4.0.0"
},
Expand Down
8 changes: 8 additions & 0 deletions packages/account-fetch-cache/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.2.22](https://github.com/helium/helium-program-library/compare/v0.2.21...v0.2.22) (2023-09-13)

**Note:** Version bump only for package @helium/account-fetch-cache





## [0.2.15](https://github.com/helium/helium-program-library/compare/v0.2.14...v0.2.15) (2023-07-31)

**Note:** Version bump only for package @helium/account-fetch-cache
Expand Down
2 changes: 1 addition & 1 deletion packages/account-fetch-cache/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@helium/account-fetch-cache",
"version": "0.2.21",
"version": "0.2.22",
"description": "Solana account fetch cache to eliminate reduntant fetching, and batch fetches",
"publishConfig": {
"access": "public",
Expand Down
136 changes: 100 additions & 36 deletions packages/account-fetch-cache/src/accountFetchCache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,26 @@ function getWriteableAccounts(

let id = 0;

let singletons: Record<string, AccountFetchCache | undefined> = {};
export function getSingleton(conn: Connection): AccountFetchCache {
const commitment = conn.commitment || "confirmed"
const endp = conn.rpcEndpoint
if (!singletons[endp + commitment]) {
singletons[endp + commitment] = new AccountFetchCache({
connection: conn,
commitment,
})

}
return singletons[endp + commitment]!
}

function setSingleton(conn: Connection, cache: AccountFetchCache) {
const commitment = conn.commitment || "confirmed";
const endp = conn.rpcEndpoint;
singletons[endp + commitment] = cache;
}

export class AccountFetchCache {
connection: Connection;
chunkSize: number;
Expand All @@ -64,9 +84,7 @@ export class AccountFetchCache {
publicKey: PublicKey,
com?: Commitment
) => Promise<AccountInfo<Buffer> | null>;
oldSendTransaction: (
...args: any[]
) => Promise<string>;
oldSendTransaction: (...args: any[]) => Promise<string>;
oldSendRawTransaction: (
rawTransaction: Buffer | Uint8Array | Array<number>,
options?: SendOptions
Expand Down Expand Up @@ -101,8 +119,7 @@ export class AccountFetchCache {
);

this.oldSendTransaction = connection.sendTransaction.bind(connection);
this.oldSendRawTransaction =
connection.sendRawTransaction.bind(connection);
this.oldSendRawTransaction = connection.sendRawTransaction.bind(connection);

const self = this;

Expand Down Expand Up @@ -152,12 +169,14 @@ export class AccountFetchCache {
return self.requeryMissing(instructions);
})
.catch(console.error);
} catch(e: any) {
} catch (e: any) {
// TODO: handle transaction v2
}

return result;
};

setSingleton(connection, this)
}

async requeryMissing(instructions: TransactionInstruction[]) {
Expand Down Expand Up @@ -214,7 +233,10 @@ export class AccountFetchCache {
this.currentBatch = new Set(); // Erase current batch from state, so we can fetch multiple at a time
try {
const keys = Array.from(currentBatch);
const array = await this.connection.getMultipleAccountsInfo(keys.map(b => new PublicKey(b)), this.commitment)
const array = await this.connection.getMultipleAccountsInfo(
keys.map((b) => new PublicKey(b)),
this.commitment
);
keys.forEach((key, index) => {
const callback = this.pendingCallbacks.get(key);
callback && callback(array[index], null);
Expand Down Expand Up @@ -363,6 +385,48 @@ export class AccountFetchCache {
return query;
}

async searchMultiple<T>(
pubKeys: PublicKey[],
parser?: AccountParser<T> | undefined,
isStatic: Boolean = false, // optimization, set if the data will never change
forceRequery = false
): Promise<(ParsedAccountBase<T> | undefined)[]> {
pubKeys.forEach(key => {
this.registerParser(key, parser);
const address = key.toBase58();
if (isStatic) {
this.statics.add(address);
} else if (this.statics.has(address)) {
this.statics.delete(address); // If trying to use this as not static, need to rm it from the statics list.
}

if (forceRequery || !this.genericCache.has(address)) {
this.currentBatch.add(address);
}
})

const searched = new Set(pubKeys.map((p) => p.toBase58()));
// Force a batch fetch to resolve all accounts
const { keys, array } = await this.fetchBatch();
// Cache these results if they aren't going to change
if (isStatic) {
keys.forEach((key, index) => {
if (searched.has(key)) {
const item = array[index];
if (item) {
this.genericCache.set(
key,
this.getParsed(key, item, parser) || null
);
}
}
});
}


return pubKeys.map(key => this.genericCache.get(key.toBase58()) as ParsedAccountBase<T> | undefined);
}

onAccountChange<T>(
key: PublicKey,
parser: AccountParser<T> | undefined,
Expand All @@ -373,7 +437,7 @@ export class AccountFetchCache {
const address = key.toBase58();
this.updateCache(address, parsed || null);
} catch (e: any) {
console.error("accountFetchCache", "Failed to update account", e)
console.error("accountFetchCache", "Failed to update account", e);
}
}

Expand All @@ -391,43 +455,43 @@ export class AccountFetchCache {
// Only websocket watch accounts that exist
// Don't recreate listeners
// xNFT doesn't support onAccountChange, so we have to make a new usable connection.
if (!this.accountChangeListeners.has(address)) {
try {
if (!this.accountChangeListeners.has(address)) {
try {
this.accountChangeListeners.set(
address,
this.connection.onAccountChange(
id,
(account) => this.onAccountChange(id, undefined, account),
this.commitment
)
);
} catch (e: any) {
if (e.toString().includes("not implemented")) {
// @ts-ignore
this.usableConnection =
// @ts-ignore
this.usableConnection ||
new Connection(
// @ts-ignore
this.connection._rpcEndpoint,
this.commitment
);
this.accountChangeListeners.set(
address,
this.connection.onAccountChange(
// @ts-ignore
this.usableConnection.onAccountChange(
id,
// @ts-ignore
(account) => this.onAccountChange(id, undefined, account),
this.commitment
)
);
} catch (e: any) {
if (e.toString().includes("not implemented")) {
// @ts-ignore
this.usableConnection =
// @ts-ignore
this.usableConnection ||
new Connection(
// @ts-ignore
this.connection._rpcEndpoint,
this.commitment
);
this.accountChangeListeners.set(
address,
// @ts-ignore
this.usableConnection.onAccountChange(
id,
// @ts-ignore
(account) => this.onAccountChange(id, undefined, account),
this.commitment
)
);
} else {
console.error(e)
throw e
}
} else {
console.error(e);
throw e;
}
}
}
} else if (!exists) {
// Poll accounts that don't exist
this.missingAccounts.set(
Expand Down
8 changes: 8 additions & 0 deletions packages/account-postgres-sink-service/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.2.22](https://github.com/helium/helium-program-libary/compare/v0.2.21...v0.2.22) (2023-09-13)

**Note:** Version bump only for package @helium/account-postgres-sink-service





## [0.2.15](https://github.com/helium/helium-program-libary/compare/v0.2.14...v0.2.15) (2023-07-31)

**Note:** Version bump only for package @helium/account-postgres-sink-service
Expand Down
4 changes: 2 additions & 2 deletions packages/account-postgres-sink-service/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"registry": "https://registry.npmjs.org/"
},
"license": "Apache-2.0",
"version": "0.2.21",
"version": "0.2.22",
"description": "Sync account data to postgres",
"repository": {
"type": "git",
Expand Down Expand Up @@ -34,7 +34,7 @@
"dependencies": {
"@coral-xyz/anchor": "^0.26.0",
"@fastify/cors": "^8.1.1",
"@helium/account-fetch-cache": "^0.2.21",
"@helium/account-fetch-cache": "^0.2.22",
"@metaplex-foundation/mpl-token-metadata": "^2.10.0",
"@solana/web3.js": "^1.78.4",
"aws-sdk": "^2.1344.0",
Expand Down
8 changes: 8 additions & 0 deletions packages/active-device-oracle/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.2.22](https://github.com/helium/helium-program-libary/compare/v0.2.21...v0.2.22) (2023-09-13)

**Note:** Version bump only for package @helium/active-device-oracle





## [0.2.15](https://github.com/helium/helium-program-libary/compare/v0.2.14...v0.2.15) (2023-07-31)

**Note:** Version bump only for package @helium/active-device-oracle
Expand Down
2 changes: 1 addition & 1 deletion packages/active-device-oracle/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"registry": "https://registry.npmjs.org/"
},
"license": "Apache-2.0",
"version": "0.2.21",
"version": "0.2.22",
"description": "Oracle of helium active devices",
"repository": {
"type": "git",
Expand Down
8 changes: 8 additions & 0 deletions packages/anchor-resolvers/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.2.22](https://github.com/helium/helium-program-library/compare/v0.2.21...v0.2.22) (2023-09-13)

**Note:** Version bump only for package @helium/anchor-resolvers





## [0.2.15](https://github.com/helium/helium-program-library/compare/v0.2.14...v0.2.15) (2023-07-31)

**Note:** Version bump only for package @helium/anchor-resolvers
Expand Down
2 changes: 1 addition & 1 deletion packages/anchor-resolvers/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@helium/anchor-resolvers",
"version": "0.2.21",
"version": "0.2.22",
"description": "Wrappers around anchor custom resolvers to make composition easier",
"publishConfig": {
"access": "public",
Expand Down
8 changes: 8 additions & 0 deletions packages/circuit-breaker-sdk/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.2.22](https://github.com/helium/helium-program-libary/compare/v0.2.21...v0.2.22) (2023-09-13)

**Note:** Version bump only for package @helium/circuit-breaker-sdk





## [0.2.15](https://github.com/helium/helium-program-libary/compare/v0.2.14...v0.2.15) (2023-07-31)

**Note:** Version bump only for package @helium/circuit-breaker-sdk
Expand Down
Loading
Loading