Skip to content

Commit

Permalink
Remove token registry dependency (#281)
Browse files Browse the repository at this point in the history 
This PR removes and replaces the `@solana/spl-token-registry`
dependency, which was by far the largest dependency and was pulling in
an >3MB outdated JSON file on every page. It should improve performance
across the board, especially on slower networks. Token data will also be
much more up to date, and tokens created since we deprecated the token
registry will be represented correctly.

The main replacement for this dependency is [Solflare's Unified Token
List (UTL) SDK](https://github.com/solflare-wallet/utl-sdk). This is a
typescript SDK that provides information about a token. It's got great
performance and covers most use cases we have really well.

It has some tradeoffs:

- It's quite a heavy dependency, mostly because of its
`metaplex-foundation/js` dependency which allows it to fetch on-chain
data for tokens that aren't in its API/CDN already. For this reason I've
used the underlying API directly for the search bar, to avoid pulling
the SDK into the root layout. This is actually the same behaviour as the
SDK has for search anyway, since falling back to an on-chain query can't
be done for search.
- It doesn't have all the per-token data from the old token registry,
specifically for extensions it only has the coingecko one, and is
missing eg bridge contracts and websites. We only use these on the token
address page. Longer term we might want to remove these since we don't
have a way to get this data for new tokens. But for now I'm using a CDN
version of our last token list to fetch the token info including these
legacy fields. For tokens that aren't in that list I use the SDK as a
fallback, and just return the data without those extensions.

Other than these cases, we now use the SDK for all token info. 

Some specific implementation decisions:

- The address/[address] page fetches token info once (using the legacy
CDN, then API fallback), and uses prop drilling for child components.
- `useAccountOwnedTokens` now returns tokens with the fetched
name/logo/symbol for each, fetched from the UTL SDK.
- The `<Address>` component used to search the token registry in all
cases, like it does with programs etc. This is no longer the case since
we no longer have the entire token list on every page. A new prop
`tokenListInfo` can be used to pass known token info in. This is used
for example when rendering the user's owned tokens. Alternatively
`fetchTokenLabelInfo` can be used to asynchronously fetch the token for
the address, and update the address to display it after fetching. This
should be used for addresses that might be a token, for example
transaction accounts. This is the equivalent of `useMetadata` which
fetches NFT metadata.
- I've used SWR in a few places where we fetch token info as an
enhancement, usually just to display the symbol for a token as a suffix
for a count, eg 5 USDC. In these cases I use the loading state of SWR to
distinguish between data not fetched yet (we display no suffix) and no
data available (we typically add the 'tokens' suffix)

The result of all this is that the huge tokenlist file which dominated
our bundle is gone:

Master:
<img width="2386" alt="master-all-min"
src="https://github.com/solana-labs/explorer/assets/1711350/335135f5-7750-4962-8149-5fe542dd3cbd">

This branch:
<img width="2387" alt="rtr-all-min"
src="https://github.com/solana-labs/explorer/assets/1711350/da3b5c42-65fd-4121-8cf7-e2a8bc858047">

Looking at just the layout shared by all pages, the effect is even more
drastic.

Master:
<img width="2388" alt="master-layout-min"
src="https://github.com/solana-labs/explorer/assets/1711350/3e843dd0-7fc4-4c21-9a57-36b95bb393d8">

This branch:
<img width="2388" alt="rtr-layout-min"
src="https://github.com/solana-labs/explorer/assets/1711350/748f6fdc-d16f-4600-b840-6b72848e3ff6">

Closes #201

---------

Co-authored-by: steveluscher <[email protected]>
  • Loading branch information
@mcintyre94 @steveluscher
mcintyre94 and steveluscher authored Jul 31, 2023
1 parent 9639bef commit 450736d
Show file tree
Hide file tree
Showing 23 changed files with 2,783 additions and 375 deletions.
52 changes: 35 additions & 17 deletions app/address/[address]/layout.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,17 @@ import isMetaplexNFT from '@providers/accounts/utils/isMetaplexNFT';
import { useAnchorProgram } from '@providers/anchor';
import { CacheEntry, FetchStatus } from '@providers/cache';
import { useCluster } from '@providers/cluster';
import { useTokenRegistry } from '@providers/token-registry';
import { PROGRAM_ID as ACCOUNT_COMPRESSION_ID } from '@solana/spl-account-compression';
import { PublicKey } from '@solana/web3.js';
import { ClusterStatus } from '@utils/cluster';
import { Cluster, ClusterStatus } from '@utils/cluster';
import { FEATURE_PROGRAM_ID } from '@utils/parseFeatureAccount';
import { useClusterPath } from '@utils/url';
import Link from 'next/link';
import { redirect, useSelectedLayoutSegment } from 'next/navigation';
import React, { PropsWithChildren } from 'react';
import useSWRImmutable from 'swr/immutable';

import { FullLegacyTokenInfo, getFullTokenInfo } from '@/app/utils/token-info';

const IDENTICON_WIDTH = 64;

Expand Down Expand Up @@ -143,10 +145,15 @@ const TOKEN_TABS_HIDDEN = ['spl-token:mint', 'config', 'vote', 'sysvar', 'config

type Props = PropsWithChildren<{ params: { address: string } }>;

async function fetchFullTokenInfo([_, pubkey, cluster, url]: ['get-full-token-info', string, Cluster, string]) {
return await getFullTokenInfo(new PublicKey(pubkey), cluster, url);
}

function AddressLayoutInner({ children, params: { address } }: Props) {
const fetchAccount = useFetchAccountInfo();
const { status } = useCluster();
const { status, cluster, url } = useCluster();
const info = useAccountInfo(address);

let pubkey: PublicKey | undefined;

try {
Expand All @@ -155,6 +162,14 @@ function AddressLayoutInner({ children, params: { address } }: Props) {
/* empty */
}

const infoStatus = info?.status;
const infoProgram = info?.data?.data.parsed?.program;

const { data: fullTokenInfo, isLoading: isFullTokenInfoLoading } = useSWRImmutable(
infoStatus === FetchStatus.Fetched && infoProgram === "spl-token" && pubkey ? ['get-full-token-info', address, cluster, url] : null,
fetchFullTokenInfo
);

// Fetch account on load
React.useEffect(() => {
if (!info && status === ClusterStatus.Connected && pubkey) {
Expand All @@ -166,13 +181,13 @@ function AddressLayoutInner({ children, params: { address } }: Props) {
<div className="container mt-n3">
<div className="header">
<div className="header-body">
<AccountHeader address={address} account={info?.data} />
<AccountHeader address={address} account={info?.data} tokenInfo={fullTokenInfo} isTokenInfoLoading={isFullTokenInfoLoading} />
</div>
</div>
{!pubkey ? (
<ErrorCard text={`Address "${address}" is not valid`} />
) : (
<DetailsSections info={info} pubkey={pubkey}>
<DetailsSections info={info} pubkey={pubkey} tokenInfo={fullTokenInfo} isTokenInfoLoading={isFullTokenInfoLoading}>
{children}
</DetailsSections>
)}
Expand All @@ -188,10 +203,9 @@ export default function AddressLayout({ children, params }: Props) {
);
}

function AccountHeader({ address, account }: { address: string; account?: Account }) {
const { tokenRegistry } = useTokenRegistry();
const tokenDetails = tokenRegistry.get(address);
function AccountHeader({ address, account, tokenInfo, isTokenInfoLoading }: { address: string; account?: Account, tokenInfo?: FullLegacyTokenInfo, isTokenInfoLoading: boolean }) {
const mintInfo = useMintAccountInfo(address);

const parsedData = account?.data.parsed;
const isToken = parsedData?.program === 'spl-token' && parsedData?.parsed.type === 'mint';

Expand All @@ -204,21 +218,21 @@ function AccountHeader({ address, account }: { address: string; account?: Accoun
return <NFTokenAccountHeader account={account} />;
}

if (isToken) {
if (isToken && !isTokenInfoLoading) {
let token;
let unverified = false;

// Fall back to legacy token list when there is stub metadata (blank uri), updatable by default by the mint authority
if (!parsedData?.nftData?.metadata.data.uri && tokenDetails) {
token = tokenDetails;
if (!parsedData?.nftData?.metadata.data.uri && tokenInfo) {
token = tokenInfo;
} else if (parsedData?.nftData) {
token = {
logoURI: parsedData?.nftData?.json?.image,
name: parsedData?.nftData?.json?.name ?? parsedData?.nftData.metadata.data.name,
};
unverified = true;
} else if (tokenDetails) {
token = tokenDetails;
} else if (tokenInfo) {
token = tokenInfo;
}

return (
Expand Down Expand Up @@ -271,16 +285,20 @@ function DetailsSections({
pubkey,
tab,
info,
tokenInfo,
isTokenInfoLoading
}: {
children: React.ReactNode;
pubkey: PublicKey;
tab?: string;
info?: CacheEntry<Account>;
tokenInfo?: FullLegacyTokenInfo;
isTokenInfoLoading: boolean;
}) {
const fetchAccount = useFetchAccountInfo();
const address = pubkey.toBase58();

if (!info || info.status === FetchStatus.Fetching) {
if (!info || info.status === FetchStatus.Fetching || isTokenInfoLoading) {
return <LoadingCard />;
} else if (info.status === FetchStatus.FetchFailed || info.data?.lamports === undefined) {
return <ErrorCard retry={() => fetchAccount(pubkey, 'parsed')} text="Fetch Failed" />;
Expand All @@ -296,13 +314,13 @@ function DetailsSections({
return (
<>
{FLAGGED_ACCOUNTS_WARNING[address] ?? null}
<InfoSection account={account} />
<InfoSection account={account} tokenInfo={tokenInfo} />
<MoreSection tabs={tabComponents.map(({ component }) => component)}>{children}</MoreSection>
</>
);
}

function InfoSection({ account }: { account: Account }) {
function InfoSection({ account, tokenInfo }: { account: Account, tokenInfo?: FullLegacyTokenInfo }) {
const parsedData = account.data.parsed;
const rawData = account.data.raw;

Expand All @@ -326,7 +344,7 @@ function InfoSection({ account }: { account: Account }) {
} else if (account.owner.toBase58() === NFTOKEN_ADDRESS) {
return <NFTokenAccountSection account={account} />;
} else if (parsedData && parsedData.program === 'spl-token') {
return <TokenAccountSection account={account} tokenAccount={parsedData.parsed} />;
return <TokenAccountSection account={account} tokenAccount={parsedData.parsed} tokenInfo={tokenInfo} />;
} else if (parsedData && parsedData.program === 'nonce') {
return <NonceAccountSection account={account} nonceAccount={parsedData.parsed} />;
} else if (parsedData && parsedData.program === 'vote') {
Expand Down
136 changes: 47 additions & 89 deletions app/components/SearchBar.tsx
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
'use client';

import { useCluster } from '@providers/cluster';
import { useTokenRegistry } from '@providers/token-registry';
import { TokenInfoMap } from '@solana/spl-token-registry';
import { Cluster } from '@utils/cluster';
import bs58 from 'bs58';
import { useRouter, useSearchParams } from 'next/navigation';
import React, { useId } from 'react';
import { Search } from 'react-feather';
import Select, { ActionMeta, InputActionMeta, ValueType } from 'react-select';
import { ActionMeta, InputActionMeta, ValueType } from 'react-select';
import AsyncSelect from 'react-select/async';

import { FetchedDomainInfo } from '../api/domain-info/[domain]/route';
import { LOADER_IDS, LoaderName, PROGRAM_INFO_BY_ID, SPECIAL_IDS, SYSVAR_IDS } from '../utils/programs';
import { searchTokens } from '../utils/token-search';

interface SearchOptions {
label: string;
Expand All @@ -28,13 +28,8 @@ const hasDomainSyntax = (value: string) => {

export function SearchBar() {
const [search, setSearch] = React.useState('');
const searchRef = React.useRef('');
const [searchOptions, setSearchOptions] = React.useState<SearchOptions[]>([]);
const [loadingSearch, setLoadingSearch] = React.useState<boolean>(false);
const [loadingSearchMessage, setLoadingSearchMessage] = React.useState<string>('loading...');
const selectRef = React.useRef<Select<any> | null>(null);
const selectRef = React.useRef<AsyncSelect<any> | null>(null);
const router = useRouter();
const { tokenRegistry } = useTokenRegistry();
const { cluster, clusterInfo } = useCluster();
const searchParams = useSearchParams();
const onChange = ({ pathname }: ValueType<any, false>, meta: ActionMeta<any>) => {
Expand All @@ -51,53 +46,30 @@ export function SearchBar() {
}
};

React.useEffect(() => {
searchRef.current = search;
setLoadingSearchMessage('Loading...');
setLoadingSearch(true);
async function performSearch(search: string): Promise<SearchOptions[]> {
const localOptions = buildOptions(search, cluster, clusterInfo?.epochInfo.epoch);
const tokenOptions = await buildTokenOptions(search, cluster);
const tokenOptionsAppendable = tokenOptions ? [tokenOptions] : [];
const domainOptions = hasDomainSyntax(search) && cluster === Cluster.MainnetBeta ?
await buildDomainOptions(search) ?? [] : [];

// builds and sets local search output
const options = buildOptions(search, cluster, tokenRegistry, clusterInfo?.epochInfo.epoch);

setSearchOptions(options);

// checking for non local search output
if (hasDomainSyntax(search) && cluster === Cluster.MainnetBeta) {
// if search input is a potential domain we continue the loading state
domainSearch(options);
} else {
// if search input is not a potential domain we can conclude the search has finished
setLoadingSearch(false);
}

// eslint-disable-next-line react-hooks/exhaustive-deps
}, [search]);

// appends domain lookup results to the local search state
const domainSearch = async (options: SearchOptions[]) => {
setLoadingSearchMessage('Looking up domain...');
const searchTerm = search;
const updatedOptions = await buildDomainOptions(search, options);
if (searchRef.current === searchTerm) {
setSearchOptions(updatedOptions);
// after attempting to fetch the domain name we can conclude the loading state
setLoadingSearch(false);
setLoadingSearchMessage('Loading...');
}
};
return [...localOptions, ...tokenOptionsAppendable, ...domainOptions];
}

const resetValue = '' as any;
return (
<div className="container my-4">
<div className="row align-items-center">
<div className="col">
<Select
<AsyncSelect
cacheOptions
defaultOptions
loadOptions={performSearch}
autoFocus
inputId={useId()}
ref={ref => (selectRef.current = ref)}
options={searchOptions}
noOptionsMessage={() => 'No Results'}
loadingMessage={() => loadingSearchMessage}
loadingMessage={() => 'loading...'}
placeholder="Search for blocks, accounts, transactions, programs, and tokens"
value={resetValue}
inputValue={search}
Expand All @@ -112,7 +84,8 @@ export function SearchBar() {
onInputChange={onInputChange}
components={{ DropdownIndicator }}
classNamePrefix="search-bar"
isLoading={loadingSearch}
/* workaround for https://github.com/JedWatson/react-select/issues/5714 */
onFocus={() => { selectRef.current?.handleInputChange(search, { action: 'set-value' }) }}
/>
</div>
</div>
Expand Down Expand Up @@ -194,59 +167,49 @@ function buildSpecialOptions(search: string) {
}
}

function buildTokenOptions(search: string, cluster: Cluster, tokenRegistry: TokenInfoMap) {
const matchedTokens = Array.from(tokenRegistry.entries()).filter(([address, details]) => {
const searchLower = search.toLowerCase();
return (
details.name.toLowerCase().includes(searchLower) ||
details.symbol.toLowerCase().includes(searchLower) ||
address.includes(search)
);
});
async function buildTokenOptions(search: string, cluster: Cluster): Promise<SearchOptions | undefined> {
const matchedTokens = await searchTokens(search, cluster);

if (matchedTokens.length > 0) {
return {
label: 'Tokens',
options: matchedTokens.slice(0, 10).map(([id, details]) => ({
label: details.name,
pathname: '/address/' + id,
value: [details.name, details.symbol, id],
})),
options: matchedTokens
};
}
}

async function buildDomainOptions(search: string, options: SearchOptions[]) {
async function buildDomainOptions(search: string) {
const domainInfoResponse = await fetch(`/api/domain-info/${search}`);
const domainInfo = await domainInfoResponse.json() as FetchedDomainInfo;
const updatedOptions: SearchOptions[] = [...options];

if (domainInfo && domainInfo.owner && domainInfo.address) {
updatedOptions.push({
label: 'Domain Owner',
options: [
{
label: domainInfo.owner,
pathname: '/address/' + domainInfo.owner,
value: [search],
},
],
});
updatedOptions.push({
label: 'Name Service Account',
options: [
{
label: search,
pathname: '/address/' + domainInfo.address,
value: [search],
},
],
});

return [
{
label: 'Domain Owner',
options: [
{
label: domainInfo.owner,
pathname: '/address/' + domainInfo.owner,
value: [search],
},
],
},
{
label: 'Name Service Account',
options: [
{
label: search,
pathname: '/address/' + domainInfo.address,
value: [search],
},
],
}];
}
return updatedOptions;
}

// builds local search options
function buildOptions(rawSearch: string, cluster: Cluster, tokenRegistry: TokenInfoMap, currentEpoch?: bigint) {
function buildOptions(rawSearch: string, cluster: Cluster, currentEpoch?: bigint) {
const search = rawSearch.trim();
if (search.length === 0) return [];

Expand All @@ -272,11 +235,6 @@ function buildOptions(rawSearch: string, cluster: Cluster, tokenRegistry: TokenI
options.push(specialOptions);
}

const tokenOptions = buildTokenOptions(search, cluster, tokenRegistry);
if (tokenOptions) {
options.push(tokenOptions);
}

if (!isNaN(Number(search))) {
options.push({
label: 'Block',
Expand Down
Loading

1 comment on commit 450736d

@vercel
Copy link

@vercel vercel bot commented on 450736d Jul 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successfully deployed to the following URLs:

explorer – ./

explorer-git-master-solana-labs.vercel.app
explorer.solana.com
explorer-solana-labs.vercel.app

Please sign in to comment.