diff --git a/content/week-19/index.md b/content/week-19/index.md new file mode 100644 index 0000000..49efb9e --- /dev/null +++ b/content/week-19/index.md @@ -0,0 +1,296 @@ ++++ +title="WTF News (Week 19)" +description="Are you ready to rumble?" +date=2023-08-18 + +[taxonomies] +tags = ["Q2", "August"] +categories = ["status update", "test report"] ++++ + +Hello and welcome to issue number nine of WTF News, our biweekly status update. + +Over the last two weeks, we ran a handful of wallets through their paces, and +today we can reveal the results! Spoiler alert, but there were more issues in +our tests than in the best scoring wallets. + +First, the regular status update! + +## Call to Action + +### CI/Build Pipelines + +What good is a testing framework if it isn't used? Get in touch if you want to +integrate WTF into your wallet's testing pipeline. We'll even build the +automation ourselves! + +### Testing the Tests + +We want to expand our tests for WTF, to catch issues with the framework itself. +To that end, if you work on a wallet that works offline (aside from the RPC +provider) and want to try out WTF, get in touch with us! + +## Code Changes + + * [Upgraded to viem 1.6][viem]! + +## Bug Fixes + + * [Use smaller integers to make ganache happy][smallint]. + * [Send WebSocket pings to keep the connection alive][ping]. + * [Use `from` instead of `account` in `eth_signTransaction`][from]. + * [Use hex instead of decimal in `eth_signTransaction`][hex]. + * [Convert `BigInt` to hex in `eth_getProof`][proof]. + * [Actually put `jsonrpc` in responses to the wallet][jsonrpc]. + * [Handle race condition when falling back to adding a chain manually][chain]. + +## Upstream Issues + + * [`eth_call` unclear when `Block` parameter is omitted][block]. + +## Test Report + +### Summary + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+

+ Wallet Test Framework – + + f999503092a10f2b409b5cc733db065c124e56e5 + +
+ + https://github.com/wallet-test-framework/framework + +

+

+ Not a statement of a wallet’s quality or correctness. +

+
WalletVersionConnectionPassing TestsPass %PlatformsBrowserNotes
Brave WalletEIP-308561100%BrowserBrv 1.57.42
Taho0.46.1EIP-308561100%ExtensionChr 115.0.5790.170Had to impersonate chainlist.org
Frame0.6.7EIP-30856098%DesktopFf 102.14.0esr (64-bit)Missing signTransaction
Enkrypt1.24.0Manual5997%ExtensionFf 102.14.0esr (64-bit)Attempted EIP-3085, but popup doesn’t show RPC url
Coinbase3.30.2EIP-30854879%ExtensionChr 115.0.5790.170Missing hex prefix in logs
Metamask10.34.0EIP-30854879%ExtensionFf 102.14.0esr (64-bit)
OKX2.58.1EIP-30851830%ExtensionChr 115.0.5790.170Tests stopped running somewhere around getFilterChanges
TokenPocket1.1.19EIP-30851525%ExtensionChr 115.0.5790.170Tests stopped running somewhere around getFilterChanges
Argent X5.7.2Can’t Connect00%ExtensionChr 115.0.5790.170No window.ethereum
Bitkeep1.4.8Can’t Connect00%ExtensionChr 115.0.5790.170Attempted EIP-3085, and Manual, but got system error
Sequence2.0.4Can’t Connect00%ExtensionChr 115.0.5790.170Cannot add custom chains
Exodus23.8.14Can’t Connect00%ExtensionChr 115.0.5790.170Cannot add custom chains
+
+ +The astute reader may notice that there are in fact 62 tests in the repository +at that commit. Our `eth_getProof` test is broken, and we are excluding it from +this report. + +### Disclaimer + +Our intent is not to claim that certain wallets are better than others, and a +low score here is as likely to indicate that our tests have problems as it +is to indicate bugs in a wallet. We're very interested in improving scores +across the board, so we're happy to accept contributions to fix tests and just +as happy to help debug. + +### Wallet Selection + +Very little of this report is scientific/rigorous, and the wallet selection +process was no exception. We chose a few wallets we're familiar with, and a +small sampling of random wallets from the [Find an Ethereum Wallet][find-wallet] +list. + +Because Wallet Test Framework only supports `window.ethereum` so far, we're +limited to wallets that provide it; that pretty much excludes mobile wallets for +the time being. Our [architecture] requires connecting to a custom chain, so +wallets that don't support custom chains cannot be tested (labelled with `Can't +Connect` in the table.) + +For future reports, get in touch if you're interested in having your results +published (or if you'd prefer we didn't mention you.) + +### Discussion + +Congratulations to [Brave] and [Taho] for getting perfect scores! Considering +this was our first time trying them out with the Framework (we've been using +[Frame] to develop), this was quite surprising. + +#### `eth_signTransaction` + +This is a contentious endpoint that we've [discussed previously](../week-13). +Very few wallets support `eth_signTransaction`, mostly by choice. + +Interestingly, this was Frame's only failure: instead of signing in the wallet, +Frame passes these requests through to the RPC provider. + +#### Caching + +> There are only two hard things in Computer Science: cache invalidation and +> naming things. _Phil Karlton_ + +Caching is hard. [We've been bitten by it](../week-07/). In a space where you +have to pay for every API request, but your users expect a free wallet +experience, caching is a necessary evil. + +We suspect, but haven't verified for all wallets, that many of the test failures +are caused by stale values cached by wallets. + +In the case of `eth_call`, we have verified that MetaMask specifies an old block +number, causing a call into a freshly deployed contract to fail. This is +unlikely to cause major issues on a real network with many seconds between +blocks, but the tests mine blocks much more frequently, triggering the issue. +Regardless, we're following up with their team to see what we can do. + +#### Post-merge Endpoints + +Some of the newer endpoints like `eth_maxPriorityFeePerGas` aren't yet supported +in all wallets. Expect these tests to start passing as they are implemented. + +### Future Plans + +We'll continue adding tests, and we'll try to run through this report somewhat +regularly. That said, this is entirely a manual process. Once wallets start +running WTF as part of their continuous integration processes, we can make some +pretty graphs, include more wallets, and make reports more often. + +In the meantime, [join us on Discord](https://allwallet.dev/)! + +[viem]: https://github.com/wallet-test-framework/framework/commit/b55b9729be2b82bd113ab821697dc77fc1e876b1 +[smallint]: https://github.com/wallet-test-framework/framework/commit/f999503092a10f2b409b5cc733db065c124e56e5 +[ping]: https://github.com/wallet-test-framework/framework/commit/801d8f9ca0e081a130e04f1ba91121fc29de84c2 +[from]: https://github.com/wallet-test-framework/framework/commit/a86d4284e9fbada0653105a08deab7b9f88569c1 +[hex]: https://github.com/wallet-test-framework/framework/commit/759cbe74604ac56d1f75a367bf285423265625d6 +[proof]: https://github.com/wallet-test-framework/framework/commit/52bb9661d135719e6c092a9e87dcd21e577e125c +[jsonrpc]: https://github.com/wallet-test-framework/framework/commit/aab09bd782a75f513d2e8ab6678a384ba32b15ce +[chain]: https://github.com/wallet-test-framework/framework/commit/22a116d8754f379c44745d7231e14a1442f9f1fd +[find-wallet]: https://ethereum.org/en/wallets/find-wallet/ +[block]: https://github.com/ethereum/execution-apis/issues/461 +[Brave]: https://brave.com/wallet/ +[Taho]: https://taho.xyz/ +[Frame]: https://frame.sh/ +[architecture]: ../kick-off/ diff --git a/static/reports.css b/static/reports.css new file mode 100644 index 0000000..64b07b7 --- /dev/null +++ b/static/reports.css @@ -0,0 +1,48 @@ +.report-holder { + width: 100%; + overflow-x: scroll; +} + +table.report { + border-collapse: collapse; + min-width: 100%; + width: max-content; + caption-side: bottom; +} + +table.report > caption { + text-align: left; +} + +table.report td, table.report th { + padding: 0 2em; +} + +table.report > thead > tr > th { + background-color: #999999; + color: white; +} + +table.report tr + tr, table.report > tbody > tr:first-child { + border-top: 1px solid white; +} + +table.report tr.platinum { + background-color: #e5e4e2; +} + +table.report tr.gold { + background-color: #fff600; +} + +table.report tr.silver { + background-color: #a9a9a9; +} + +table.report tr.bronze { + background-color: #daa520; +} + +table.report tr.garbage { + background-color: #999966; +} diff --git a/templates/index.html b/templates/index.html new file mode 100644 index 0000000..53083fc --- /dev/null +++ b/templates/index.html @@ -0,0 +1,6 @@ +{% extends "zhuia/templates/index.html" %} + +{% block css %} +{{ super() }} + +{% endblock %}