New ABI Package

[ryandotsmith]

A Go package for encoding/decoding ABI data.

Click here for a primer on ABI encoding.

The most common way for a Go project to deal with ABI encoded data is to use Geth's ABI packages. But today I’m excited to announce another option: Index Supply’s ABI package. Here are a few key differences between Geth and Index Supply:

1. Reduced memory usage
2. More functionality with a simpler interface
3. Smaller implementation

Our goal is to decode logs from common contracts in ~1000 ns which enables us to process millions of logs per second. This is important for Index Supply because we are building a system that can quickly backfill new indexes.

Reduced Memory Usage

We avoided using Go's reflection package and paid close attention to allocations. Our benchmarks indicate 0 allocations occur during a decode operation once the package has been initialized. Here are some benchmarks comparing Geth and Index Supply:

ERC721 Transfer

Geth	Index Supply

Index Supply’s ABI package allocates 0 memory because for an ERC721 Transfer, all of the data is indexed, meaning there is no ABI encoded data to decode!

Seaport OrderFulfilled

Geth	Index Supply

	Time	Allocations	Data
Geth	7.0k ns	124	6.7 kb
Index Supply	1.1k ns	2	0.5 kb

The 2 allocations in the Index Supply benchmark occur in the generated code. We have to allocate some slices to hold the nested tuple arrays inside the Seaport event structure. It's possible that these allocations can be factored out as well. You can still achieve 0 allocations if you use the abi Decode package directly.

More Functionality With a Simpler Interface

The primary way to interact with the abi package is by using abi.Encode, abi.Decode, and schema.Parse. For example:

t := schema.Parse("(uint8, bytes)")
d := []byte{/* ... */}
item, n, err := abi.Decode(d, t) // n = number of bytes read from d
x := item.At(0).Uint8()
y := item.At(1).Bytes()

Of course this code can be generated based on an ABI JSON file using our genabi command:

Smaller Implementation

Having a concise implementation makes debugging, feature addition, and performance work a pleasant experience. The core encoding and decoding functions are ~200 LOC and are found in a single file: Encode & Decode.

Our ABI package has a not-so-bad test corpus as well.

Owning our stack

In addition to the form factor and performance improvements we made, another big motivation for the rewrite is that we need to own our stack. Index Supply is in the business of indexing ethereum data –most of which is ABI encoded. As we have gone live with our first customer, we have already seen ROI from our ability to quickly add features and diagnose performance issues. We won't reinvent everything, but investing in the core parts of our stack will be worth our while –and it's a lot of fun!

Roadmap

We are on a mission to create an Ethereum node that is purpose built for indexing operations. This is a large undertaking and will require years of development. That being said, we aren’t developing in the dark. We are finding sub-projects that help teams in the short term that also fit into our long term vision. ABI decoding is something that a lot of teams currently deal with and also something that is necessary for our node.

If you have an idea for a project please reach out!

Examples

Parsing a Seaport OrderFulfilled Log

package main

import (
	"encoding/hex"
	"fmt"

	"github.com/indexsupply/contrib/abis/seaport"
	"github.com/indexsupply/x/abi"
)

func h2b(s string) []byte { b, _ := hex.DecodeString(s); return b }

func main() {
	log := abi.Log{
		Topics: [][32]byte{
			[32]byte(h2b("9d9af8e38d66c62e2c12f0225249fd9d721c54b83f48d9352c97c6cacdcb6f31")),
			[32]byte(h2b("00000000000000000000000074dcc6d44d4c8486ace698bbb776d8ccd985cdfa")),
			[32]byte(h2b("000000000000000000000000004c00500000ad104d7dbd00e3ae0a5c00560c00")),
		},
		Data: h2b(`6829f00e466642492794a76fe191447432a19eb75b9ca5aa472f91143ba77f0f00000000000000000000000030614bc1e56bf9f30f2dda7898b520a20b560ef30000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000012000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000002000000000000000000000000496aeac4ae585b68cee97dadc003662f7b808a1c00000000000000000000000000000000000000000000000000000000000001190000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000279dc681e3f00000000000000000000000000074dcc6d44d4c8486ace698bbb776d8ccd985cdfa000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000035a9020ab0000000000000000000000000000000a26b00c1f0df003000390027140000faa7190000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002189a146ae000000000000000000000000000240efcc648e264e2e76367e5d7e95b6e9bee72c2`),
	}
	order, err := seaport.MatchOrderFulfilled(log)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%x\n", order.Recipient)
}

Generating from an ABI

First install genabi

% go install github.com/indexsupply/x/cmd/genabi@latest

package main

import (
	"encoding/hex"
	"fmt"

	"github.com/indexsupply/x/abi"
)

//go:generate genabi -i punks.json -o punks.go -p main

func h2b(s string) []byte { b, _ := hex.DecodeString(s); return b }

func main() {
	log := abi.Log{
		Topics: [][32]byte{
			[32]byte(h2b("58e5d5a525e3b40bc15abaa38b5882678db1ee68befd2f60bafe3a7fd06db9e3")),
			[32]byte(h2b("0000000000000000000000000000000000000000000000000000000000001f98")),
			[32]byte(h2b("000000000000000000000000d985bd504d737c6533335dfa4ad74e571bb3bd88")),
			[32]byte(h2b("0000000000000000000000002c771def8f394e4f0ed4313cd0d1a29ca61313c8")),
		},
		Data: h2b(`000000000000000000000000000000000000000000000003d7a858762fb00000`),
	}
	pb, err := MatchPunkBought(log)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s\n", pb.Value.ToBig().String())
}

Using Decode Directly

package main

import (
	"encoding/hex"
	"fmt"

	"github.com/indexsupply/x/abi"
	"github.com/indexsupply/x/abi/schema"
)

func h2b(s string) []byte { b, _ := hex.DecodeString(s); return b }

func main() {
	log := abi.Log{
		Topics: [][32]byte{
			[32]byte(h2b("82fbbe47003f3ca8230f7ab30eeabd25f09023e15ca04dc5bd49ac1a11a223b9")),
		},
		Data: h2b(`0000000000000000000000000000000000000000000000000000000000000d590000000000000000000000000000000000000000000066ad799bcef571dbdc1000000000000000000000000000000000000000000000734a1f3d475b7d7083c2`),
	}
	item, n, err := abi.Decode(log.Data, schema.Tuple(
		schema.Static(),
		schema.Static(),
		schema.Static(),
	))
	defer item.Done()
	if err != nil {
		panic(err)
	}
	fmt.Printf("read: (%d) \nera: %s \nreceived: %s \naccounted: %s\n",
		n,
		item.At(0).BigInt().String(),
		item.At(1).BigInt().String(),
		item.At(2).BigInt().String(),
	)
}

Using Decode and Parsing a Schema

package main

import (
	"encoding/hex"
	"fmt"

	"github.com/indexsupply/x/abi"
	"github.com/indexsupply/x/abi/schema"
)

func h2b(s string) []byte { b, _ := hex.DecodeString(s); return b }

func main() {
	log := abi.Log{
		Topics: [][32]byte{
			[32]byte(h2b("82fbbe47003f3ca8230f7ab30eeabd25f09023e15ca04dc5bd49ac1a11a223b9")),
		},
		Data: h2b(`0000000000000000000000000000000000000000000000000000000000000d590000000000000000000000000000000000000000000066ad799bcef571dbdc1000000000000000000000000000000000000000000000734a1f3d475b7d7083c2`),
	}
	item, n, err := abi.Decode(log.Data, schema.Parse("(uint256,uint256,uint256)"))
	defer item.Done()
	if err != nil {
		panic(err)
	}
	fmt.Printf("read: (%d) \nera: %s \nreceived: %s \naccounted: %s\n",
		n,
		item.At(0).BigInt().String(),
		item.At(1).BigInt().String(),
		item.At(2).BigInt().String(),
	)
}

Bugs

Give the abi package a try! If you hit any snags, please open an issue ASAP! I'm eager to get more mileage on this code.

[ryandotsmith]

I'd also like to explicitly say that I have nothing but respect for the heroic effort put forth by the Geth team. I am a huge fan of their work! Without it the world would not be as interesting.