kv: abort span access is expensive (2% cpu on oltp_read_write)

[nvanbenschoten]

The abort span (pkg/kv/kvserver/abortspan) is a mechanism that sets markers for aborted transactions to provide protection against an aborted but active transaction not reading values it wrote due to those intents having been removed.

The "span" is a slice of the range-id-local keyspace which is read on each BatchRequest that is part of a read-write transaction. The logic for this is here:

cockroach/pkg/kv/kvserver/replica_evaluate.go

Lines 208 to 227 in 55991cb

	// Check whether this transaction has been aborted, if applicable. This
	// applies to reads and writes once a transaction that has begun to
	// acquire locks (see #2231 for more about why we check for aborted
	// transactions on reads). Note that 1PC transactions have had their
	// transaction field cleared by this point so we do not execute this
	// check in that case.
	if baHeader.Txn.IsLocking() {
	// We don't check the abort span for a couple of special requests:
	// - if the request is asking to abort the transaction, then don't check the
	// AbortSpan; we don't want the request to be rejected if the transaction
	// has already been aborted.
	// - heartbeats don't check the abort span. If the txn is aborted, they'll
	// return an aborted proto in their otherwise successful response.
	// TODO(nvanbenschoten): Let's remove heartbeats from this allowlist when
	// we rationalize the TODO in txnHeartbeater.heartbeat.
	if !ba.IsSingleAbortTxnRequest() && !ba.IsSingleHeartbeatTxnRequest() {
	if pErr := checkIfTxnAborted(ctx, rec, readWriter, *baHeader.Txn); pErr != nil {
	return nil, result.Result{}, pErr
	}
	}

This is an additional LSM read per BatchRequest, which can be seen prominently in CPU profiles under checkIfTxnAborted, accounting for 3.59% of CPU time on write-heavy workloads:

profile_abort_span.pb.gz

Some basic experimentation with the sysbench workload (sysbench/oltp_write_only/nodes=7/cpu=16/conc=128) demonstrates about a 2% increase in throughput by disabling this abort span read (i.e. not calling checkIfTxnAborted). This testing reveals the cost of the mechanism. Optimizations (up and including disabling it) could provide up to this much benefit to throughput.

Given how significant this cost is and how much of an edge case the scenarios that the abort span is protecting against are, we should reevaluate whether there's something better that we can do here. Are there simple optimizations that could make this mechanism perform better? Could we make it a little weaker to avoid most of the cost? These questions are worthwhile to explore.

At a minimum, we should expose an option to disable these abort span checks.

Jira issue: CRDB-38032

Epic CRDB-40199

[tbg]

@nvanbenschoten and I saw this while looking at an (admittedly, overloaded) 3x8vcpu tpcc-nowait wh=1000 cluster:

Notably, this was running with #136523 as well as #122862, both of which in principle reduce abort span access.