Skip to content

Commit

Permalink
Merge bitcoin#19866: eBPF Linux tracepoints
Browse files Browse the repository at this point in the history 
22eb793 tracing: add tracing framework (William Casarin)
933ab8a build: detect sys/sdt.h for eBPF tracing (William Casarin)

Pull request description:

  Instead of writing ad-hoc logging everywhere (eg: bitcoin#19509), we can take advantage of linux user static defined traces, aka. USDTs ( not the stablecoin 😅 )

  The linux kernel can hook into these tracepoints at runtime, but otherwise they have little to no performance impact. Traces can pass data which can be printed externally via tools such as bpftrace. For example, here's one that prints incoming and outgoing network messages:

  # Examples

  ## Network Messages

  ```
  #!/usr/bin/env bpftrace

  BEGIN
  {
    printf("bitcoin net msgs\n");
    @start = nsecs;
  }

  usdt:./src/bitcoind:net:push_message
  {
    $ip = str(arg0);
    $peer_id = (int64)arg1;
    $command = str(arg2);
    $data_len = arg3;
    $data = buf(arg3,arg4);
    $t = (nsecs - @start) / 100000;

    printf("%zu outbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);

    @outbound[$command]++;
  }

  usdt:./src/bitcoind:net:process_message
  {
    $ip = str(arg0);
    $peer_id = (int64)arg1;
    $command = str(arg2);
    $data_len = arg3;
    $data = buf(arg3,arg4);
    $t = (nsecs - @start) / 100000;

    printf("%zu inbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);

    @inbound[$ip, $command]++;
  }

  ```

      $ sudo bpftrace netmsg.bt

  output: https://jb55.com/s/b11312484b601fb3.txt

  if you look at the bottom of the output you can see a histogram of all the messages grouped by message type and IP. nice!

  ## IBD Benchmarking

  ```
  #!/usr/bin/env bpftrace
  BEGIN
  {
    printf("IBD to 500,000 bench\n");
  }

  usdt:./src/bitcoind:CChainState:ConnectBlock
  {
    $height = (uint32)arg0;

    if ($height == 1) {
      printf("block 1 found, starting benchmark\n");
      @start = nsecs;
    }

    if ($height >= 500000) {
      @EnD = nsecs;
      @duration = @EnD - @start;
      exit();
    }
  }

  END {
    printf("duration %d ms\n", @duration / 1000000)
  }
  ```
  This one hooks into ConnectBlock and prints the IBD time to height 500,000 starting from the first call to ConnectBlock

  Userspace static tracepoints give lots of flexibility without invasive logging code. It's also more flexible than ad-hoc logging code, allowing you to instrument many different aspects of the system without having to enable per-subsystem logging.

  Other ideas: tracepoints for lock contention, threads, what else?

  Let me know what ya'll think and if this is worth adding to bitcoin.

  ## TODO

  - [ ] docs?
  - [x] Integrate systemtap-std-dev/libsystemtap into build (provides the <sys/sdt.h> header)
  - [x] ~dtrace macos support? (is this still a thing?)~ going to focus on linux for now

ACKs for top commit:
  laanwj:
    Tested ACK 22eb793
  0xB10C:
    Tested ACK 22eb793

Tree-SHA512: 69242242112b679c8a12a22b3bc50252c305894fb3055ae6e13d5f56221d858e58af1d698af55e23b69bdb7abedb5565ac6b45fa5144087b77a17acd04646a75
  • Loading branch information
@laanwj
laanwj committed Jan 18, 2021
2 parents f7fd76b + 22eb793 commit 43f3ada
Show file tree
Hide file tree
Showing 3 changed files with 64 additions and 0 deletions.
18 changes: 18 additions & 0 deletions configure.ac
View file
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,12 @@ AC_ARG_WITH([bdb],
[use_bdb=$withval],
[use_bdb=auto])

AC_ARG_ENABLE([ebpf],
[AS_HELP_STRING([--enable-ebpf],
[enable eBPF tracing (default is yes if sys/sdt.h is found)])],
[use_ebpf=$enableval],
[use_ebpf=yes])

AC_ARG_WITH([miniupnpc],
[AS_HELP_STRING([--with-miniupnpc],
[enable UPNP (default is yes if libminiupnpc is found)])],
Expand Down Expand Up @@ -1295,6 +1301,16 @@ if test x$enable_wallet != xno; then
fi
fi

if test x$use_ebpf != xno; then
AC_CHECK_HEADER([sys/sdt.h], [have_sdt=yes], [have_sdt=no])
else
have_sdt=no
fi

if test x$have_sdt = xyes; then
AC_DEFINE([ENABLE_TRACING], [1], [Define to 1 to enable eBPF user static defined tracepoints])
fi

dnl Check for libminiupnpc (optional)
if test x$use_upnp != xno; then
AC_CHECK_HEADERS(
Expand Down Expand Up @@ -1689,6 +1705,7 @@ AM_CONDITIONAL([TARGET_WINDOWS], [test x$TARGET_OS = xwindows])
AM_CONDITIONAL([ENABLE_WALLET],[test x$enable_wallet = xyes])
AM_CONDITIONAL([USE_SQLITE], [test "x$use_sqlite" = "xyes"])
AM_CONDITIONAL([USE_BDB], [test "x$use_bdb" = "xyes"])
AM_CONDITIONAL([ENABLE_TRACING],[test x$have_sdt = xyes])
AM_CONDITIONAL([ENABLE_TESTS],[test x$BUILD_TEST = xyes])
AM_CONDITIONAL([ENABLE_FUZZ],[test x$enable_fuzz = xyes])
AM_CONDITIONAL([ENABLE_FUZZ_LINK_ALL],[test x$enable_danger_fuzz_link_all = xyes])
Expand Down Expand Up @@ -1855,6 +1872,7 @@ echo " with bench = $use_bench"
echo " with upnp = $use_upnp"
echo " with natpmp = $use_natpmp"
echo " use asm = $use_asm"
echo " ebpf tracing = $have_sdt"
echo " sanitizers = $use_sanitizers"
echo " debug enabled = $enable_debug"
echo " gprof enabled = $enable_gprof"
Expand Down
1 change: 1 addition & 0 deletions src/Makefile.am
View file
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,7 @@ BITCOIN_CORE_H = \
util/system.h \
util/threadnames.h \
util/time.h \
util/trace.h \
util/translation.h \
util/ui_change_type.h \
util/url.h \
Expand Down
45 changes: 45 additions & 0 deletions src/util/trace.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// Copyright (c) 2020 The Bitcoin Core developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.

#ifndef BITCOIN_UTIL_TRACE_H
#define BITCOIN_UTIL_TRACE_H

#ifdef ENABLE_TRACING

#include <sys/sdt.h>

#define TRACE(context, event) DTRACE_PROBE(context, event)
#define TRACE1(context, event, a) DTRACE_PROBE1(context, event, a)
#define TRACE2(context, event, a, b) DTRACE_PROBE2(context, event, a, b)
#define TRACE3(context, event, a, b, c) DTRACE_PROBE3(context, event, a, b, c)
#define TRACE4(context, event, a, b, c, d) DTRACE_PROBE4(context, event, a, b, c, d)
#define TRACE5(context, event, a, b, c, d, e) DTRACE_PROBE5(context, event, a, b, c, d, e)
#define TRACE6(context, event, a, b, c, d, e, f) DTRACE_PROBE6(context, event, a, b, c, d, e, f)
#define TRACE7(context, event, a, b, c, d, e, f, g) DTRACE_PROBE7(context, event, a, b, c, d, e, f, g)
#define TRACE8(context, event, a, b, c, d, e, f, g, h) DTRACE_PROBE8(context, event, a, b, c, d, e, f, g, h)
#define TRACE9(context, event, a, b, c, d, e, f, g, h, i) DTRACE_PROBE9(context, event, a, b, c, d, e, f, g, h, i)
#define TRACE10(context, event, a, b, c, d, e, f, g, h, i, j) DTRACE_PROBE10(context, event, a, b, c, d, e, f, g, h, i, j)
#define TRACE11(context, event, a, b, c, d, e, f, g, h, i, j, k) DTRACE_PROBE11(context, event, a, b, c, d, e, f, g, h, i, j, k)
#define TRACE12(context, event, a, b, c, d, e, f, g, h, i, j, k, l) DTRACE_PROBE12(context, event, a, b, c, d, e, f, g, h, i, j, k, l)

#else

#define TRACE(context, event)
#define TRACE1(context, event, a)
#define TRACE2(context, event, a, b)
#define TRACE3(context, event, a, b, c)
#define TRACE4(context, event, a, b, c, d)
#define TRACE5(context, event, a, b, c, d, e)
#define TRACE6(context, event, a, b, c, d, e, f)
#define TRACE7(context, event, a, b, c, d, e, f, g)
#define TRACE8(context, event, a, b, c, d, e, f, g, h)
#define TRACE9(context, event, a, b, c, d, e, f, g, h, i)
#define TRACE10(context, event, a, b, c, d, e, f, g, h, i, j)
#define TRACE11(context, event, a, b, c, d, e, f, g, h, i, j, k)
#define TRACE12(context, event, a, b, c, d, e, f, g, h, i, j, k, l)

#endif


#endif /* BITCOIN_UTIL_TRACE_H */

0 comments on commit 43f3ada

Please sign in to comment.