-
Notifications
You must be signed in to change notification settings - Fork 2
/
README
585 lines (450 loc) · 24.2 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
------------------------------
spoa-opentracing for HAProxy
------------------------------
Summary
------------------------------------------------------------------------
1. Introduction
2. Build instructions
3. Use of the program
4. Basic concepts in the OpenTracing
5. SPOE configuration
6. Tracing example
7. Benchmarking results
8. Known bugs and limitations
1. Introduction
------------------------------------------------------------------------
This is a very simple program that allows you to send data to distributed
tracing systems via the OpenTracing API, using the HAProxy SPOP; i.e. Stream
Processing Offload Protocol.
The program originated from a test program located in the contrib/spoa_example
directory. spoa.c is an original program that comes with HAProxy source.
The program uses the libev event library, as opposed to the original program
that uses the libevent library.
2. Build instructions
------------------------------------------------------------------------
Prerequisites for configuring and compiling the program:
--------------------------------------------------------
* GNU GCC Compiler and Development Environment
The main components of this system are C standard library, C and C++
compilers, make and cmake utilities and other development tools that are
not explicitly listed because they depend on the system on which we want
to compile the program.
The program itself is very likely possible to compile with another C
compiler (non-GCC), but I'm not sure about the other required libraries.
On debian-like linux systems it is necessary to run the following, in
order to install development packages:
# apt-get install automake autoconf make cmake libc6-dev gcc g++ binutils libev-dev libtool pkgconf gawk git
* GNU autoconf
https://www.gnu.org/software/autoconf/
* GNU automake
https://www.gnu.org/software/automake/
* POSIX threads library
* libev library
http://software.schmorp.de/pkg/libev.html
* C++ implementation of the OpenTracing API http://opentracing.io
https://github.com/opentracing/opentracing-cpp
* C wrapper for the opentracing-cpp library
https://github.com/haproxytech/opentracing-c-wrapper
* the tracing plugins (one of the following may be used):
* Jaeger SDK with OpenTracing API for C++ binding
https://github.com/jaegertracing/jaeger-client-cpp
* the LightStep distributed tracing library for C++
https://github.com/lightstep/lightstep-tracer-cpp
* OpenTracing implementation for Zipkin in C++
https://github.com/rnburn/zipkin-cpp-opentracing
* Datadog OpenTracing C++ Client
https://github.com/DataDog/dd-opentracing-cpp
Note: prompt '%' indicates that the command is executed under a unprivileged
user, while prompt '#' indicates that the command is executed under the
root user.
Compiling and installing the opentracing-cpp library:
-----------------------------------------------------
The assumption is that we want to install the library in the /opt directory.
% wget https://github.com/opentracing/opentracing-cpp/archive/v1.5.0.tar.gz
% tar xf v1.5.0.tar.gz
% cd opentracing-cpp-1.5.0
% mkdir build
% cd build
% cmake -DCMAKE_INSTALL_PREFIX=/opt ..
% make
# make install
Of course, we can take another version of the library (or download the master
branch via git). For example, this version is used here due to compatibility
with the Jaeger plugin, which I mostly used when testing the operation of the
program.
Compiling and installing the opentracing-c-wrapper library:
-----------------------------------------------------------
We will also install this library in the /opt directory, and when configuring
the library we must specify where the opentracing-cpp library is located.
In this example we will install two builds of the library, first the
release version and then the debug version.
% git clone https://github.com/haproxytech/opentracing-c-wrapper.git
% cd opentracing-c-wrapper
% ./scripts/bootstrap
% ./configure --prefix=/opt --with-opentracing=/opt
% make
# make install
% ./scripts/distclean
% ./scripts/bootstrap
% ./configure --prefix=/opt --enable-debug --with-opentracing=/opt
% make
# make install
Compiling the Jaeger tracing plugin:
------------------------------------
We will use the 1.5.0 version of the plugin, a newer one can be taken (or
the git master branch) if you want to try it (as is the case with previous
libraries, you can try a newer version).
Important note: the GCC version must be at least 4.9 or later.
% wget https://github.com/jaegertracing/jaeger-client-cpp/archive/v0.5.0.tar.gz
% tar xf v0.5.0.tar.gz
% cd jaeger-client-cpp-0.5.0
% mkdir build
% cd build
% cmake -DCMAKE_INSTALL_PREFIX=/opt -DJAEGERTRACING_PLUGIN=ON -DHUNTER_CONFIGURATION_TYPES=Release -DHUNTER_BUILD_SHARED_LIBS=OFF ..
% make
After the plugin is compiled, it will be in the current directory.
The name of the plugin is libjaegertracing_plugin.so.
Of course, we can download the precompiled version of the Jaeger plugin
(unfortunately, the latest version that can be downloaded is 0.4.2).
% wget https://github.com/jaegertracing/jaeger-client-cpp/releases/download/v0.4.2/libjaegertracing_plugin.linux_amd64.so
Creating the build subsystem for the program (configure script):
----------------------------------------------------------------
% ./scripts/bootstrap
Configuring the program:
------------------------
% ./configure --with-libev[=DIR] --with-opentracing[=DIR]
The program can be configured with debugging features enabled, but it is
not recommended for actual use:
% ./configure --enable-debug --with-libev[=DIR] --with-opentracing[=DIR]
To configure the program, the system must have installed development packages
for opentracing-c-wrapper and libev; which will be selected automatically
when running the configure script, if their path is not specified explicitly.
Compiling the program:
----------------------
% make all
After that, in the directory src will be located spoa-opentracing program.
3. Use of the program
------------------------------------------------------------------------
The options supported by the program can be found using the '-h' option:
% ./src/spoa-opentracing -h
--- help output -------
Usage: spoa-opentracing { -h --help }
spoa-opentracing { -V --version }
spoa-opentracing { -r --runtime=TIME } [OPTION]...
Options are:
-a, --address=NAME Specify the address to listen on (default: "0.0.0.0").
-B, --libev-backend=TYPE Specify the libev backend type (default: AUTO).
-b, --connection-backlog=VALUE Specify the connection backlog size (default: 10).
-c, --capability=NAME Enable the support of the specified capability.
-D, --daemonize Run this program as a daemon.
-d, --debug=LEVEL Enable and specify the debug mode level (default: 129).
-F, --pidfile=FILE Specifies a file to write the process-id to.
-h, --help Show this text.
-i, --monitor-interval=TIME Set the monitor interval (default: 5.00s).
-l, --logfile=[MODE:]FILE Log all messages to logfile (default: stdout/stderr).
-m, --max-frame-size=VALUE Specify the maximum frame size (default: 16384 bytes).
-n, --num-workers=VALUE Specify the number of workers (default: 10).
-p, --port=VALUE Specify the port to listen on (default: 12345).
-r, --runtime=TIME Run this program for the specified time (0 = unlimited).
-t, --processing-delay=TIME Set a delay to process a message (default: 0).
-o, --ot-timeout-msg=STR Specify a timeout log message (default: "connection timeout").
-C, --ot-tracer-config=FILE Specify the configuration of the used tracer.
-T, --ot-tracer-plugin=FILE Specify the OpenTracing compatible plugin library.
-V, --version Show program version.
Supported libev backends: select, poll, epoll, linuxaio.
Supported capabilities: fragmentation, pipelining, async.
Allowed logging file opening modes: a, w. The 'a' mode allows openning or
creating file for writing at end-of-file. The 'w' mode allows truncating
the file to zero length or creating a new file. If a capital letter is used
for the mode, then line buffering is used when writing to the log file.
The time delay/interval is specified in milliseconds by default, but can be
in any other unit if the number is suffixed by a unit (us, ms, s, m, h, d).
Copyright 2020 HAProxy Technologies
SPDX-License-Identifier: GPL-2.0-or-later
--- help output -------
The program simultaneously supports short and long format options.
The test directory contains the configuration files for HAProxy (haproxy.cfg)
and SPOE (spoe.cfg). ssl-cert.pem is a certificate intended for testing if we
want to use HTTPS HAProxy frontend.
During the normal work of the program, should be placed under the comment the
following lines in the spoe.cfg:
--- spoe.cfg ----------
messages check-client-ip
messages test
--- spoe.cfg ----------
These two rows can be used when debugging the program, especially if it is
configured with the option --enable-debug .
Example of using the program:
% ./src/spoa-opentracing -r0 -Ctest/cfg-jaeger.yml -Ttest/libjaeger_opentracing_plugin-0.4.2.so
'-r' is the option that must be specified when launching a program. It is
used to prevent the program from starting unnecessarily when testing the
program options, or printing the program help. In addition, with this option,
the program runtime is set. If the time is set to 0, the program runtime is
unlimited.
With the '-C' option, we specify the configuration of the used tracer (in this
case it is Jeager); while the '-T' option selects the plugin library that the
selected tracer uses.
After that, the HAProxy should be started, using haproxy.cfg configuration
file:
% /path-to/haproxy -f haproxy.cfg
Of course, we can also use the master-worker mode of HAProxy and specify the
command to start the program directly in the configuration. In that case we
just need to run the HAProxy which will run the program on its own.
The test directory contains several HAProxy configurations prepared for
different modes of operation, as well as configurations for supported tracers.
The HAProxy configurations:
- haproxy.cfg - simple configuration with defined SPOE filter
- haproxy-htx.cfg - configuration similar to the above using the HTTPS
protocol
- haproxy-mw.cfg - master-worker configuration
Configurations for the selected tracer type:
- cfg-dd.json - Datadog tracer
- cfg-jaeger.yml - Jaeger tracer
- cfg-zipkin.json - Zipkin tracer
SPOE configuration (The OpenTracing Data Model):
- spoe.cfg - universal SPOE configuration that can be used with any tracer
SSL certificate used if we run HAProxy in HTTPS mode:
- ssl-cert.pem
Script to download all supported tracer plugins:
- get-opentracing-plugins.sh - using this script we can download the latest
versions of tracer plugins (the latest versions that are available as a
pre-compiled plugin, this does not mean that they are the latest versions
of the current source)
Jaeger docker image installation:
---------------------------------
Installation instructions can be found on the website
https://www.jaegertracing.io/download/. For the impatient, here we will list
how the image to test the operation of the tracer system can be installed
without much reading of the documentation.
# docker pull jaegertracing/all-in-one:latest
# docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:latest
The last command will also initialize and run the Jaeger container.
If we want to use that container later, it can be started and stopped
in the classic way, using the 'docker container start/stop' commands.
4. Basic concepts in the OpenTracing
------------------------------------------------------------------------
Basic concepts of the OpenTracing can be read on the OpenTracing documentation
website https://opentracing.io/docs/overview/.
Here we will list only the most important elements of distributed tracing and
these are 'trace', 'span' and 'span context'. Trace is a description of the
complete transaction we want to record in the tracing system. A span is an
operation that represents a unit of work that is recorded in a tracing system.
Span context is a group of information related to a particular span that is
passed on to the system (from service to service). Using this context, we can
add new spans to already open trace (or supplement data in already open spans).
An individual span may contain one or more tags, logs and baggage items.
The tag is a key-value element that is valid for the entire span. Log is a
key-value element that allows you to write some data at a certain time, it
can be used for debugging. A baggage item is a key-value data pair that can
be used for the duration of an entire trace, from the moment it is added to
the span.
5. SPOE configuration
------------------------------------------------------------------------
The SPOE configuration supports everything described in the documentation
for HAProxy, of course (haproxy-src/doc/SPOE.txt). Therefore, we will
focus here only on configuring specific elements related to the OpenTracing.
Everything related to the OpenTracing is defined through the 'spoe-message'
and 'spoe-group' sections (actually 'spoe-group' is again reduced to the
'spoe-message' section).
--- spoe.cfg ----------
spoe-message
args id=unique-id
[ span=str("span id") [ child-of=str("parent span id") | follows-from=str("parent span id") ] ]
[ tag=str("tag id") data ... ]
[ baggage=str("baggage id") data ... ]
[ log=str("log id") data ... ]
[ finish=str("span id") ]
[ event <name> [ { if | unless } condition ] ]
--- spoe.cfg ----------
The syntax of the 'spoe-message' definition related to the OpenTracing is
shown above. Mandatory elements are shown without square brackets. Each
argument must have its own predefined name, specific to the OpenTracing.
The following is a brief description of all the arguments used:
id=unique-id
Unique ID associated with a particular trace, its content is defined via
the 'unique-id-format' keyword in the HAProxy configuration. This is an
argument that must always be present at the beginning of the definition
of a 'spoe-message' section.
span=str("span id") [ child-of=str("parent span id") | follows-from=str("parent span id") ]
Creating a new span (or referencing an already opened one). If a new span
is created, it can be a child of or follows from the previous span. In case
we did not specify a reference to the previously created span, the new span
will become the root span. We need to pay attention to the fact that in one
trace there can be only one root span. In case we have specified a
non-existent span as a reference, a new span will not be created.
tag=str("tag id") data ...
Setting a tag for the currently active span. The first argument is the name
of the tag (tag ID) and the second its value. A value can consist of one or
more data. If the value is only one data, then the type of that data depends
on the type of SPOE sample. If the value contains more data, then the data
type is string. The data conversion table is below:
SPOE sample data type | the OpenTracing data type
-----------------------+---------------------------
NULL | NULL
BOOL | BOOL
INT32 | INT64
UINT32 | UINT64
INT64 | INT64
UINT64 | UINT64
IPV4 | STRING
IPV6 | STRING
STRING | STRING
BINARY | UNSUPPORTED
-----------------------+---------------------------
baggage=str("baggage id") data ...
Setting the baggage for the currently active span. The data type is always
a string, ie any SPOE sample type is converted to a string. The exception
is a binary value that is not supported by the program.
log=str("log id") data ...
Setting the log for the currently active span. The data type is always a
string, ie any SPOE sample type is converted to a string. The exception is
a binary value that is not supported by the program.
finish=str("span id")
Closing a particular span. The '*' character can be used instead of the
span name to indicate all currently open spans.
Important: all spans must be explicitly closed, otherwise the program will
accumulate open spans and unnecessarily lose memory on them. Also, unclosed
spans will not be displayed in the tracing system.
Eventually, the program will close all open spans itself after HAProxy closes
communication to the program. However, such spans will be displayed in the
tracing system with incorrect duration and as such will not be of any use.
event <name> [ { if | unless } condition ]
Set the event that triggers sending of the message. It may optionally be
followed by an ACL-based condition, in which case it will only be evaluated
if the condition is true.
6. Tracing example (the one from the test directory)
------------------------------------------------------------------------
For example we will use the configuration file haproxy-mw.cfg which will start
HAProxy in master-worker mode, and it will run the spoa-opentracing program.
HAProxy runs in HTTP mode and its frontend is bound to all network interfaces
on ports 10080 and 10443 (the latter for SSL communication). HAProxy backend
is located at local address 127.0.0.1 on port 8000.
The SPOE configuration is located in the spoe.cfg file (it is common to any
HAProxy configuration we use). All possible SPOE events are used.
Below are two possible cases, the first in which all services are working
properly and the second in which the backend server is not working. This
latter case after a timeout of 10 seconds closes all open spans and sets
the error status on the root span.
9 spans, backend server active
HAProxy sess |---------------------------------------------------------- 2ms |
Client session |-----------------------------------------------------------|
FE TCP req |-----|
FE HTTP req |-----|
BE TCP req |-----|
BE HTTP req |-----|
Server session |-----------------------------|
TCP res |--------------|
HTTP res |------|
SPOE event on-* | SPAN created | SPAN finished
-----------------------+-----------------------------+------------------------
client-session | HAProxy session | -
| -> Client session | -
frontend-tcp-request | -> Frontend TCP request | -
frontend-http-request | Frontend HTTP request | Frontend TCP request
backend-tcp-request | Backend TCP request | Frontend HTTP request
backend-http-request | Backend HTTP request | Backend TCP request
server-session | -> Server session | Backend HTTP request
tcp-response | -> TCP response | -
http-response | HTTP response | TCP response
| | HTTP response
| | Server session
| | *
-----------------------+-----------------------------+------------------------
6 spans, 1 error set, backend server timeout
HAProxy session |------------------------------------------------- 10s |
Client session |------------------------------------------------------|
FE TCP req |
FE HTTP req |
BE TCP req |
BE HTTP req |------------------------------------------------------|
SPOE event on-* | SPAN created | SPAN finished
-----------------------+-----------------------------+------------------------
client-session | HAProxy session | -
| -> Client session | -
frontend-tcp-request | -> Frontend TCP request | -
frontend-http-request | Frontend HTTP request | Frontend TCP request
backend-tcp-request | Backend TCP request | Frontend HTTP request
backend-http-request | Backend HTTP request | Backend TCP request
**server timeout** | | *
-----------------------+-----------------------------+------------------------
7. Benchmarking results
------------------------------------------------------------------------
Here we will publish several results of measuring the speed of HAProxy SPOE
functionality. The first result was obtained by using HAProxy without SPOE
filter on, the second and third with SPOE filter on and using test mode of
programs that give a random number from 0 to 100 for an IP address; while
the fourth result was the real use of the OpenTracing system.
The wrk benchmarking tool was used for testing (https://github.com/wg/wrk).
./src/spoa-opentracing SPOE not used
------------------------------------------------------------------------------
% wrk -c8 -d300 -t8 --latency http://localhost:10080/index.html
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 379.15us 524.77us 22.81ms 98.05%
Req/Sec 2.56k 123.07 2.94k 85.92%
Latency Distribution
50% 315.00us
75% 369.00us
90% 439.00us
99% 2.17ms
6119626 requests in 5.00m, 2.38GB read
Requests/sec: 20396.66
Transfer/sec: 8.13MB
./contrib/spoa_example/spoa check-client-ip
------------------------------------------------------------------------------
% wrk -c8 -d300 -t8 --latency http://localhost:10080/index.html
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.98ms 6.11ms 106.74ms 96.54%
Req/Sec 0.95k 192.80 2.23k 82.05%
Latency Distribution
50% 428.00us
75% 1.74ms
90% 2.39ms
99% 34.17ms
2271564 requests in 5.00m, 0.89GB read
Requests/sec: 7569.43
Transfer/sec: 3.05MB
./src/spoa-opentracing check-client-ip
------------------------------------------------------------------------------
% wrk -c8 -d300 -t8 --latency http://localhost:10080/index.html
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.29ms 2.61ms 104.56ms 96.23%
Req/Sec 0.98k 123.04 3.56k 77.43%
Latency Distribution
50% 435.00us
75% 1.70ms
90% 2.19ms
99% 10.41ms
2342172 requests in 5.00m, 0.92GB read
Requests/sec: 7804.66
Transfer/sec: 3.14MB
./src/spoa-opentracing opentracing
------------------------------------------------------------------------------
% wrk -c8 -d300 -t8 --latency http://localhost:10080/index.html
Running 5m test @ http://localhost:10080/index.html
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.92ms 3.24ms 220.71ms 93.76%
Req/Sec 363.93 51.27 626.00 73.00%
Latency Distribution
50% 1.65ms
75% 4.57ms
90% 5.64ms
99% 9.64ms
869602 requests in 5.00m, 348.32MB read
Requests/sec: 2897.88
Transfer/sec: 1.16MB
8. Known bugs and limitations
------------------------------------------------------------------------
The program does not support writing/reading span context, ie you cannot use
the program to add spans to an already created trace from another service.
In case the backend server does not work, the traces (and spans of course)
will not be closed properly. In that case the program has some kind of
security and it will mark such traces as backed server timeout operations
but this is a rather unreliable way of marking the concerned because it is
performed only when haproxy closes the communication channel with the program.