-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
318 lines (234 loc) · 11 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
sp-rtrace
---------
Contents:
1. Introduction
2. Components
2.1 Tracing modules
2.1.1 Main module (sp-rtrace-main.so)
2.1.2 Tracing modules
2.2 Data pre-processor (rtrace)
2.3 Data post-processor (rtrace-postproc)
2.3.1 Internal data storage
2.3.2 Post-processing
2.3.2.1 leak filtering (--filter-leaks)
2.3.2.2 backtrace compressing (--compress)
2.3.3.3 future post-processing options
3 Tracing Configuration
4 Trace data flow
5 Visualization tools
6 Notes
6.1 Limitations
6.1 Scratchbox (Maemo)
6.2 Debugging
1. Introduction
The sp-rtrace resource consumption tracking suite contains a set of
utilities for different resource types allocation / deallocation call
tracing, trace post-processing and visualization.
2. Components
+----------------------+ +---------------+ +----------------+
| main module | -> | pre-processor | ,-> | post-processor |
+----------------------+ +---------------+ | +----------------+
| tracing module 1 | | | | | ^
+------------------+ v | v v |
| ... | binary output ---+ text output
+------------------+ / | ^
| tracing module N | / v |
+------------------+ +------------+ <------' +----------+
| visualizer | <---------- | resolver |
+------------+ +----------+
2.1 Tracing modules
The tracing modules are the primary data source. They feed function
trace data to the pre-processor via named (or simple in managed mode)
pipe.
2.1.1 Main module (sp-rtrace-main.so)
The main module is always preloaded for the target process when it
is started with rtrace. The main module provides the following
functionality:
1) pre-processor connection management,
2) data writing API used by tracing modules to report trace events,
3) tracking of new libraries.
The main module is configured by environment variables. rtrace
automatically sets the necessary variables before it launches
the target process. It is possible to set the variables and
launch the target process manually, however it is not recommended.
By default when tracing is enabled the main module attempts to open
a named pipe /tmp/rtrace-<pid> where pid is the process id of the
current (traced) process. But in managed mode the main module
starts the rtrace process by itself and uses its standard input to
pipe trace data (see sp-rtrace manual page --manage-preproc option
description for when to use managed mode).
By default the main module buffers (KB) trace data for performance
reasons. However it's possible to disable the data buffering with
an environment variable.
The main module ensures that memory mapping information is being
kept up to date by tracking dlopen() calls and sending new library
(NL) packet whenever a new library is opened. This packet instructs
pre-processor to rescan /proc/<pid>/maps file for the target process.
The main module keeps registry of loaded tracing modules to be able
disable tracing when necessary.
2.1.2 Tracing modules
The tracing modules are either preloaded with LD_PRELOAD or LD_AUDIT.
They provide tracing of resource alloaction/deallocation calls.
Usually one module covers single resource type (memory, file etc).
For example memory tracing module monitors memory allocation/deallocation
calls - malloc, alloc, calloc, posix_memalign, free.
Tracing modules must register themselves to the main module during
initialization to allow the main module enable/disable tracing when
necessary.
Tracing modules uses data writing API provided by the main module to
report resource allocation/deallocation calls.
2.2 Data pre-processor (rtrace)
The data pre-processor provides the following functionality:
1) launching of a binary file with enabled resource tracing,
2) pre-processing data sent by tracing modules,
3) trace enabling/disabling for already launched process.
If tracing is enabled (with the "-s" option) at start, after launching
the given process pre-processor creates a named pipe /tmp/rtrace-<pid>
and listens it for the trace data.
If tracing wasn't enabled at start, sp-rtrace exits after launching
the process-to-trace onto background. This way one can start large
number of processes e.g. at device boot to wait for being traced,
without redundant sp-rtrace instances.
When managed mode is used, sp-rtrace uses exec to replace itself with
the target process. A new pre-processor is started by the sp-rtrace
LD_PRELOAD main module itself for data pre-processing when tracing is
started.
The post-processor keeps track of the module memory mapping and
rescans /proc/<pid>/maps file whenever new library (NL) packet is
received. All executable module changes are reported by generating
memory mapping (MM) packets.
If post-processing options are specified, the pre-processor spawns
post-processor (rtrace-postproc) process and pipes the trace data
to it. Otherwise the data is stored in new file in the output
directory (home directory by default) - rtrace-raw-<pid>-<number>
Pre-processor never overwrites the existing data files, it increases
the file number until a new file can be created.
The pre-processor also uses internal buffer (4KB) for buffering
pipe data which can be also disabled (by the same option as
main module trace data buffer).
When --toggle option is specified pre-processor sends the toggle
signal to the target process. Then it either opens a named pipe and
starts listening for trace data (non-managed mode) or simply exits
(managed mode).
2.3 Data post-processor (rtrace-postproc)
The post-processor provides binary data conversion to human readable text format
and additional data post-processing. It also can parse the text format files,
allowing it to process its own generated output.
If the --resolve option is specified, the post-processor spawns the resolver
(rtrace-resolve) process and pipes the output data to it for name resolving.
The data processing steps are:
1) parse input file and create internal data structures,
2) apply post-processing filters,
3) generate output data.
2.3.1 Internal data storage
The memory mapping, context registry, comments, function calls
are stored into double linked lists.
The function traces are stored into hash table and shared between
similar calls. Every function trace record holds double linked
list of the function call references.
2.3.2 Post-processing
Currently two post-processing options are implemented - leak filtering
and backtrace compressing.
2.3.2.1 leak filtering (--filter-leaks)
This option removes allocation records of resources that was later
deallocated. The deallocation records also are removed, leaving only
allocations of the leaked resources. To speed up the resource lookup
the resource identifiers are indexed in hash table during freed
allocation removal.
2.3.2.2 backtrace compressing (--compress)
This option groups function calls by their backtraces and for all
functions with the same backtrace only one is printed.
2.3.3.3 future post-processing options
A plugin system could be added to post-processor allowing users to
write their own custom plugins.
2.4 Name resolver (rtrace-resolve)
The resolver processes the input file by replacing backtrace
addresses with resolved names. To resolve an address a memory
mapping record covering it must be located before the address.
3 Tracing configuration
The tracing module configuration is done with environment
variables (usually set by the rtrace launcher and other helper
scripts).
* SP_RTRACE_OUTPUT_DIR
Specifies the output directory for rtrace logs. By default
the home directory is used.
* SP_RTRACE_TOGGLE_SIGNAL
Specifies the trace toggling signal number. By default SIGUSR1
is used.
* SP_RTRACE_START
Defines that trace should be started automatically when the target
process is spawned.
* SP_RTRACE_MANAGE_PREPROC
Defines that the main trace module must spawn pre-processor
process when necessary. By default sp-rtrace is spawned externally
and the main module just connects to its named pipe.
* SP_RTRACE_BACKTRACE_DEPTH
Specifies the maximum number of frames listed in the backtraces.
* SP_RTRACE_DISABLE_TIMESTAMP
Disables timestamps reported by the tracing modules.
* SP_RTRACE_ENABLE_ARGUMENTS
Enables sending function argument (FA) packets.
* SP_RTRACE_POSTPROC
Specifies post-processing options.
* SP_RTRACE_DISABLE_EVENT_BUFFERING
Disables data buffering in main module and pre-processor.
4 Trace data flow
The usual data flow is:
libsp-rtrace-<module>.so
-> libsp-rtrace-main.so
-> sp-rtrace
-> sp-rtrace-postproc
-> sp-rtrace-resolve
Depending on sp-rtrace options used to invoke the traced process,
the sp-rtrace instance that actually collects the data can be:
* that sp-rtrace instance
(-s option given without -m)
* sp-rtrace instance run later by user to toggle tracing
(when neither -s nor -m options were given)
* sp-rtrace instance invoked by the main module
(-m option given, regardless of whether -s was given)
5 Visualization tools
There are several resource trace visualization tools.
Binaries:
* sp-rtrace-pagemap
ASCII graphs for all the writable memory mappings the traced process
has, what kind of memory page types they have and how much of these
pages are utilized by the traced (non-freed) allocations. Trace
data needs to contain both pagemap and memory modules output.
* sp-rtrace-timeline
o allocation timelines:
+ non-freed allocation total + alloc overhead
+ alloc & free counts
+ resource lifetimes
o histograms:
+ allocation size vs. freed & non-freed allocation count
+ allocation size vs. freed & non-freed allocation amount
Scripts:
* rtrace-calltree
generate .dot callgraphs from the trace data, used by
rtrace-graph-* scripts
* rtrace-graphs-overview
o overview alloc/dealloc timeline and allocation count/size
histogram charts (using sp-rtrace-timeline)
o callgraphs to resolved functions which are allocating
non-freed resources (using rtrace-calltree) with (hopefully)
irrelevant functions removed from the graph to make it readable
* rtrace-graphs-function
create callgraph(s) of allocations going through given function
* rtrace-from-function
list amount of allocations from the given function
6 Notes
6.1 Limitations
Binary data is portable only between machines having same endianess
and pointer size. If you want to process e.g. 32-bit ARM trace data
on a 64-bit x86 machine, convert the trace data first to ASCII format.
6.1 Scratchbox (Maemo)
In scratchbox (arm) to launch process with enabled tracing the managed mode
must be used. Otherwise the pre-processor will create named pipe with
emulation process id while the main module will try to connect to a named
pipe created with its (target) process id.
6.2 Debugging
It's possible to enable additional debug output (mainly used for rtrace
debugging) with --enable-debug-info configure option. Note that log functions
don't use async signal safe functions and therefore toggling trace could
lead to corruption.