4 **barectf** is a command-line utility which generates pure C99
5 code that is able to write native
6 [CTF](http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.txt;hb=master)
7 (the Common Trace Format) out of a pre-written CTF metadata file.
9 You will find barectf interesting if:
11 1. You need to trace a program.
12 2. You need tracing to be as fast as possible, but also very flexible:
13 record integers of custom sizes, custom floating point numbers,
14 enumerations mapped to a specific integer type, structure fields,
15 NULL-terminated strings, static and dynamic arrays, etc.
16 3. You need to be able to convert the recorded binary events to
17 human-readable text, as well as analyze them with Python scripts
18 ([Babeltrace](http://www.efficios.com/babeltrace) does all that,
20 4. You _cannot_ use [LTTng](http://lttng.org/), an efficient tracing
21 framework for the Linux kernel and Linux/BSD user applications, which
24 The target audience of barectf is developers who need to trace bare metal
25 systems (without an operating system). The code produced by barectf
26 is pure C99 and is lightweight enough to fit on a tiny microcontroller.
27 Each event described in the CTF metadata input becomes one C function with
28 one parameter mapped to one event field. CTF data is recorded in a buffer of
29 any size provided by the user. This buffer corresponds to one CTF packet.
30 The generated tracing functions report when the buffer is full. The user
31 is entirely responsible for the buffering scheme: leave the buffer in memory,
32 save it to some permanent storage, swap it with another empty buffer and
33 concatenate recorded packets, etc.
35 barectf is written in Python 3 and currently uses
36 [pytsdl](https://github.com/efficios/pytsdl) to parse the CTF metadata file
43 Make sure you have `pip` for Python 3. On the latest Ubuntu releases,
46 sudo apt-get install python3-pip
48 On Ubuntu 12.04, you need to install `setuptools` first, then use
49 `easy_install3` to install `pip3`:
51 sudo apt-get install python3-setuptools
52 sudo easy_install3 pip
56 sudo pip3 install barectf
62 Using barectf involves:
64 1. Writing the CTF metadata file describing the various headers,
65 contexts and event fields.
66 2. Running the `barectf` command to generate C99 files out of
67 the CTF metadata file.
68 3. Using the generated C code in your specific application.
70 The following subsections explain the three steps above.
72 Also, have a look at the [`examples`](examples) directory which
73 contains a few complete examples.
76 ### writing the CTF metadata
78 The **Common Trace Format** is a specialized file format for recording
79 trace data. CTF is designed to be very fast to write and very flexible.
80 All headers, contexts and event fields written in binary files are
81 described using a custom C-like, declarative language, TSDL (Trace
82 Stream Description Language). The file containing this description is
83 called the **CTF metadata**. The latter may be automatically generated
84 by a tracer, like it is the case of LTTng, or written by hand. This
85 metadata file is then used by CTF trace readers to know the layout of
86 CTF binary files containing actual event contexts and fields.
88 The CTF metadata file contains several blocks describing various CTF
89 binary layouts. A CTF trace file is a concatenation of several CTF
90 packets. Here's the anatomy of a CTF packet:
92 ![CTF packet anatomy](doc/ctf-packet.png)
94 A CTF packet belongs to a specific CTF stream. While the packet header
95 is the same for all streams of a given CTF trace, everything else is
96 specified per stream. Following this packet header is a packet context,
97 and then actual recorded events. Each event starts with a mandatory
98 header (same event header for all events of a given stream). The event
99 header is followed by an optional event context with a layout shared
100 by all events of a given stream. Then follows another optional event
101 context, although this one has a layout specific to the event type.
102 Finally, event fields are written.
104 barectf asks you to write the CTF metadata by hand. Although its official
105 [specification](http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.txt;hb=master)
106 is thorough, you will almost always start from this template:
111 /* a few useful standard integer aliases */
112 typealias integer {size = 8; align = 8;} := uint8_t;
113 typealias integer {size = 16; align = 16;} := uint16_t;
114 typealias integer {size = 32; align = 32;} := uint32_t;
115 typealias integer {size = 64; align = 64;} := uint64_t;
116 typealias integer {size = 8; align = 8; signed = true;} := int8_t;
117 typealias integer {size = 16; align = 16; signed = true;} := int16_t;
118 typealias integer {size = 32; align = 32; signed = true;} := int32_t;
119 typealias integer {size = 64; align = 64; signed = true;} := int64_t;
121 /* IEEE 754 standard-precision floating point alias */
122 typealias floating_point {
128 /* IEEE 754 double-precision floating point alias */
129 typealias floating_point {
137 /* CTF version 1.8; leave this as is */
142 * Native byte order (`le` or `be`). This is used by barectf to generate
143 * the appropriate code when writing data to the packet.
148 * Packet header. All packets (buffers) will have the same header.
150 * Special fields recognized by barectf (must appear in this order):
152 * magic: will be set to CTF's magic number (must be the first field)
153 * (32-bit unsigned integer) (mandatory)
154 * stream_id: will be set to the ID of the stream associated with
155 * this packet (unsigned integer of your choice) (mandatory)
157 packet.header := struct {
163 /* environment variables; you may add custom entries */
166 tracer_name = "barectf";
169 tracer_patchlevel = 0;
172 /* clock descriptor */
177 /* clock frequency (Hz) */
180 /* optional clock value offset; offset from Epoch is: offset * (1 / freq) */
184 /* alias for integer used to hold clock cycles */
188 /* map to the appropriate clock using its name */
189 map = clock.my_clock.value;
193 * A stream. You may have as many streams as you want. Events are unique
194 * within their own stream. The main advantage of having multiple streams
195 * is having different event headers, stream event contexts and stream
196 * packet contexts for each one.
200 * Mandatory stream ID (must fit the integer type of
201 * `trace.packet.header.stream_id`).
206 * Mandatory packet context. This structure follows the packet header
207 * (see `trace.packet.header`) immediately in CTF binary streams.
209 * Special fields recognized by barectf:
211 * timestamp_begin: will be set to the current clock value when opening
212 * the packet (same integer type as the clock's value)
213 * timestamp_end: will be set to the current clock value when closing
214 * the packet (same integer type as the clock's value)
215 * content_size: will be set to the content size, in bits, of this
216 * stream (unsigned 32-bit or 64-bit integer) (mandatory)
217 * packet_size: will be set to the packet size, in bits, of this
218 * stream (unsigned 32-bit or 64-bit integer) (mandatory)
219 * cpu_id: if present, the barectf_open_packet() function of this
220 * stream will accept an additional parameter to specify the
221 * ID of the CPU associated with this stream (a given
222 * stream should only be written to by a specific CPU)
223 * (unsigned integer of your choice)
225 * `timestamp_end` must be present if `timestamp_begin` exists.
227 packet.context := struct {
228 my_clock_int_t timestamp_begin;
229 my_clock_int_t timestamp_end;
230 uint64_t content_size;
231 uint64_t packet_size;
236 * Mandatory event header. All events recorded in this stream will start
237 * with this structure.
239 * Special fields recognized by barectf:
241 * id: will be filled by the event ID corresponding to a tracing
242 * function (unsigned integer of your choice)
243 * timestamp: will be filled by the current clock's value (same integer
244 * type as the clock's value)
246 event.header := struct {
248 my_clock_int_t timestamp;
252 * Optional stream event context (you may remove the whole block or leave
253 * the structure empty if you don't want any). This structure follows the
254 * event header (see `stream.event.header`) immediately in CTF binary
257 event.context := struct {
258 int32_t _some_stream_event_context_field;
263 * An event. Events have an ID, a name, an optional context and fields. An
264 * event is associated to a specific stream using its stream ID.
268 * Mandatory event name. This is used by barectf to generate the suffix
269 * of this event's corresponding tracing function, so make sure it follows
270 * the C identifier syntax even though it's a quoted string here.
275 * Mandatory event ID (must fit the integer type of in
276 * `stream.event.header.id` of the associated stream).
280 /* ID of the stream in which this event will be recorded */
284 * Optional event context (you may remove the whole block or leave the
285 * structure empty if you don't want one). This structure follows the
286 * stream event context (if it exists) immediately in CTF binary streams.
289 int32_t _some_event_context_field;
293 * Mandatory event fields (although the structure may be left empty if this
294 * event has no fields). This structure follows the event context (if it
295 * exists) immediately in CTF binary streams.
306 The top `/* CTF 1.8 */` is actually needed right there, and as is, since it
307 acts as a CTF metadata magic number for CTF readers.
309 Only one stream and one event (belonging to this single stream) are described
310 in this template, but you may add as many as you need.
312 The following subsections describe the features of CTF metadata supported
318 The supported structure field types are:
320 * **integers** of any size (64-bit and less), any alignment (power of two)
321 * **floating point numbers** of any total size (64-bit and less), any
322 alignment (power of two)
323 * NULL-terminated **strings** of bytes
324 * **enumerations** associated with a specific integer type
325 * **static** and **dynamic arrays** of any type
326 * **structures** containing only integers, floating point numbers,
327 enumerations and _static_ arrays
329 CTF also supports _variants_ (dynamic selection between different types),
330 but barectf **does not**. Any detected variant will throw an error when
336 CTF integers are defined like this:
340 /* mandatory size in bits (64-bit and less) */
344 * Optional alignment in bits (power of two). Default is 8 when the
345 * size is a multiple of 8, and 1 otherwise.
349 /* optional signedness (`true` or `false`); default is unsigned */
353 * Optional byte order (`le`, `be`, `native` or `network`). `native`
354 * will use the byte order specified by the `trace.byte_order`.
355 * Default is `native`.
360 * Optional display base, used to display the integer value when
361 * reading the trace. Valid values are 2 (or `binary`, `bin` and `b`),
362 * 8 (or `o`, `oct` or `octal`), 10 (or `u`, `i`, `d`, `dec` or
363 * `decimal`), and 16 (or `x`, `X`, `p`, `hex` or `hexadecimal`).
369 * Encoding (if this integer represents a character). Valid values
370 * are `none`, `UTF8` and `ASCII`. Default is `none`.
376 The size (the only mandatory property) does _not_ have to be a power of two:
384 A CTF integer field will make barectf produce a corresponding C integer
385 function parameter with an appropriate size. For example, the 23-bit integer
386 above would produce an `uint32_t` parameter (of which only the first 23
387 least significant bits will be written to the trace), while the first
388 16-bit one will produce an `int16_t` parameter.
390 The `integer` block also accepts a `map` property which is only used
391 when defining the integer used to carry the value of a specified
392 clock. You may always follow the example above.
395 ##### floating point numbers
397 CTF floating point numbers are defined like this:
401 /* exponent size in bits */
404 /* mantissa size in bits */
408 * Optional alignment (power of two). Default is 8 when the total
409 * size (exponent + mantissa) is a multiple of 8, and 1 otherwise.
414 * Optional byte order (`le`, `be`, `native` or `network`). `native`
415 * will use the byte order specified by the `trace.byte_order`.
416 * Default is `native`.
422 If a CTF floating point number is defined with an 8-bit exponent, a 24-bit
423 mantissa and a 32-bit alignment, its barectf C function parameter type will
424 be `float`. It will be `double` for an 11-bit exponent, 53-bit mantissa
425 and 64-bit aligned CTF floating point number. Any other configuration
426 will produce a `uint64_t` function parameter (you will need to cast your
427 custom floating point number to this when calling the tracing function).
432 CTF strings are pretty simple to define:
438 They may also have an encoding property:
442 /* encoding: `none`, `UTF8` or `ASCII`; default is `none` */
447 CTF strings are always byte-aligned.
449 A CTF string field will make barectf produce a corresponding C function
450 parameter of type `const char*`. Bytes will be copied from this pointer
451 until a byte of value 0 is found (which will also be written to the
452 buffer to mark the end of the recorded string).
457 CTF enumerations associate labels to ranges of integer values. They
458 are a great way to trace named states using an integer. Here's an
473 Unless the first entry specifies a value, CTF enumerations are
474 always started at 0. They work pretty much like their C counterpart,
475 although they support ranges and literal strings as labels.
477 CTF enumerations are associated with a CTF integer type (`uint32_t`
478 above). This identifier must be an existing integer type alias.
480 A CTF enumeration field will make barectf produce a corresponding C
481 integer function parameter compatible with the associated CTF integer type.
486 Structure field names may be followed by a subscripted constant to
487 define a static array of the field type:
491 integer {size = 16;} _field[10];
495 In the above structure, `_field` is a static array of ten 16-bit integers.
497 A CTF static array field will make barectf produce a `const void*` C function
498 parameter. Bytes will be copied from this pointer to match the total static
499 array size. In the example above, the integer size is 16-bit, thus its
500 default alignment is 8-bit, so 20 bytes would be copied.
502 The inner element of a CTF static array _must be at least byte-aligned_
503 (8-bit), either by forcing its alignment, or by ensuring it manually
504 when placing fields one after the other. This means the following static
505 array is valid for barectf:
510 integer {size = 5;} _field[10];
514 as long as the very first 5-bit, 1-bit aligned integer element starts
515 on an 8-bit boundary.
520 Just like static arrays, dynamic arrays are defined using a subscripted
521 length, albeit in this case, this length refers to another field using
522 the dot notation. Dynamic arrays are called _sequences_ in the CTF
530 integer {size = 16;} _field[_length];
534 In the above structure, `_field` is a dynamic array of `_length`
537 There are various scopes to which a dynamic array may refer:
539 * no prefix: previous field in the same structure, or in parent
540 structures until found
541 * `event.fields.` prefix: field of the event fields
542 * `event.context.` prefix: field of the event context if it exists
543 * `stream.event.context.` prefix: field of the stream event context
545 * `stream.event.header.` prefix: field of the event header
546 * `stream.packet.context.` prefix: field of the packet context
547 * `trace.packet.header.` prefix: field of the packet header
548 * `env.` prefix: static property of the environment block
550 Here's another, more complex example:
555 string _other_field[stream.event.context.length];
556 float _static_array_of_dynamic_arrays[10][_length];
560 The above examples also demonstrates that dynamic arrays and static
561 arrays may contain eachother. `_other_field` is a dynamic array of
562 `stream.event.context.length` strings. `_static_array_of_dynamic_arrays`
563 is a static array of 10 dynamic arrays of `_length` floating point
564 numbers. This syntax follows the C language.
566 A CTF dynamic array field will make barectf produce a `const void*` C function
567 parameter. Bytes will be copied from this pointer to match the
568 total dynamic array size. The previously recorded length will be
569 found automatically (always an offset from the beginning of the
570 stream packet, or from the beginning of the current event).
572 barectf has a few limitations concerning dynamic arrays:
574 * The inner element of a CTF dynamic array _must be at least byte-aligned_
575 (8-bit), either by forcing its alignment, or by ensuring it manually
576 when placing fields one after the other.
577 * The length type must be a 32-bit, byte-aligned unsigned integer
578 with a native byte order.
583 Structures contain fields associating a name to a type. The fields
584 are recorded in the specified order within the CTF binary stream.
592 string {encoding = ASCII;} _c;
596 The default alignment of a structure is the largest alignment amongst
597 its fields. For example, the following structure has a 32-bit alignment:
601 uint16_t _a; /* alignment: 16 */
602 struct { /* alignment: 32 */
603 uint32_t _a; /* alignment: 32 */
604 string; _b; /* alignment: 8 */
606 integer {size = 64;} _c; /* alignment: 8 */
610 This default alignment may be overridden using a special `align()`
611 option after the structure is closed:
620 integer {size = 64;} _c;
624 You may use structures as field types, although they must have a
625 _known size_ when running barectf. This means they cannot contain
626 sequences or strings.
628 A CTF structure field will make barectf produce a `const void*` C function
629 parameter. The structure (of known size) will be copied as is to the
630 current buffer, respecting its alignment.
632 Note that barectf requires inner structures to be at least byte-aligned.
634 Be careful when using CTF structures for recording binary structures
635 declared in C. You need to make sure your C compiler aligns structure
636 fields and adds padding exactly in the way you define your equivalent
637 CTF structure. For example, using GCC on the x86 architecture, 3 bytes
638 are added after field `a` in the following C structure since `b` is
648 It would be wrong to use the following CTF structure:
652 integer {size = 8; signed = true;} a;
653 integer {size = 32;} b;
657 since field `b` is byte-aligned by default. This one would work fine:
661 integer {size = 8; signed = true;} a;
662 integer {size = 32; align = 32;} b;
666 CTF structures can prove very useful for recording protocols with named
667 fields when reading the trace. For example, here's the CTF structure
668 describing the IPv4 header (excluding options):
672 integer {size = 4;} version;
673 integer {size = 4;} ihl;
674 integer {size = 6;} dscp;
675 integer {size = 2;} ecn;
676 integer {size = 16; byte_order = network;} total_length;
677 integer {size = 16; byte_order = network;} identification;
678 integer {size = 1;} flag_more_fragment;
679 integer {size = 1;} flag_dont_fragment;
680 integer {size = 1;} flag_reserved;
681 integer {size = 13; byte_order = network;} fragment_offset;
682 integer {size = 8;} ttl;
683 integer {size = 8;} protocol;
684 integer {size = 16; byte_order = network;} header_checksum;
685 integer {size = 8;} src_ip_addr[4];
686 integer {size = 8;} dst_ip_addr[4];
690 Although this complex structure has more than ten independent fields,
691 the generated C function would only call a 20-byte `memcpy()`, making
692 it fast to record. Bits will be unpacked properly and values displayed
693 in a human-readable form by the CTF reader thanks to the CTF metadata.
698 Type aliases associate a name with a type definition. Any type may have
699 any name. They are similar to C `typedef`s.
708 byte_order = network;
715 typealias floating_point {
730 typealias enum : uint32_t {
745 float _hello[10][_length];
746 } align(8) := my_struct;
750 ### running the `barectf` command
752 Using the `barectf` command-line utility is easy. In its simplest form,
753 it outputs a few C99 files out of a CTF metadata file:
757 will output in the current working directory:
759 * `barectf_bitfield.h`: macros used by tracing functions to pack bits
760 * `barectf.h`: other macros and prototypes of context/tracing functions
761 * `barectf.c`: context/tracing functions
763 You may also want to produce `static inline` functions if your target
764 system has enough memory to hold the extra code:
766 barectf --static-inline metadata
768 `barectf` is the default name of the files and the default prefix of
769 barectf C functions and structures. You may use a custom prefix:
771 barectf --prefix trace metadata
773 You may also output the files elsewhere:
775 barectf --output /custom/path metadata
777 ### using the generated C99 code
779 This section assumes you ran `barectf` with no options:
783 The command generates C99 structures and functions to initialize
784 and finalize bare CTF contexts. It also generates as many tracing functions
785 as there are events described in the CTF metadata file.
787 Before starting the record events, you must initialize a barectf
788 context. This is done using `barectf_init()`.
790 The clock callback parameter (`clock_cb`) is used to get the clock whenever
791 a tracing function is called. Each platform has its own way of obtaining
792 the a clock value, so this is left to user implementation. The actual
793 return type of the clock callback depends on the clock value CTF integer
794 type defined in the CTF metadata.
796 The `barectf_init()` function name will contain the decimal stream
797 ID if you have more than one stream. You must allocate the context
803 struct barectf_ctx* barectf_ctx = platform_alloc(sizeof(*barectf_ctx));
805 barectf_init(barectf_ctx, buf, 8192, platform_get_clock, NULL);
808 This initializes a barectf context with a buffer of 8192 bytes.
810 After the barectf context is initialized, open a packet using
811 `barectf_open_packet()`. If you have any non-special fields in
812 your stream packet context, `barectf_open_packet()` accepts a
813 parameter for each of them since the packet context is written
817 barectf_open_packet(barectf_ctx);
820 Once the packet is opened, you may call any of the tracing functions to record
821 CTF events into the context's buffer.
823 As an example, let's take the following CTF event definition:
831 integer {size = 32;} _a;
832 integer {size = 14; signed = true;} _b;
833 floating_point {exp_dig = 8; mant_dig = 24; align = 32;} _c;
843 In this example, we assume the stream event context and the event context
844 are not defined for this event. `barectf` generates the following tracing
848 int barectf_trace_my_event(
849 struct barectf_ctx* ctx,
850 uint32_t param_ef__a,
853 const void* param_ef__d,
854 const char* param_ef__e
858 When called, this function first calls the clock callback to get a clock
859 value as soon as possible. It then proceeds to record each field with
860 proper alignment and updates the barectf context. On success, 0 is returned.
861 Otherwise, one of the following negative errors is returned:
863 * `-EBARECTF_NOSPC`: no space left in the context's buffer; the event
864 was **not** recorded. You should call `barectf_close_packet()` to finalize the
867 `barectf_close_packet()` may be called at any time.
868 When `barectf_close_packet()` returns, the packet is complete and ready
869 to be read by a CTF reader. CTF packets may be concatenated in a single
870 CTF stream file. You may reuse the same context and buffer to record another
871 CTF packet, as long as you call `barectf_open_packet()` before calling any
875 ### reading CTF traces
877 To form a complete CTF trace, put your CTF metadata file (it should be
878 named `metadata`) and your binary stream files (concatenations of CTF
879 packets written by C code generated by barectf) in the same directory.
881 To read a CTF trace, use [Babeltrace](http://www.efficios.com/babeltrace).
882 Babeltrace is packaged by most major distributions (`babeltrace`).
883 Babeltrace ships with a command-line utility that can convert a CTF trace
884 to human-readable text output. Also, it includes a Python binding so
885 that you may analyze a CTF trace using a custom script.
887 In its simplest form, the `babeltrace` command-line converter is quite
890 babeltrace /path/to/directory/containing/ctf/files
892 See `babeltrace --help` for more options.
894 You may also use the Python 3 binding of Babeltrace to create custom