Add CHANGELOG.md

[barectf.git] / README.md

# barectf

[![](https://img.shields.io/pypi/v/barectf.svg)](https://pypi.python.org/pypi/barectf)

**barectf** is a command-line utility which generates pure C99
code that is able to write native [Common Trace Format](http://diamon.org/ctf)
(CTF) binary streams.

You will find barectf interesting if:

  1. You need to trace an application.
  2. You need tracing to be efficient, yet flexible:
     record integers of custom sizes, custom floating point numbers,
     enumerations supported by a specific integer type, and
     null-terminated UTF-8/ASCII strings (C strings).
  3. You need to be able to convert the recorded binary events to
     human-readable text, as well as analyze them with Python scripts
     ([Babeltrace](http://www.efficios.com/babeltrace) does all that,
     given a CTF input).
  4. You _cannot_ use [LTTng](http://lttng.org/), an efficient tracing
     framework for the Linux kernel and Linux/BSD user applications, which
     also outputs CTF.

The target audience of barectf is developers who need to trace bare metal
systems (without an operating system). The code produced by barectf
is pure C99 and can be lightweight enough to fit on a tiny microcontroller.

**Key features**:

  * Single input: easy-to-write [YAML](https://en.wikipedia.org/wiki/YAML)
    configuration file (documentation below)
  * 1-to-1 mapping from tracing function parameters to event fields
  * Custom and bundled _platforms_ hiding the details of opening/closing
    packets and writing them to a back-end (continuous tracing), getting
    the clock values, etc.:
    * _linux-fs_: basic Linux application tracing writing stream files to
      the file system for demonstration purposes
    * _parallella_: Adapteva Epiphany/[Parallella](http://parallella.org/)
      with host-side consumer
  * CTF metadata generated by the command-line tool (automatic trace UUID,
    stream IDs, and event IDs)
  * All basic CTF types are supported: integers, floating point numbers,
    enumerations, and null-terminated strings (C strings)
  * Binary streams produced by the generated C code and metadata file
    produced by barectf are CTF 1.8-compliant
  * Human-readable error reporting

**Current limitations**:

As of this version:

  * All the generated tracing C functions, for a given barectf
    stream-specific context, need to be called from the same thread, and cannot
    be called from an interrupt handler, unless a user-provided
    synchronization mechanism is used.
  * CTF compound types (array, sequence, structure, variant) are not supported
    yet, except at some very specific locations in the metadata.

barectf is written in Python 3.


## Installing

Make sure you have Python 3 and `pip` for Python 3 installed, then
install barectf.

Note that you may pass the `--user` argument to
`pip install` to install the tool in your home directory (instead of
installing globally).

**Latest Ubuntu**:

    sudo apt-get install python3-pip
    sudo pip3 install barectf

**Ubuntu 12.04 and lower**:

    sudo apt-get install python3-setuptools
    sudo easy_install3 pip
    sudo pip3 install barectf

**Debian**:

    sudo apt-get install python3-pip
    sudo pip3 install barectf

**Fedora 20 and up**:

    sudo yum install python3-pip
    sudo pip3 install barectf

**Arch Linux**:

    sudo pacman -S python-pip
    sudo pip install barectf

**OS X**

With [Homebrew](http://brew.sh/):

    brew install python3
    pip3 install barectf


## What is CTF?

See the [CTF in a nutshell](http://diamon.org/ctf/#ctf-in-a-nutshell)
section of CTF's website to understand the basics of this
trace format.

The most important thing to understand about CTF, for barectf use
cases, is the layout of a binary stream packet:

  * Packet header (defined at the trace level)
  * Packet context (defined at the stream level)
  * Sequence of events (defined at the stream level):
    * Event header (defined at the stream level)
    * Stream event context (defined at the stream level)
    * Event context (defined at the event level)
    * Event payload (defined at the event level)

The following diagram, stolen without remorse from CTF's website, shows
said packet layout:

![](http://diamon.org/ctf/img/ctf-stream-packet.png)

Any of those six dynamic scopes, if defined at all, has an associated
CTF type. barectf requires them to be structure types.


## Using

Using barectf involves the following steps:

  1. Writing the YAML configuration file defining the various header,
     context, and event field types.
  2. Running the `barectf` command-line tool with this configuration file
     to generate the CTF metadata and C files.
  3. Using the generated C code (tracing functions), along with the C code
     provided by the appropriate barectf platform, in the source code of
     your own application.
  4. Running your application, along with anything the barectf platform
     you chose requires, to generate the binary streams of a CTF trace.

Your application, when running, will generate CTF packets. Depending
on the chosen barectf platform, those packets will be consumed and
sequentially written at some place for later viewing/analysis.

Here's a diagram summarizing the steps described above:

![](http://0x3b.org/ss/cardiectasis400.png)

The following subsections explain the four steps above.

Also, have a look at the [`doc/examples`](doc/examples) directory, which
contains complete examples.


### Writing the YAML configuration file

The barectf [YAML](https://en.wikipedia.org/wiki/YAML) configuration file
is the only input the `barectf` command-line tool needs in order to generate
the corresponding CTF metadata and C files.

To start with a concrete configuration, here's some minimal configuration:

```yaml
version: '2.0'
metadata:
  type-aliases:
    uint16:
      class: int
      size: 16
  trace:
    byte-order: le
  streams:
    my_stream:
      packet-context-type:
        class: struct
        fields:
          packet_size: uint16
          content_size: uint16
      events:
        my_event:
          payload-type:
            class: struct
            fields:
              my_field:
                class: int
                size: 8
```

The `version` property must be set to the `2.0` _string_ (hence the single
quotes). As features are added to barectf and to its configuration file schema,
this version will be bumped accordingly.

The `metadata` property is where the properties and layout of the
eventual CTF trace are defined. The accepted properties of each object
are documented later in this document. For the moment, note simply
that the native byte order of the trace is set to `le` (little-endian),
and that there's one defined stream named `my_stream`, having one
defined event named `my_event`, having a structure as its payload
type, with a single 8-bit unsigned integer type field named `my_field`. Also,
the stream packet context type is a structure defining the mandatory
`packet_size` and `content_size` special fields as 16-bit unsigned integer
types.

Running `barectf` with the configuration above (as a file named `config.yaml`):

    barectf config.yaml

will produce a C file (`barectf.c`), and its header file (`barectf.h`),
the latter declaring the following function:

```c
void barectf_my_stream_trace_my_event(
    struct barectf_my_stream_ctx *ctx, uint8_t ep_my_field);
```

`ctx` is the barectf context for the stream named `my_stream` (usually
initialized and provided by the barectf platform), and `ep_my_field` is the
value of the `my_event` event payload's `my_field` field.

The following subsections define all the objects of the YAML configuration
file.


#### Configuration object

The top-level object of the YAML configuration file.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `version` | String | Must be set to `'2.0'` | Required | N/A |
| `prefix` | String | Prefix to be used for function names, file names, etc. | Optional | `barectf_` |
| `metadata` | [Metadata object](#metadata-object) | Trace metadata | Required | N/A |

The `prefix` property must be set to a valid C identifier. It can be
overridden by the `barectf` command-line tool's `--prefix` option.

**Example**:

```yaml
version: '2.0'
prefix: axx_
metadata:
  type-aliases:
    uint16:
      class: int
      size: 16
  trace:
    byte-order: le
  streams:
    my_stream:
      packet-context-type:
        class: struct
        fields:
          packet_size: uint16
          content_size: uint16
      events:
        my_event:
          payload-type:
            class: struct
            fields:
              a:
                class: int
                size: 8
```


#### Metadata object

A metadata object defines the desired layout of the CTF trace to be
produced by the generated C code. It is used by barectf to generate C code,
as well as a corresponding CTF metadata file.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `type-aliases` | Associative array of strings (alias names) to [type objects](#type-objects) or strings (previous alias names) | Type aliases to be used in trace, stream, and event objects | Optional | `{}` |
| `log-levels` | Associative array of strings (log level names) to log level constant integers | Log levels to be used in event objects | Optional | `{}` |
| `clocks` | Associative array of strings (clock names) to [clock objects](#clock-object) | Trace clocks | Optional | `{}` |
| `env` | Associative array of strings (names) to strings or integers (values) | Trace environment variables | Optional | `{}` |
| `trace` | [Trace object](#trace-object) | Metadata common to the whole trace | Required | N/A |
| `streams` | Associative array of strings (stream names) to [stream objects](#stream-object) | Trace streams | Required | N/A |

Each clock name of the `clocks` property must be a valid C identifier.

The `streams` property must contain at least one entry. Each stream name must be
a valid C identifier.

Each environment variable name in the `env` property must be a valid
C identifier. Those variables will be appended to some environment
variables set by barectf itself.

The order of the `type-aliases` entries is important: a type alias may only
inherit from another type alias if the latter is defined before.

**Example**:

```yaml
type-aliases:
  uint8:
    class: integer
    size: 8
  uint16:
    class: integer
    size: 16
  uint32:
    class: integer
    size: 32
  uint64:
    class: integer
    size: 64
  clock-int:
    inherit: uint64
    property-mappings:
      - type: clock
        name: my_clock
        property: value
  byte: uint8
  uuid:
    class: array
    length: 16
    element-type: byte
log-levels:
  emerg: 0
  alert: 1
  critical: 2
  error: 3
  warning: 4
  notice: 5
  info: 6
clocks:
  my_clock:
    freq: 1000000000
    offset:
      seconds: 1434072888
    return-ctype: uint64_t
env:
  my_system_version: '0.3.2-2015.03'
  bID: 15
trace:
  byte-order: le
  uuid: auto
  packet-header-type:
    class: struct
    min-align: 8
    fields:
      magic: uint32
      uuid: uuid
      stream_id: uint8
streams:
  my_stream:
    packet-context-type:
      class: struct
      fields:
        timestamp_begin: clock-int
        timestamp_end: clock-int
        packet_size: uint32
        something: float
        content_size: uint32
        events_discarded: uint32
    event-header-type:
      class: struct
      fields:
        timestamp: clock-int
        id: uint16
    events:
      simple_uint32:
        log-level: error
        payload-type:
          class: struct
          fields:
            value: uint32
      simple_int16:
        payload-type:
          class: struct
          fields:
            value:
              inherit: uint16
              signed: true
```


#### Clock object

A CTF clock.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `freq` | Integer (positive) | Frequency (Hz) | Optional | 1000000000 |
| `description` | String | Description | Optional | No description |
| `uuid` | String (UUID canonical format) | UUID (unique identifier of this clock) | Optional | No UUID |
| `error-cycles` | Integer (zero or positive) | Error (uncertainty) of clock in clock cycles | Optional | 0 |
| `offset` | [Clock offset object](#clock-offset-object) | Offset | Optional | Default clock offset object |
| `absolute` | Boolean | Absolute clock | Optional | `false` |
| `return-ctype` | String | Return C type of the associated clock callback | Optional | `uint32_t` |

The `return-ctype` property must be set to a valid C integer type
(or valid type definition). This is not currently validated by barectf
itself, but the C compiler will fail to compile the generated C code
if the clock's return type is not a valid C integer type.

**Example**:

```yaml
freq: 2450000000
description: CCLK/A2 (System clock, A2 clock domain)
uuid: 184883f6-6b6e-4bfd-bcf7-1e45c055c56a
error-cycles: 23
offset:
  seconds: 1434072888
  cycles: 2003912
absolute: false
return-ctype: unsigned long long
```


##### Clock offset object

An offset in seconds and clock cycles from the Unix epoch.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `seconds` | Integer (zero or positive) | Seconds since the Unix epoch | Optional | 0 |
| `cycles` | Integer (zero or positive) | Clock cycles since the Unix epoch plus the value of the `seconds` property | Optional | 0 |

**Example**:

```yaml
seconds: 1435617321
cycles: 194570
```


#### Trace object

Metadata common to the whole trace.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `byte-order` | String | Native byte order (`le` for little-endian or `be` for big-endian) | Required | N/A |
| `uuid` | String (UUID canonical format or `auto`) | UUID (unique identifier of this trace); automatically generated if value is `auto` | Optional | No UUID |
| `packet-header-type` | [Type object](#type-objects) or string (alias name) | Type of packet header (must be a [structure type object](#structure-type-object)) | Optional | No packet header |

Each field of the packet header structure type (`packet-header-type` property)
corresponds to one parameter
of the generated packet opening function (prefixed with `tph_`), except for the
following special fields, which are automatically written if present:

  * `magic` (32-bit unsigned [integer type object](#integer-type-object)):
    packet magic number
  * `uuid` ([array type object](#array-type-object) of 8-bit unsigned
    [integer type objects](#integer-type-object), of length 16):
    trace UUID (`uuid` property of trace object must be set)
  * `stream_id` (unsigned [integer type object](#integer-type-object)):
    stream ID

As per CTF 1.8, the `stream_id` field is mandatory if there's more
than one defined stream.

**Example**:

```yaml
byte-order: le
uuid: auto
packet-header-type:
  class: struct
  fields:
    magic: uint32
    uuid:
      class: array
      length: 16
      element-type: uint8
    stream_id: uint16
```


#### Stream object

A CTF stream.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `packet-context-type` | [Type object](#type-objects) or string (alias name) | Type of packet context (must be a [structure type object](#structure-type-object)) | Required | N/A |
| `event-header-type` | [Type object]((#type-objects)) or string (alias name) | Type of event header (must be a [structure type object](#structure-type-object)) | Optional | No event header |
| `event-context-type` | [Type object]((#type-objects)) or string (alias name) | Type of stream event context (must be a [structure type object](#structure-type-object)) | Optional | No stream event context |
| `events` | Associative array of event names (string) to [event objects](#event-object) | Stream events | Required | N/A |

Each field of the packet context structure type (`packet-context-type` property)
corresponds to one parameter
of the generated packet opening function (prefixed with `spc_`), except for the
following special fields, which are automatically written if present:

  * `timestamp_begin` and `timestamp_end` (unsigned
    [integer type objects](#integer-type-object), with
    a clock value property mapping): resp. open and close timestamps
  * `packet_size` (unsigned [integer type object](#integer-type-object),
    mandatory): packet size
  * `content_size` (unsigned [integer type object](#integer-type-object),
    mandatory): content size
  * `events_discarded` (unsigned [integer type object](#integer-type-object)):
    number of discarded events so far

The `timestamp_end` field must exist if the `timestamp_begin` field exists,
and vice versa.

Each field of the event header structure type (`event-header-type` property)
corresponds to one parameter of the generated tracing function
(prefixed with `eh_`) (for a given event), except for the following special
fields, which are automatically written if present:

  * `id` (unsigned [integer type object](#integer-type-object)): event ID
  * `timestamp` (unsigned [integer type object](#integer-type-object), with
    a clock value property mapping): event timestamp

The `id` field must exist if there's more than one defined event in the
stream.

Each field of the stream event context structure type (`event-context-type`
property) corresponds to one parameter of the generated tracing function
(prefixed with `seh_`) (for a given event).

Each field name of the `packet-context-type`, `event-header-type`,
and `event-context-type` properties must be a valid C identifier.

The `events` property must contain at least one entry.

**Example**:

```yaml
packet-context-type:
  class: struct
  fields:
    timestamp_begin: clock-int
    timestamp_end: clock-int
    packet_size: uint32
    content_size: uint32
    events_discarded: uint16
    my_custom_field: int12
event-header-type:
  class: struct
  fields:
    id: uint16
    timestamp: clock-int
event-context-type:
  class: struct
  fields:
    obj_id: uint8
events:
  msg_in:
    payload-type: msg-type
```


#### Event object

A CTF event.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `log-level` | String (predefined log level name) or integer (zero or positive) | Log level of this event | Optional | No log level |
| `context-type` | [Type object](#type-objects) or string (alias name) | Type of event context (must be a [structure type object](#structure-type-object)) | Optional | No event context |
| `payload-type` | [Type object](#type-objects) or string (alias name) | Type of event payload (must be a [structure type object](#structure-type-object)) | Required | N/A |

Available log level names, for a given event, are defined by the
`log-levels` property of the [metadata object](#metadata-object)
containing it.

Each field of the event context structure type (`context-type` property)
corresponds to one parameter
of the generated tracing function (prefixed with `ec_`).

Each field of the event payload structure type (`payload-type` property)
corresponds to one parameter
of the generated tracing function (prefixed with `ep_`). The event
payload structure type must contain at least one field.

Each field name of the `context-type` and `payload-type` properties must be a
valid C identifier.

**Example**:

```yaml
log-level: error
context-type:
  class: struct
  fields:
    msg_id: uint16
payload-type:
  class: struct
  fields:
    src:
      type: string
    dst:
      type: string
    payload_sz: uint32
```


#### Type objects

Type objects represent CTF types.

**Common properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `class` | String | Type class | Required if `inherit` property is absent | N/A |
| `inherit` | String | Name of type alias from which to inherit properties | Required if `class` property is absent | N/A |

The accepted values for the `class` property are:

| `class` property value | CTF type |
|---|---|
| `int`<br>`integer` | Integer type |
| `flt`<br>`float`<br>`floating-point` | Floating point number type |
| `enum`<br>`enumeration` | Enumeration type |
| `str`<br>`string` | String type |
| `struct`<br>`structure` | Structure type |
| `array` | Array/sequence types |
| `var`<br>`variant` | Variant type |

The `inherit` property accepts the name of any previously defined
type alias. Any propery in a type object that inherits from another
type object overrides the parent properties as follows:

  * Booleans, numbers, and strings: value of parent property with
    the same name is replaced
  * Arrays: new elements are appended to parent array
  * Associative arrays: properties sharing the name of parent
    properties completely replace them; new properties are
    added to the parent associative array


##### Integer type object

A CTF integer type.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `size` | Integer (positive) | Size (bits) (1 to 64) | Required | N/A |
| `align` | Integer (positive) | Alignment (bits) (power of two) | Optional | 8 if `size` property is a multiple of 8, else 1 |
| `signed` | Boolean | Signedness | Optional | `false` (unsigned) |
| `base` | Integer | Display radix (2, 8, 10, or 16) | Optional | 10 |
| `byte-order` | String | Byte order (`le` for little-endian, `be` for big-endian, or `native` to use the byte order defined at the trace level) | Optional | `native` |
| `property-mappings` | Array of [property mapping objects](#property-mapping-object) | Property mappings of this integer type | Optional | N/A |

The `property-mappings` array property currently accepts only one element.

**Example**:

```yaml
class: int
size: 12
signed: false
base: 8
byte-order: le
property-mappings:
  - type: clock
    name: my_clock
    property: value
```

**Equivalent C type**:

  * Unsigned: `uint8_t`, `uint16_t`, `uint32_t`, or `uint64_t`, depending on the
    `size` property
  * Signed: `int8_t`, `int16_t`, `int32_t`, or `int64_t`, depending on the
    `size` property


###### Property mapping object

A property mapping object associates an integer type with a stateful
object's property. When the integer type is decoded from a CTF binary
stream, the associated object's property is updated.

Currently, the only available stateful object's property is the
current value of a given clock.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `type` | String | Object type (always `clock`) | Required | N/A |
| `name` | String | Clock name | Required | N/A |
| `property` | String | Clock property name (always `value`) | Required | N/A |

**Example**:

```yaml
type: clock
name: my_clock
property: value
```


##### Floating point number type object

A CTF floating point number type.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `size` | [Floating point number type size object](#floating-point-number-type-size-object) | Size parameters | Required | N/A |
| `align` | Integer (positive) | Alignment (bits) (power of two) | Optional | 8 |
| `byte-order` | String | Byte order (`le` for little-endian, `be` for big-endian, or `native` to use the byte order defined at the trace level) | Optional | `native` |

**Example**:

```yaml
class: float
size:
  exp: 11
  mant: 53
align: 64
byte-order: be
```

**Equivalent C type**:

  * 8-bit exponent, 24-bit mantissa, 32-bit alignment: `float`
  * 11-bit exponent, 53-bit mantissa, 64-bit alignment: `double`
  * Every other combination: `uint64_t`


###### Floating point number type size object

The CTF floating point number type is encoded, in a binary stream,
following [IEEE 754-2008](https://en.wikipedia.org/wiki/IEEE_floating_point)'s
interchange format. The required parameters are the exponent and
significand sizes, in bits. In CTF, the _mantissa_ size includes the
sign bit, whereas IEEE 754-2008's significand size does not include it.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `exp` | Integer (positive) | Exponent size (bits) | Required | N/A |
| `mant` | Integer (positive) | Mantissa size (significand size + 1) (bits) | Required | N/A |

As per IEEE 754-2008, the sum of the `exp` and `mant` properties must be a
multiple of 32.

The sum of the `exp` and `mant` properties must be lesser than or equal to 64.

**Example**:

```yaml
exp: 8
mant: 24
```


##### Enumeration type object

A CTF enumeration type.

Each label of an enumeration type is mapped to a single value, or to a
range of values.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `value-type` | [Integer type object](#integer-type-object) or string (alias name) | Supporting integer type | Required | N/A |
| `members` | Array of [enumeration type member objects](#enumeration-type-member-object) | Enumeration members | Required | N/A |

The `members` property must contain at least one element. If the member
is a string, its associated value is computed as follows:

  * If the member is the first one of the `members` array, its value
    is 0.
  * If the previous member is a string, its value is the previous
    member's computed value + 1.
  * If the previous member is a single value member, its value is
    the previous member's value + 1.
  * If the previous member is a range member, its value is the previous
    member's upper bound + 1.

The member values must not overlap each other.

**Example**:

```yaml
class: enum
value-type: uint8
members:
  - ZERO
  - ONE
  - TWO
  - label: SIX
    value: 6
  - SE7EN
  - label: TWENTY TO FOURTY
    value: [10, 40]
  - FORTY-ONE
```

**Equivalent C type**: equivalent C type of supporting integer type
(see [integer type object documentation](#integer-type-object) above).


###### Enumeration type member object

The member of a CTF enumeration type.

If it's a string, the string is the member's label, and the members's
value depends on the last member's value (see explanation in
[enumeration type object documentation](#enumeration-type-object) above).

Otherwise, it's a complete member object, with the following properties:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `label` | String | Member's label | Required | N/A |
| `value` | Integer (single value) or array of two integers (range value) | Member's value | Required | N/A |

If the `value` property is an array of two integers, the member's label is
associated to this range, both lower and upper bounds included. The array's
first element must be lesser than or equal to the second element.

**Example**:

```yaml
label: my enum label
value: [-25, 78]
```


##### String type object

A CTF null-terminated string type.

This object has no properties.

**Example**:

```yaml
class: string
```

**Equivalent C type**: `const char *`.


##### Array type object

A CTF array or sequence (variable-length array) type.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `element-type` | [Type object](#type-objects) or string (alias name) | Type of array's elements | Required | N/A |
| `length` | Positive integer (static array) or string (variable-length array) | Array type's length | Required | N/A |

If the `length` property is a string, the array type has a
variable length (CTF sequence). In this case, the property's value
refers to a previous structure field. The `length` property's value
may be prefixed with one of the following strings to indicate an
absolute lookup within a previous (or current) dynamic scope:

  * `trace.packet.header.`: trace packet header
  * `stream.packet.context.`: stream packet context
  * `stream.event.header.`: stream event header
  * `stream.event.context.`: stream event context
  * `event.context.`: event context
  * `event.payload.`: event payload

The pointed field must have an unsigned integer type.

**Example** (16 bytes):

```yaml
class: array
length: 16
element-type:
  class: int
  size: 8
```

**Example** (variable-length array of null-terminated strings):

```yaml
class: array
length: previous_field
element-type:
  class: string
```


##### Structure type object

A CTF structure type, i.e. a list of fields, each field
having a name and a CTF type.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `min-align` | Integer (positive) | Minimum alignment (bits) (power of two) | Optional | 1 |
| `fields` | Associative array of field names (string) to [type objects](#type-objects) or strings (alias names) | Structure type's fields | Optional | `{}` |

The order of the entries in the `fields` property is important; it is in
this order that the fields are serialized in binary streams.

**Example**:

```yaml
class: struct
min-align: 32
fields:
  msg_id: uint8
  src:
    class: string
  dst:
    class: string
```


##### Variant type object

A CTF variant type, i.e. a tagged union of CTF types.

**Properties**:

| Property | Type | Description | Required? | Default value |
|---|---|---|---|---|
| `tag` | String | Variant type's tag | Required | N/A |
| `types` | Associative array of strings to [type objects](#type-objects) or strings (alias names) | Possible types | Required | N/A |

The `tag` property's value refers to a previous structure field.
The value may be prefixed with one of the following strings to indicate
an absolute lookup within a previous (or current) dynamic scope:

  * `trace.packet.header.`: trace packet header
  * `stream.packet.context.`: stream packet context
  * `stream.event.header.`: stream event header
  * `stream.event.context.`: stream event context
  * `event.context.`: event context
  * `event.payload.`: event payload

The pointed field must have an enumeration type. Each type name in the
`types` property must have its equivalent member's label in this
enumeration type. This is how a variant's type is selected using the
value of its tag.

**Example**:

```yaml
class: variant
tag: my_choice
types:
  a:
    class: string
  b: int32
  c:
    class: float
    size:
      align: 32
      exp: 8
      mant: 24
```


### Running the `barectf` command

Using the `barectf` command-line utility is easy. In its simplest form,
it outputs a CTF metadata file and a few C files out of a
YAML configuration file:

    barectf config.yaml

will output, in the current working directory:

  * `metadata`: CTF metadata file
  * `barectf-bitfield.h`: macros used by tracing functions to pack bits
  * `barectf.h`: other macros and prototypes of context/tracing functions
  * `barectf.c`: context/tracing functions

`barectf_` is the default name of the files and the default prefix of
barectf C functions and structures. The prefix is read from the
configuration file (see the
[configuration object documentation](#configuration-object)), but
you may override it on the command line:

    barectf --prefix my_app_ config.yaml

You may also output the files elsewhere:

    barectf --code-dir src --headers-dir include --metadata-dir ctf config.yaml


### Using the generated C code

This section assumes you ran `barectf` with no options:

    barectf config.yaml

The command generates C structures and functions to initialize
barectf contexts, open packets, and close packets. It also generates as many
tracing functions as there are events defined in the YAML configuration
file.

An application should never have to initialize barectf contexts,
open packets, or close packets; this is the purpose of a specific barectf
platform, which wraps those calls in its own initialization and
finalization functions.

The barectf project provides a few platforms in the [`platforms`](platforms)
directory. Each one contains a `README.md` file explaining how to use
the platform. If you're planning to write your own platform,
read the next subsection. Otherwise, skip it.


#### Writing a barectf platform

A **_barectf platform_** is responsible for:

  1. Providing some initialization and finalization functions
     for the tracing infrastructure of the target. The initialization
     function is responsible for initializing a barectf context,
     providing the platform callback functions, and for opening the very
     first stream packet(s). The finalization function is responsible
     for closing, usually when not empty, the very last stream
     packet(s).
  2. Implementing the platform callback functions to accomodate the target
     system. The main purposes of those callback functions are:
     * Getting the current value of clock(s).
     * Doing something with a packet once it's full. This is how
       a ring buffer of packets may be implemented. The platform
       may also be naive and write the full packets to the file system
       directly.

Thus, the traced application itself should never have to call
the barectf initialization, packet opening, and packet closing
funcions. The application only deals with initializing/finalizing
the platform, and calling the tracing functions.

The following diagram shows how each part connects with
each other:

![](http://0x3b.org/ss/placoderm625.png)

The following subsections explain what should exist in each
platform function.


##### Platform initialization function

A barectf platform initialization function is responsible for
initializing barectf context(s) (calling `barectf_init()`,
where `barectf_` is the configured prefix), and opening the very
first packet (calling `barectf_stream_open_packet()` with
target-specific parameters, for each stream, where `stream` is
the stream name).

barectf generates one context C structure for each defined stream.
They all contain the same first member, a structure with common
properties.

barectf generates a single context initialization function:

```c
void barectf_init(
    void *ctx,
    uint8_t *buf,
    uint32_t buf_size,
    struct barectf_platform_callbacks cbs,
    void *data
);
```

This function must be called with each stream-specific context
structure to be used afterwards. The parameters are:

  * `ctx`: stream-specific barectf context (allocated by caller)
  * `buf`: buffer to use for this stream's packet (allocated by caller)
  * `buf_size`: size of `buf` in bytes
  * `cbs`: platform callback functions to be used with this
    stream-specific context
  * `data`: user data passed to platform callback functions (`cbs`)

**Example**:

```c
#define BUF_SZ 4096

void platform_init(/* ... */)
{
    struct barectf_my_stream_ctx *ctx;
    uint8_t *buf;
    struct my_data *my_data;
    struct barectf_platform_callbacks cbs = {
        /* ... */
    };

    ctx = platform_alloc(sizeof(*ctx));
    buf = platform_alloc(BUF_SZ);
    my_data = platform_alloc(sizeof(*my_data));
    my_data->ctx = ctx;
    barectf_init(ctx, buf, BUF_SZ, cbs, my_data);

    /* ... */
}
```

barectf generates one packet opening and one packet closing
function per defined stream, since each stream may have custom
parameters at the packet opening time, and custom offsets of
fields to write at packet closing time.

The platform initialization should open the very first packet
of each stream to use because the tracing functions expect the
current packet to be opened.

Here's an example of a packet opening function prototype:

```c
void barectf_my_stream_open_packet(
    struct barectf_my_stream_ctx *ctx,
    float spc_something
);
```

The function needs the stream-specific barectf context, as well as any
custom trace packet header or stream packet context field; in this
last example, `something` is a floating point number stream packet context
field.


##### barectf packet information API

There's a small API to query stuff about the current packet of a
given barectf context:

```c
uint32_t barectf_packet_size(void *ctx);
int barectf_packet_is_full(void *ctx);
int barectf_packet_is_empty(void *ctx);
uint32_t barectf_packet_events_discarded(void *ctx);
uint8_t *barectf_packet_buf(void *ctx);
void barectf_packet_set_buf(void *ctx, uint8_t *buf, uint32_t buf_size);
uint32_t barectf_packet_buf_size(void *ctx);
int barectf_packet_is_open(void *ctx);
```

`barectf_packet_is_full()` returns 1 if the context's current packet
is full (no space left for any event), 0 otherwise.

`barectf_packet_is_empty()` returns 1 if the context's current packet
is empty (no recorded events), 0 otherwise.

`barectf_packet_events_discarded()` returns the number of lost (discarded)
events _so far_ for a given stream.

The buffer size (`buf_size` parameter of `barectf_packet_set_buf()` and
return value of `barectf_packet_buf_size()`) is always a number of bytes.

`barectf_packet_is_open()` returns 1 if the context's current packet
is open (the packet opening function was called with this context).


##### Platform callback functions

The callback functions to implement for a given platform are
in the generated `barectf_platform_callbacks` C structure. This
structure will contain:

  * One callback function per defined clock, using the clock's
    return C type. Those functions must return the current clock
    values.
  * `is_backend_full()`: is the back-end full? If a new packet
    is opened now, does it have its reserved space in the back-end?
    Return 0 if it does, 1 otherwise.
  * `open_packet()`: this callback function **must** call the relevant
    packet opening function.
  * `close_packet()`: this callback function **must** call the
    relevant packet closing function _and_ copy/move the current packet
    to the back-end.

What exactly is a _back-end_ is left to the platform implementor. It
could be a ring buffer of packets, or it could be dumber: `close_packet()`
always appends the current packet to some medium, and `is_backend_full()`
always returns 0 (back-end is never full).

Typically, if `is_backend_full()` returns 0, then the next
call to `close_packet()` should be able to write the current packet.
If `is_backend_full()` returns 1, there will be lost (discarded)
events. If a stream packet context has an `events_discarded` field,
it will be written to accordingly when a packet is closed.

If a platform needs double buffering, `open_packet()` is the callback
function where packet buffers would be swapped (before calling
the barectf packet opening function).


##### Platform finalization function

The platform finalization function should be called by the application
when tracing is no more required. It is responsible for closing the
very last packet of each stream.

Typically, assuming there's only one stream (named `my_stream` in this
example), the finalization function will look like this:

```c
void platform_tracing_finalize(struct platform_data *platform_data)
{
    if (barectf_packet_is_open(platform_data->ctx) &&
            !barectf_packet_is_empty(platform_data->ctx)) {
        barectf_my_stream_close_packet(platform_data->ctx);

        /*
         * Do whatever is necessary here to write the packet
         * to the platform's back-end.
         */
    }
}
```

That is: if the packet is still open (thus not closed and written yet)
_and_ it contains at least one event (not empty), close and write the last
packet.

Note, however, that you might be interested in closing an open empty
packet, since its packet context could update the discarded events count
(if there were lost events between the last packet closing time and
now, which is quite possible if the back-end became full after closing
and writing the previous packet).


#### Calling the generated tracing functions

Calling the generated tracing functions is what the traced application
actually does.

For a given prefix named `barectf`, a given stream named `stream`, and
a given event named `event`, the generated tracing function name is
`barectf_stream_trace_event()`.

The first parameter of a tracing function is always the stream-specific
barectf context. Then, in this order:

  * One parameter for each custom event header field
    (prefixed with `seh_`)
  * One parameter for each custom stream event context field
    (prefixed with `sec_`)
  * One parameter for each custom event context field
    (prefixed with `ec_`)
  * One parameter for each custom event payload field
    (prefixed with `ep_`)

A tracing function returns nothing: it either succeeds (the event
is serialized in the current packet) or fails when there's no
space left (the context's discarded events count is incremented).

**Example**:

Given the following [event object](#event-object), named `my_event`,
placed in a stream named `default` with no custom event header/stream event
context fields:

```yaml
context-type:
  class: struct
  fields:
    msg_id:
      class: int
      size: 16
payload-type:
  class: struct
  fields:
    src:
      class: string
    dst:
      class: string
    a_id:
      class: int
      size: 3
    b_id:
      class: int
      size: 7
      signed: true
    c_id:
      class: int
      size: 15
    amt:
      class: float
      align: 32
      size:
        exp: 8
        mant: 24
```

barectf will generate the following tracing function prototype:

```c
/* trace (stream "default", event "my_event") */
void barectf_default_trace_my_event(
    struct barectf_default_ctx *ctx,
    uint16_t ec_msg_id,
    const char *ep_src,
    const char *ep_dst,
    uint8_t ep_a_id,
    int8_t ep_b_id,
    uint16_t ep_c_id,
    float amt
);
```


### Reading CTF traces

To form a complete CTF trace, the `metadata` file generated by the
`barectf` command-line tool and the binary stream files generated
by the application (or by an external consumer, depending on the
platform) should be placed in the same directory.

To read a CTF trace, use [Babeltrace](http://www.efficios.com/babeltrace).
Babeltrace is packaged by most major distributions as the `babeltrace`
package. Babeltrace ships with a command-line utility that can convert a
CTF trace to human-readable text output. Also, it includes Python bindings
so that you may analyze a CTF trace using a custom script.

In its simplest form, the `babeltrace` command-line converter is quite
easy to use:

    babeltrace /path/to/directory/containing/ctf/files

See `babeltrace --help` and `man babeltrace` for more options.