This manual is for program, version version.

Copyright ©  copyright-owner.

     Permission is granted to ...

LTTng Userspace Tracer
**********************

This manual is for UST 0.11.

Overview
********

What is UST?
============

The LTTng Userspace Tracer (UST) is a library accompanied by a set of tools to trace userspace code.

Code may be instrumented with either markers or tracepoints. A highly efficient lockless tracer records these events to a trace buffers. These buffers are reaped by a deamon which writes trace data to disk.

High performance is achieved by the use of lockless buffering algorithms, RCU and per-cpu buffers. In addition, special care is taken to minize cache impact.

License
=======

The LTTng Userspace Tracer is intended to be linkable to open source software as well as to proprietary applications. This was accomplished by licensing the code that needs to be linked to the traced program as LGPL.

Components licensed as LGPL v2.1:

  * libust

  * libinterfork

  * libustcomm

Components licensed as GPL v2:

  * ustctl

  * libustctl

  * ust-consumerd

Supported platforms
===================

UST can currently trace applications running on Linux, on the x86-32, x86-64 and PowerPC 32 architectures.

Installation
************

The LTTng userspace tracer is a library and a set of userspace tools.

The following packages are required:

  * ust

     This contains the tracing library, the ust-consumerd daemon, trace control tools and other helper tools.

     Repository: http://git.dorsal.polymtl.ca

  * liburcu

     This is the userspace read-copy update library by Mathieu Desnoyers.

     Available in Debian as package liburcu-dev.

     Home page: http://lttng.org/urcu

  * LTTV

     LTTV is a graphical (and text) viewer for LTTng traces.

     Home page: http://lttng.org

Liburcu should be installed first. UST may then be compiled and installed. LTTV has no dependency on the other packages; it may therefore be installed on a system which does not have UST installed.

Refer to the README in each of these packages for installation instructions.

Quick start
***********

First, instrument a program with a marker.

     #include <ust/marker.h>

     int main(int argc, char **argv)
     {
     	int v;
     	char *st;

     	/* ... set values of v and st ... */

     	/* a marker: */
     	ust_marker(myevent, "firstarg %d secondarg %s", v, st);

     	/* a marker without arguments: */
     	ust_marker(myotherevent, MARK_NOARGS);

     	return 0;
     }

Then compile it in the regular way, linking it with libust. For example:

     gcc -o foo -lust foo.c

Run the program with usttrace. The usttrace output says where the trace was written.

     usttrace ./foo

Finally, open the trace in LTTV.

     lttv-gui -t /path/to/trace

The trace can also be dumped as text in the console:

     lttv -m textDump -t /path/to/trace

Instrumenting an application
****************************

In order to record a trace of events occurring in a application, the application must be instrumented. Instrumentation points resemble function calls. When the program reaches an instrumentation point, an event is generated.

There are no limitations on the type of code that may be instrumented. Multi-threaded programs may be instrumented without problem. Signal handlers may be instrumented as well.

There are two APIs to instrument programs: markers and tracepoints. Markers are quick to add and are usually used for temporary instrumentation. Tracepoints provide a way to instrument code more cleanly and are suited for permanent instrumentation.

In addition to executable programs, shared libraries may also be instrumented with the methods described in this chapter.

Markers
=======

Adding a marker is simply a matter of inserting one line in the program.

     #include <ust/marker.h>

     int main(int argc, char **argv)
     {
     	int v;
     	char *st;

     	/* ... set values of v and st ... */

     	/* a marker: */
     	ust_marker(myevent, "firstarg %d secondarg %s", v, st);

     	/* another marker without arguments: */
     	ust_marker(myotherevent, MARK_NOARGS);

     	return 0;
     }

The invocation of the ust_marker() macro requires at least 2 arguments. The first, "myevent", is the name of the event. The second is a format string that announces the names and the types of the event arguments. Its format resembles that of a printf() format string; it is described thoroughly in Appendix x.

A given Marker may appear more than once in the same program. Other Markers may have the same name and a different format string, although this might induce some confusion at analysis time.

Tracepoints
===========

The Tracepoints API uses the Markers, but provides a higher-level abstraction. Whereas the markers API provides limited type checking, the Tracepoints API provides more thorough type checking and discharges from the need to insert format strings directly in the code and to have format strings appear more than once if a given marker is reused.

     The usttrace tool always uses the early tracing mode.

A function instrumented with a tracepoint looks like this:

     #include "tp.h"

     void function(void)
     {
     	int v;
     	char *st;

     	/* ... set values of v and st ... */

     	/* a tracepoint: */
     	trace_myevent(v, st);
     }

Another file, here tp.h, contains declarations for the tracepoint.

     #include <ust/tracepoint.h>

     DECLARE_TRACEPOINT(myevent, TP_PROTO(int v, char *st),
     		   TP_ARGS(v, st));

A third file, here tp.c, contains definitions for the tracepoint.

     #include <ust/marker.h>
     #include "tp.h"

     DEFINE_TRACEPOINT(myevent);

     void myevent_probe(int v, char *st)
     {
     	ust_marker(myevent, "v %d st %s", v, st);
     }

     static void __attribute__((constructor)) init()
     {
     	register_trace_myevent(myevent_probe);
     }

Here, tp.h and tp.c could contain declarations and definitions for other tracepoints. The constructor would contain other register_* calls.

Recording a trace
*****************

Using usttrace
==============

The simplest way to record a trace is to use the usttrace script. An example is given in the quickstart above.

The usttrace script automatically:

  * creates a daemon

  * enables all markers

  * runs the command specified on the command line

  * after the command ends, prints the location where the trace was saved

Each subdirectory of the save location contains the trace of one process that was generated by the command. The name of a subdirectory consists in the the PID of the process, followed by the timestamp of its creation.

The save location also contains logs of the tracing.

When using usttrace, the early tracing is always active, which means that the tracing is guaranteed to be started by the time the process enters its main() function.

Several usttrace's may be run simultaneously without risk of conflict. This facilitates the use of the tracer by idependent users on a system. Each instance of usttrace starts its own daemon which collects the events of the processes it creates.

Setting up the recording manually
=================================

Instead of using usttrace, a trace may be recorded on an already running process.

First the daemon must be started.

     # Make sure the directory for the communication sockets exists.
     $ mkdir /tmp/ustsocks

     # Make sure the directory where ust-consumerd will write the trace exists.
     $ mkdir /tmp/trace

     # Start the daemon
     $ ust-consumerd

     # We assume the program we want to trace is already running and that
     # it has pid 1234.

     # List the available markers
     $ ustctl list-markers 1234
     # A column indicates 0 for an inactive marker and 1 for an active marker.

     # Enable a marker
     $ ustctl enable-marker 1234 auto ust/mymark

     # Create a trace
     $ ustctl create-trace 1234 auto

     # Start tracing
     $ ustctl start-trace 1234 auto

     # Do things...

     # Stop tracing
     $ ustctl stop-trace 1234 auto

     # Destroy the trace
     $ ustctl destroy-trace 1234 auto

For more information about the manual mode, see the ustctl(1) man page.

Using early tracing
===================

Early tracing consists in starting the tracing as early as possible in the program, so no events are lost between program start and the point where the command to start the tracing is given. When using early tracing, it is guaranteed that by the time the traced program enters its main() function, the tracing will be started.

When using usttrace, the early tracing is always active.

When using the manual mode (ustctl), early tracing is enabled using environment variables. Setting UST_TRACE to 1, enables early tracing, while setting UST_AUTOPROBE to 1 enables all markers automatically.

Crash recovery
==============

When a process being traced crashes, the daemon is able to recover all the events in its buffers that were successfully commited. This is possible because the buffers are in a shared memory segment which remains available to the daemon even after the termination of the traced process.

Tracing across fork() and clone()
=================================

Tracing across clone() when the CLONE_VM flag is specified is supported without any particular action.

When clone() is called without CLONE_VM or fork() is called, a new address space is created and the tracer must be notified to create new buffers for it.

This can be done automatically, by LD_PRELOAD'ing libinterfork.so. This library intercepts calls to fork() and informs the tracer it is being called. When using usttrace, this is accomplied by specifying the -f command line argument.

Alternatively, the program can call ust_before_fork() before calling fork() or clone() with CLONE_VM. After the call, ust_after_fork_parent() must be called in the parent process and ust_after_fork_child() must be called in the child process.

Tracing programs and libraries that were not linked to libust
=============================================================

Some programs need to be traced even though they were not linked to libust either because they were not instrumented or because it was not practical.

An executable that is not instrumented can still yield interesting traces when at least one of its dynamic libraries is instrumented. It is also possible to trace certain function calls by intercepting them with a specially crafted library that is linked with LD_PRELOAD at program start.

In any case, a program that was not linked to libust at compile time must be linked to it at run time with LD_PRELOAD. This can be accomplished with usttrace's -l option. It can also be done by setting the LD_PRELOAD environment variable on the command line. For example:

     # Run ls with usttrace, LD_PRELOAD'ing libust
     # (assuming one of the libraries used by ls is instrumented).
     $ usttrace -l ls

     # Run ls, manually adding the LD_PRELOAD.
     $ LD_PRELOAD=/usr/local/lib/libust.so.0 ls

Performance
***********

Todo.

Viewing traces
**************

Traces may be viewed with LTTV. An example of command for launching LTTV is given in the quickstart.

Viewing multiple traces
=======================

When tracing multi-process applications or several applications simultaneously, more than one trace will be obtained. LTTV can open and display all these traces simultaneously.

Combined kernel-userspace tracing
=================================

In addition to multiple userspace traces, LTTV can open a kernel trace recorded with the LTTng kernel tracer. This provides events that enable the rendering of the Control Flow View and the Resource View.

When doing so, it is necessary to use the same time source for the kernel tracer as well as the userspace tracer. Currently, the recommended method is to use the timestamp counter for both. The TSC can however only be used on architectures where it is synchronized across cores.

Resource Usage
**************

The purpose of this section is to give an overview of the resource usage of libust. For a developer, knowing this can be important: because libust is linked with applications, it needs to share some resources with it. Some applications may make some assumptions that are in conflict with libust's usage of resources.

In practice however, libust is designed to be transparent and is compatible with the vast majority of applications. This means no changes are required in the application (or library) being linked to libust.

Libust is initialized by a constructor, which by definition runs before the main() function of the application starts. This constructor creates a thread called the listener thread.  The listener thread initializes a named socket and waits for connections for ust-consumerd or ustctl.

Libust-specific code may:

  * use malloc() and free()

  * map shared memory segment in the process adress space

  * intercept some library calls, specifically fork() and clone()

  * do interprocess communication with the daemon or ustctl

  * create and open named sockets

It will not:

  * handle any signal (all signals are blocked in the listener thread)

  * change any process-wide setting that could confuse the application

List of environment variables detected by libust
************************************************

The behavior of tracing can be influenced by setting special environment variables in the environment of the traced application. This section describes these variables.

  * UST_TRACE

     If set to 1, start tracing as soon as the program starts. Tracing is guaranteed to be started by the time the main() function starts.

  * UST_AUTOPROBE

     If set to 1, enable all markers by the time the main() function starts.

  * UST_AUTOCOLLECT

     If set to 0, disable notification of daemon on trace start. Useful for performance tests.

  * UST_OVERWRITE

     If set to 1, enable overwriting of buffers on overrun.

  * UST_SUBBUF_NUM

     If set, defines the default number of subbuffers per buffer.

  * UST_SUBBUF_SIZE

     If set, defines the default size of subbuffers, in bytes.

GDB integration
***************

GDB, the GNU Debugger, can use UST markers as GDB tracepoints (note GDB has its own concept of tracepoint). This feature is called GDB Static Tracepoints. When a GDB tracepoint is hit, GDB collects the marker arguments, as well as the state of the registers. Support for GDB is currently work in progress.