Simon Marchi [Mon, 22 Apr 2019 15:17:23 +0000 (11:17 -0400)]
bindings/python/bt2: Make the bt2 Python bindings build
This patch is just a little bit more than the bare minimum to make the
Python bindings build (that is, configure with --enable-python-bindings
and build). It consists of sync'ing the .i SWIG interface definition
files with the corresponding headers. It's possible that the interfaces
are incomplete, in which case they'll be adjusted in the a later pass.
For example, output parameters are currently not named in a way that
match the type maps.
For simplicity, I have included at the same time some other changes to
the .i files, such as adjusting the typemap definitions (in
native_bt.i). The helper functions (such as at the bottom of
native_bt_componentclass.i) are also changed to the "final" form they
have on my development branch. Even though these are not immediatly used,
they need to be changed for the bindings to build.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Change-Id: I87a438a692690639f956b214f878df65885ed78c
Simon Marchi [Wed, 3 Apr 2019 16:08:11 +0000 (12:08 -0400)]
lib: Make bt_value_null a const pointer
We probably don't want anything to change the value of this pointer,
which points to the only instance of BT_VALUE_TYPE_NULL,
bt_value_null_instance. Therefore, declare it as a constant value.
It also helps a little bit the Python bindings: if we copy the
declaration verbatim to the .i file, without const, bt_value_null is
available through the special "cvar" global object (and can be
modified!). With const, it becomes read-only, so can be accessed
directly as native_bt.value_null.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Thu, 25 Apr 2019 15:00:02 +0000 (11:00 -0400)]
ctf: Add dash to stream name suffixes
We add numerical suffixes to stream names, if needed, to make them
unique. For trace directories, we do something similar and decided to
format these suffixes with a dash to avoid confusion if the original
name ends with a number. This patch does the same thing but for stream
names.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Change-Id: I3041fe98bbeb3c122d06e5fa54449ed6e55017bd
Reviewed-on: https://review.gerrithub.io/c/eepp/babeltrace/+/452110
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: Philippe Proulx <eeppeliteloop@gmail.com>
Simon Marchi [Tue, 23 Apr 2019 13:32:58 +0000 (09:32 -0400)]
ctf: Use environment fields provided by LTTng to make output trace path
Starting with version 2.11, the LTTng tracer includes in the trace
environment all the information necessary to rebuild the output path
that it itself uses when outputting traces. This lets us recreate an
output path similar to what LTTng does when outputting ctf traces (with
the exception that we also add a level with the hostname, which LTTng
doesn't).
If the trace does not come from LTTng, or comes from LTTng < 2.11, we
keep using the same strategy: use a trace path based on the trace name,
which is itself based on the original directory.
This feature will especially be useful in conjunction with session
rotation: since the input trace is actually composed of multiple
traces/chunks, the trace name would otherwise be based on one particular
of these chunks. With the environment values, we can recreate a path
that describes correctly the trace as a whole. The output path is then
similar to what one would have been output by LTTng, if one had not used
the session rotation feature in the first place.
Change-Id: I03c0569869561ec11873b808c47d30196889cb36
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.gerrithub.io/c/eepp/babeltrace/+/451942
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Thu, 25 Apr 2019 00:55:47 +0000 (20:55 -0400)]
cli: put space after comma between array elements
This is more readable when looking at the output of `--run-args`.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I75a4622fb2f2c864f4af1599c18044bb696c3307
Philippe Proulx [Thu, 25 Apr 2019 00:50:39 +0000 (20:50 -0400)]
Fix: test_convert_args: fix malformed test line
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ia164a1703d12f0e3caff9e9d6fcd9a5c6deb58fd
Philippe Proulx [Thu, 25 Apr 2019 00:38:13 +0000 (20:38 -0400)]
test_convert_args: test multiple leftover arguments
When `babeltrace convert` is given multiple leftover arguments:
babeltrace /path/to/trace1 /path/to/trace2 /path/to/trace3
it configures a single `src.ctf.fs` component with a `paths` array
parameter containing the three paths.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I0c19c730bc276dba7160b9efcb72dc8077c078f5
Philippe Proulx [Thu, 25 Apr 2019 00:34:12 +0000 (20:34 -0400)]
Fix: test_convert_args: fix expected arguments
Changes:
* `src.ctf.fs` paths are given as an array now.
* What used to be `params=...` is now two arguments: `--params ...`.
* There's a new default parameter when instantiating a
`src.ctf.lttng-live` component: `session-not-found-action="end"`.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ic08db60b280ca9babb1ddc21829392d14a0d3fd3
Francis Deslauriers [Wed, 24 Apr 2019 20:13:07 +0000 (16:13 -0400)]
cli: mimic behavior of BT1 when trying to attach to live session
In Babeltrace 1, when trying to attach to a LTTng live session the
`babeltrace` command would exit with status 0 (SUCCESS) if it
successfully connected to the relay daemon _even if_ it did not find the
requested session.
To mimic the behavior of BT1, we set the `session-not-found-action`
parameter to "end". This makes the message iterator of an lttng-live
component return a END status if the requested session is not found
on the relay daemon and makes the cli return gracefully with SUCCESS.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Change-Id: I3b3810f97c090d102fd30902fda37434f62d5c01
Simon Marchi [Tue, 16 Apr 2019 20:16:01 +0000 (16:16 -0400)]
plugins/ctf/fs-src: Session rotation support: merge traces with same uuid
This patch makes the src.ctf.fs component merge traces that have the
same UUID, with the intent of supporting traces output by LTTng's
session rotation feature.
When using this feature of LTTng, the user ends up with multiple traces
that are independent (with their own metadata and data files), except
for the fact that they share the same trace uuid. The goal of this
patch is to make Babeltrace realize this and make all those traces
appear as a single trace.
The approach taken is to scan the search paths (passed as parameter to
the component) as usual and create a ctf_fs_trace for each found trace.
We then go over the list of traces in the component and merge those
having the same UUID.
Merging traces consists of merging corresponding data stream file groups
(ctf_fs_ds_file_group structs). Two ctf_fs_ds_file_group structs are
corresponding if they have the same stream_id (aka stream_instance_id in
CTF) and stream class id.
Merging two ctf_fs_ds_file_group structs consists of merging their list
of ctf_fs_ds_file_info structs, making sure to keep the resulting list
sorted by begin timestamp.
Since the trace-info query also uses create_ctf_fs_traces, its behavior
is updated as expected. Doing a trace-info on a directory containing
multiple traces with the same UUID will result in a single trace with
multiple files per stream. A little drawback/inconsistency that will be
address later is that the trace object in the output has a single
"path", which is set to the path of one of the original traces.
Ideally, it should have also have a list of paths and list all the
paths of the "physical" traces that form this "logical" trace.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Change-Id: Ib290cf08119f9be4344057b2544b4627eea3f450
Simon Marchi [Tue, 23 Apr 2019 16:03:29 +0000 (12:03 -0400)]
cli, plugins/ctf/fs-src: Make src.ctf.fs accept multiple root paths
A following patch will make it possible to process multiple traces that
share the same UUID as a single trace. One requirement is to be able to
do this event when passing multiple arguments, like:
babeltrace path1 path2 ...
With the current code, this instantiates two src.ctf.fs components, and
therefore makes it impossible to do this merging by UUID.
This patch therefore changes the design so that we only instantiate a
single src.ctf.fs component and pass an array with the paths.
We also need to make the trace-info query support receiving multiple
paths. One reason is that in general, we want trace-info queries to
support the same parameters as the components. A more pragmatic reason
is that it will need to do the same kind of UUID-merging anyway. I
therefore made the trace-info query re-use more of the code used by the
component to locate and create the ctf_fs_trace list. One difference
between the two usages is that in the case of the query, we don't want
to create IR streams or component ports. I moved those outside of the
shared code path and made the component do that as a separate step.
One notable change in babeltrace.c is in set_stream_intersections.
We previously extracted the "path" parameter from the component creation
parameter, to make the query parameters. Instead, we now directly pass
the component creation parameters to the query. The idea is the same as
above, we want to pass the same parameters to the trace-info query as
what will be used to create the component, since it might alter how the
traces will be open, and therefore the results.
So in practice the changes boil down to:
- babeltrace-cfg-cli-args.c: Make it possible to serialize bt_value of
type BT_VALUE_TYPE_ARRAY, so we can build the list of paths in a
bt_value array, then serialize it to send it to "babeltrace run".
Make it instantiate a single ctf component, this even simplifies the
code a little bit.
- fs.c: Accept "paths" parameter with a list of paths, rather than
"path" with a single path, iterate on that list to find traces and
create trace objects just as before. Separate creation or IR streams
and component ports from the ctf_fs_trace creation.
- query.c: Re-use more of fs.c's functions to locate and create
ctf_fs_trace objects.
There are no functional changes intended, except if you are the kind of
person to pass params by hand to the src.ctf.fs component.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Tue, 23 Apr 2019 15:16:00 +0000 (11:16 -0400)]
cli: Make append_parameter_to_args accept a bt_value value
append_parameter_to_args is used to append a --params=KEY=VALUE argument
to a list of arguments. It currently assumes that VALUE is a simple
string. We will want to generalize this to support arrays (in the form
of a bt_value of type array). This patch does a bit of refactoring in
preparation of that, making append_parameter_to_args accept a value of
type bt_value* instead of const char*.
append_string_parameter_to_args is introduced as a wrapper, for those
spots that still want to pass a simple string value. It converts the
string to a temporary bt_value of type string.
No functional changes intended.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Tue, 9 Apr 2019 23:00:33 +0000 (19:00 -0400)]
cli: Remove support for --key and --value arguments
There are currently two ways of passing parameters to the current
component on the Babeltrace command line:
--params=KEY=VALUE
--key KEY --value VALUE
The method with --params is more versatile, as VALUE can have different
types (string, number, bool). The value in the --key/--value method is
always converted to a string.
This patch removes the less-versatile --key/--value way, in order to
keep only --params.
I have remove or updated references to --key/--value in man pages. The
only remaining mention of --key/--value that I know of is in
babeltrace-log.c, which I was told is going to disappear anyway, so I
have left it.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Tue, 23 Apr 2019 14:14:08 +0000 (10:14 -0400)]
cli: Support arrays in parameters
This patch adds support for arrays in the parameters one can pass to
components on the Babeltrace command line. For example:
--params="value=[1,2,\"Hello\"]"
When we are starting to parse a value and the token is a '[', we start
parsing it as an array. We accept comma-separated values until we reach
the matching ']'. The code should be pretty self-explanatory/commented.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Tue, 23 Apr 2019 14:03:34 +0000 (10:03 -0400)]
cli: Adjust integer range check, replace magic numbers with constants
The value (1ULL << 63) - 1), used for checking that integer values are
within range, actually mean INT64_MAX, so use that instead.
Also, the negative case is not quite right: the value -(INT64_MAX + 1),
-
9223372036854775808, is within the range of a signed 64-bits number
(assuming two's complement), but is not accepted right now. Adjust the
check so it accepts that value.
I tested this patch manually: we accept -
9223372036854775808 but reject
-
9223372036854775809.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Tue, 23 Apr 2019 14:09:33 +0000 (10:09 -0400)]
cli: Refactor command line value parsing code
This patch breaks out the code that parses values from the command line
to bt_value objects. It introduces a new function which returns a
parsed value directly as a bt_value.
This enables the following patch to introduce support for arrays, by
having this function call itself recursively.
No functional changes intended.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Mon, 22 Apr 2019 14:41:39 +0000 (10:41 -0400)]
cli: Refactor handling of negative numbers
This patch breaks out the code that reads a number just after having
read the minus sign introducing a negative number.
Instead of keeping a state in the parser, and going through a new
iteration, we call the new function which returns a bt_value directly.
This enables the following patch to do a similar change, but for the
parsing of all values (which includes the parsing of negative numbers).
In turn, this will allow parsing values recursively, which is needed to
support array values. Orthogonally, I think the code is easier to
follow this way.
No functional changes intended.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Philippe Proulx [Wed, 24 Apr 2019 17:59:38 +0000 (13:59 -0400)]
Fix: tests/Makefile.am: do not run missing `flt.utils.muxer`
This test does not exist anymore.
Most plugins will be tested in Python or even in Bash with a specific
testing sink component class.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Francis Deslauriers [Mon, 4 Mar 2019 16:02:10 +0000 (11:02 -0500)]
Adapt `src.ctf.lttng-live` to current API
A `src.ctf.lttng-live` component now ensures the monotonic ordering of
the timestamps its outgoing messages. The component can receive messages
from multiples viewer session (UST 64bit, UST 32bit and/or Kernel) that
in turn may contain one or more traces that in turn may contain one or
more streams of messages. Messages from all these streams have to be
ordered into a single flow of messages where every message has a
timestamp equal or larger than the previous one.. This muxing is done by
tracking the next message of each of the live stream iterator and
returning the message with the smallest timestamp.
The live viewer sessions and everything required to get data and
metadata from the LTTng Relay daemon are now owned by the message
iterator and not the component as it was before.
Parameters
==========
The `url` mandatory string parameter controls the URL of the LTTng Relay
Daemon to which the component should register to received traces.
The `session-not-found-action` optional string parameter controls the
behavior of the component in case the requested session is not
configured on the LTTng Relay Daemon. The three accepted value of this
parameter are: "continue", "fail", and "end".
continue: The message iterator will try to connect (or reconnect) to
the requested session for ever.
fail: The message iterator will return an ERROR status if the session
is not available at the message iterator initialization stage and
return END status when the session is no longer available.
end: The message iterator always returns END status if the session is
not found or is no longer available.
The `session-not-found-action` defaults to "continue" if not specified.
Future improvement
==================
Using a priority queue to order the messages would speed up the muxing
of messages.
Known limitation
================
If a new live stream iterator returns a message that has a timestamp
smaller (that happened before) than the last message forwarded
downstream but the `src.ctf.lttng-live` message iterator. The message
iterator will return with an error status. I am not sure if this
situation can happen in a single host configuration (which is the only
supported configuration).
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Mon, 22 Apr 2019 20:21:07 +0000 (16:21 -0400)]
sink.text.pretty: print inactivity msg handling with BT_LOGD_STR
Instead of printing that the component received a message iterator
inactivity message to standard error; log this information with
BT_LOGD().
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Mon, 1 Apr 2019 13:58:02 +0000 (09:58 -0400)]
graph: Fix: bt_graph_configure() does not propagate errors
Issue
=====
When configuring the graph, message iterator initialize functions may fail
and return an error status (e.g. _STATUS_NOMEM and _STATUS_ERROR). Those
error status are not propagated back to the caller of
`bt_graph_configure()` function. The caller that will consider the graph
to be configured properly and run it will it assertions down the road.
Solution
========
Return `BT_GRAPH_STATUS_ERROR` if any of the sinks returned a non-OK
status from its `graph_is_configured()` function.
Known drawbacks
===============
None.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 24 Apr 2019 15:16:48 +0000 (11:16 -0400)]
Fix: flt.lttng-utils.debug-info: leaking `bin_dir` char array
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 24 Apr 2019 14:54:31 +0000 (10:54 -0400)]
Fix: flt.lttng-utils.debug-info: ".debug" extension not appended
Issue
=====
The g_strconcat() function returns a newly allocated string and does not
change the string in place as the code seems to assume. This results in
the call to g_strconcat() having no effects. Because of this, the
".debug" extension is not appended to the file name resulting in a
failure find the file on the file system.
Ignoring the return value of this call also results in the leaking of
the resulting char array.
Solution
========
Use g_snprintf() instead to concatenate the file name and its extension.
Known drawbacks
===============
None.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Philippe Proulx [Wed, 24 Apr 2019 04:19:06 +0000 (00:19 -0400)]
lib: field path: have a specific item for "current array element"
This patch makes a field path object have a list of items, which are
unique objects, instead of simple member/choice indexes. One type of
item is an index item (which contains only a structure member or variant
choice index) while the other is a "current array element" item.
The previous algorithm to "walk" a field path from a known root field
or field class was (Python pseudocode):
current_fc = root_fc
i = 0
while i < len(field_path):
index = field_path[i]
if type(current_fc) is in (StructureFieldClass, VariantFieldClass):
current_fc = current_fc[index]
i += 1
else:
current_fc = current_fc.element_field_class
As you can see, there's no special field path index to indicate that
we're in the current element of an array: you just pass through arrays
without "consuming" a field path index.
In TSDL, it can happen like this:
fields := struct {
struct {
int abc;
int len;
string strings[len];
} s[4];
};
Currently, the field path of the `strings` dynamic array is:
event payload: [0, 1]
With this patch, the same field path becomes:
event payload: [0, <CUR>, 1]
and the walking algorithm above becomes:
current_fc = root_fc
for item in field_path:
if type(current_fc) is in (StructureFieldClass, VariantFieldClass):
assert(type(item) is IndexFieldPathItem)
current_fc = current_fc[item.index]
else:
assert(type(item) is CurrentArrayElementFieldPathItem)
current_fc = current_fc.element_field_class
or it could also be:
current_fc = root_fc
for item in field_path:
if type(item) is IndexFieldPathItem:
current_fc = current_fc[item.index]
else:
assert(type(item) is CurrentArrayElementFieldPathItem)
current_fc = current_fc.element_field_class
There is no special case with this now more complete field path.
I also intend to have something like this in CTF 2. Currently, CTF 1.8
does not allow a relative field path to be rewritten as an absolute
field path if it's in an array. For example, you could not rewrite the
TSDL example above as:
fields := struct {
struct {
int abc;
int len;
string strings[event.fields.s.len];
} s[4];
};
Babeltrace 1 does not accept this, so Babeltrace 2 does not either, and
I don't think CTF is designed this way. I think it's limiting, so I'm
making this patch to accommodate CTF 2 in the future.
This patch also opens the door to having other types of field path items
in the future, like any element within an array (as long as it's before
the requesting dynamic array or variant field), so that you could have:
event payload: [1, 3, 0]
^ ^ ^-- first member of a structure
| `----- fourth element of an array
`-------- second member of a structure
This patch also makes the `flt.lttng-utils.debug-info` and `sink.ctf.fs`
component classes use the modified API.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 23 Apr 2019 23:54:42 +0000 (19:54 -0400)]
Fix: bfcr_get_sequence_length_cb(): do not set text array's length
Issue
=====
In `msg-iter.c`, bfcr_get_sequence_length_cb() gets called when starting
the decoding of any sequence, including text sequences. However, text
arrays are translated into trace IR string field classes.
bfcr_get_sequence_length_cb() calls bt_field_dynamic_array_set_length()
unconditionally, but its field argument can be a string field at this
point. This is a precondition break.
Solution
========
In bt_field_dynamic_array_set_length(), only call
bt_field_dynamic_array_set_length() if the CTF IR field class is NOT a
text sequence. I'm also adding an assertion to confirm that the trace IR
field is a dynamic array field.
Known drawbacks
===============
None.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 23 Apr 2019 22:00:34 +0000 (18:00 -0400)]
Fix: `tests/Makefile.am`: do not run missing tests
Test sources were removed, but they are still mentioned in this file, so
`make check` cannot find them.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 23 Apr 2019 21:57:44 +0000 (17:57 -0400)]
Fix: `tests/ctf-traces/succeed` traces: change CTF version from 0.1 to to 1.8
This version of CTF, 0.1, never really existed. Also, the first lines of
the `metadata` files changed by this patch already contain `CTF 1.8`,
indicating that this is a bug in the `trace` block.
Somehow Babeltrace 1 is less strict than Babeltrace 2 and accepts those
traces. We take the deliberate decision to refuse them.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 23 Apr 2019 21:46:00 +0000 (17:46 -0400)]
Fix: `test_packet_seq_num.in`: change expected disc. packets messages order
Issue
=====
The `test_packet_seq_num` test expects that Babeltrace reports 3, 2, and
1 discard packets in three different messages for CTF trace
`2_streams_lost_in_2`. Babeltrace 1 reports this, but Babeltrace 2
reports 2, 3, and 1.
I believe it's a bug in Babeltrace 1 because it also reports time ranges
which cover the packets which have an unexpected sequence number.
For example, Babeltrace 1:
[warning] Tracer lost 3 trace packets between [06:15:09.
000000090]
and [06:15:09.
000000110] in trace UUID
eb5045f7b471488eb963221ddf423a7, at path:
"../ctf-traces/packet_seq_num/2_streams_lost_in_2", within stream id
0, at relative path: "test_stream_1". You should consider recording
a new trace with larger buffers or with fewer events enabled.
Babeltrace 2:
WARNING: Tracer discarded 3 packets between [06:15:09.
000000090] and
[06:15:09.
000000091] in trace "2_streams_lost_in_2" (UUID:
eb5045f7-b471-488e-b963-
0221ddf423a7) within stream
"/home/eepp/dev/babeltrace/tests/ctf-traces/packet_seq_num/2_streams_lost_in_2/test_stream_1"
(stream class ID: 0, stream ID: 1).
The `test_packet_seq_num` test does not check the reported time ranges,
but this shows that the packet sequence number check might not be done
at the same time in Babeltrace 1 and Babeltrace 2. Because the test
trace contains two data streams with an intersection, and packets were
discarded in this intersection, the order of the warnings is not
guaranteed to be the same.
The time ranges reported by Babeltrace 2 are correct, however. In the
example above, it's okay to report that three packets were discarded
between 06:15:09.
000000090 and 06:15:09.
000000110, but it's even more
precise to report that they were discarded between 06:15:09.
000000090
and 06:15:09.
000000091. The time range of the packet with an unexpected
sequence number is 06:15:09.
000000091 to 06:15:09.
000000110.
Solution
========
Change the test so that it expects 2, 3, and 1 now.
Known drawbacks
===============
Some incompatibility with Babeltrace 1 regarding the order of the
warnings printed by `sink.text.pretty`. However, the locations and the
time ranges of the messages in Babeltrace 2's output are correct.
Most tools and scripts which rely on Babeltrace's text output only check
if there's something (anything) written to the standard error stream
anyway to detect a problem. LTTng analyses is one of those.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 23 Apr 2019 21:38:20 +0000 (17:38 -0400)]
Fix: flt.utils.trimmer: accept inited streams ending without other messages
Issue
=====
It is possible that, for a given stream, an `flt.utils.trimmer`
component receives the stream beginning and stream beginnin activity
messages, and then it needs to be ended. In this case,
`sstate->stream_act_end_ns_from_origin` is never set, so we hit the
assertion.
Solution
========
When ending a stream and when `sstate->stream_act_end_ns_from_origin` is
not set for this stream, use the trimming range's end time as the stream
activity end time.
If `sstate->stream_act_end_ns_from_origin` is not set at this point,
then we know the stream is currently active, so it is safe to use a
specific time as its activity end time. If the last received message for
this stream was a stream activity end message, then we don't reach this
code because it's conditional to
`!sstate->last_msg_is_stream_activity_end`.
Known drawbacks
===============
None.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 23 Apr 2019 21:37:57 +0000 (17:37 -0400)]
Fix: cli: `flt.utils.trimmer` now expect the `gmt` parameter
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 23 Apr 2019 20:40:08 +0000 (16:40 -0400)]
Remove `test_bt_message_iterator`
This test does not work anymore because some message iterator
postconditions changed. Instead of taking time to fix it now, it will be
much faster for me to rewrite it correctly in Python once we have
working bindings.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Simon Marchi [Thu, 18 Apr 2019 18:00:29 +0000 (14:00 -0400)]
Fix: cli: Acquire reference on bt_value_null while parsing args
In bt_config_query_from_args, we initialize "params" to point to
bt_value_null without acquiring a reference:
bt_value *params = bt_value_null;
When handling OPT_PARAMS, lower, we call "put" on it before assigning
params to something else:
bt_value_put_ref(params);
This causes bt_value_null's refcount to drop to 0 and its release to be
called (which should never happen, since its refcount is initialized to
1). This warning is shown:
$ ./cli/babeltrace query
...
04-18 13:56:12.267 10081 10081 W VALUES bt_value_null_instance_release_func@value.c:73 Releasing the null value singleton: addr=0x7f882eaf1b60
Fix it by acquiring a new reference in the beginning.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Francis Deslauriers [Wed, 17 Apr 2019 19:55:47 +0000 (15:55 -0400)]
flt.lttng-utils.debug-info: cleanup: create dedicated msg_iter destroy function
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 17 Apr 2019 18:38:10 +0000 (14:38 -0400)]
tests: remove unused automake variable
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Tue, 16 Apr 2019 20:45:57 +0000 (16:45 -0400)]
Fix: lib: Mismatch in function declarations and definitions
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 17 Apr 2019 20:07:46 +0000 (16:07 -0400)]
flt.lttng-utils.debug-info: Don't copy trace class UUID to output trace class
A `flt.lttng-utils.debug-info` component creates an output trace class
that may be different that of the input trace class, so the UUID should
not be copied. In fact, we don't need a UUID at this point. the sink
component can decide to add one later.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 3 Apr 2019 17:06:08 +0000 (13:06 -0400)]
flt.lttng-utils.debug-info: Implement file descriptor cache
A `flt.lttng-utils.debug-info` component has to open files in order to
do the address resolving when processing a trace. Those file descriptors
are currently associated with a bin_info structure which are stored in a
hash table for each vpid. If, in the same trace, a binary file is loaded
by more than one process, that binary file will be opened twice. This
can create a file descriptor exhaustion problem on long running traces
if the same UST apps are run repeatedly.
This commit implements a file descriptor caching feature to reuse file
descriptors pointing to the same files.
After creating the file descriptor cache (fdcache), users must use the
following two functions to get and put handles to file descriptors:
struct fd_handle *fd_cache_get_handle(struct fd_cache *fdc, char *path);
void fd_cache_put_handle(struct fd_cache *fdc, struct fd_handle *handle);
The fd_handles are reference counted internally and their corresponding
file descriptors are closed when the refcount reaches zero.
Files are compared based on the hashed values of inode and device number
rather than solely on the path. This is needed to make sure that a file
opened previously has not changed. This is more likely to happen when
processing a lttng-live trace where the following scenario can happen
and must be supported:
1. Binary v1 is executed,
2. Trace data received by debug-info component,
3. Handle 1 is taken on the bin_a v1 file,
4. Addresses resolved using handle 1,
5. Binary is modified,
6. Binary v2 is compiled,
7. Binary v2 is executed,
8. Trace data received by debug-info component,
9. Handle 2 is taken on the bin_a v2 file,
10. Addresses resolved using handle 2.
Limitation
----------
It's still be possible to exhaust file descriptors in traces with a
large number of UST apps with no binary file overlap. If this becomes a
problem, we may consider devising an utility to close least recently
used files and reopening file descriptors for later used.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 3 Apr 2019 16:56:17 +0000 (12:56 -0400)]
Cleanup: Move `debug-info` in its own sub-directory
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Thu, 28 Mar 2019 22:20:52 +0000 (18:20 -0400)]
Fix: flt.lttng-utils.debug-info: Error in src line reporting
Issue
=====
A `debug-info` component does not print the `src` field of a tracepoint
properly when tracepoint is within a for loop.
#include "tp.h"
int main(int argc, char *argv[])
{
tracepoint(my_app, empty);
for(int i = 0; i < 1; i++) {
tracepoint(my_app, empty);
}
return 0;
}
The problem derives from the DWARF information generated by the
compiler. The compiler may place the DWARF content of the for loop in
`DW_TAG_lexical_block` which the current `debug-info` design doesn't
expect.
Here is a summarized version of DWARF tree containing the first
tracepoint in the code above:
DW_TAG_subprogram
DW_TAG_inlined_subroutine
/* tracepoint callsite info */
Here is a summarized version of the DWARF tree containing the tracepoint
in the for loop:
DW_TAG_subprogram
DW_TAG_lexical_block
DW_TAG_inlined_subroutine
/* tracepoint callsite info */
The current implementation doesn't expect the presence of a
`DW_TAG_lexical_block` entry as child of the `DW_TAG_inlined_subroutine`
entry and won't look any further down that branch to find the source
information for the current tracepoint. This results in the component
not finding the line number for such callsite and thus leaving the `src`
field empty. Also, on some occasions, the `src` field would contain the
tracepoint definition location (e.g. "tp.h:12") rather then the actual
callsite of that tracepoint.
Solution
========
When iterating over the Debugging Information Entries, if the current
entry contains the address to resolve _but_ is not a of the type
`DW_TAG_inlined_subroutine` try to iterate over the children of that
entry.
Known drawbacks
===============
None.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Tue, 16 Apr 2019 21:15:57 +0000 (17:15 -0400)]
flt.lttng-utils.debug-info: cleanup: remove usage old struct name
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Tue, 16 Apr 2019 21:14:29 +0000 (17:14 -0400)]
flt.lttng-utils.debug-info: cleanup: only print debug message on error
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 27 Mar 2019 14:44:05 +0000 (10:44 -0400)]
flt.lttng-utils.debug-info: use glib memory and string functions
- Change use of glibc's malloc() to glib's g_new0() to follow the rest of
the project.
- Change use of glibc string functions to their glib equivalent.
- Remove uses of GNU extension of conditional expressions with omitted
operands as it would only be useful if the evaluation of the condition
had side effects which it doesn't. Let's prefer code simplicity.
[0]: https://gcc.gnu.org/onlinedocs/gcc-4.2.4/gcc/Conditionals.html
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Thu, 18 Apr 2019 14:12:24 +0000 (10:12 -0400)]
sink.ctf.fs: Remove unused function
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Simon Marchi [Thu, 11 Apr 2019 19:48:30 +0000 (15:48 -0400)]
plugins/ctf: Remove unused uuid-related fields in struct ctf_fs_metadata
They are unused. The corresponding fields holding the metadata uuid are
in struct ctf_metadata_decoder, while the fields holding the trace class
uuid are in ctf_trace_class.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Philippe Proulx [Tue, 16 Apr 2019 17:02:38 +0000 (13:02 -0400)]
.editorconfig: add `charset = utf-8`
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Simon Marchi [Tue, 16 Apr 2019 12:08:18 +0000 (08:08 -0400)]
Add .editorconfig file
This file helps editors set the indentation (and other whitespace)
settings correctly to match the project's coding style.
For now I have just put some simple settings for C, Swig interface and
Python files, but we can add more as needed.
See here for more details:
https://editorconfig.org/
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Fri, 29 Mar 2019 22:18:21 +0000 (18:18 -0400)]
Remove babeltrace 1 Python bindings and tests
We previously had the goal of providing the Babeltrace 1 Python API (the
babeltrace module) on top of the Babeltrace 2 Python API (the bt2
module).
It was decided that this is no longer required, those who have scripts
working against the Babeltrace 1 API will have to keep using Babeltrace
1.
A factor that influenced this decision is that it would be difficult to
keep the exact same behavior as Babeltrace 1 had, and therefore
guarantee that scripts will keep working. For example, CTF headers
(event headers or packet headers) are not accessible in trace-ir. So if
a script relied on reading some header fields, the best we could do
would be to simulate the value of some well-known header fields. For
example, it would be possible to simulate the timestamp field of an
event header. However, things like a packet's content_size and
packet_size would be more difficult.
So instead of putting efforts doing something that works halfway, it was
decided that such scripts would have to keep using Babeltrace 1.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Fri, 29 Mar 2019 23:04:50 +0000 (19:04 -0400)]
Remove stale trace-ir test files
These test files test things that are now pre-conditions in the library,
that used to throw exceptions.
We could in theory keep testing these cases (verify that the
precondition failure is correctly caught, when in dev mode), but we
would have to change the test strategy completely. It's not really a
priority, so for now, just remove them.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Philippe Proulx [Fri, 12 Apr 2019 21:15:38 +0000 (17:15 -0400)]
BT_ASSERT(): update rendering
* Move flip-table emoji at the beginning.
* Make flip-table emoji compatible with terminals which do not support
combining diacritics.
* Colorize file name, line number, and function name independently.
* Add `()` after function name.
* Make the whole "Assertion `...` failed." red, but the message itself
bold.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Simon Marchi [Mon, 1 Apr 2019 15:55:33 +0000 (11:55 -0400)]
Re-implement BT_ASSERT without using the assert macro
BT_ASSERT is currently defined using the assert macro. If NDEBUG is
inadvertently defined, BT_ASSERT has no effect. This is quite
unfortunate, since a user who has defined BABELTRACE_DEBUG_MODE to 1
probably wants BT_ASSERT to be effective.
This problem was encountered while building the Python bindings, where
the distutils native code builder is passing -DNDEBUG to the compiler.
The BT_ASSERT macro usages in these files were found to be ineffective.
This patch avoids this situation by making BT_ASSERT call our own
handler if the assertion fails. This also has the advantage of letting
us personalize the behavior on assertion failures. The presence of the
so-called Lenny Face having absolutely no regards for the furniture is a
good example of this (and was a requirement coming from Philippe
Proulx).
Removing the inclusion of assert.h in assert-internal.h revealed a few
spots that use the assert macro without including assert.h, so they were
adjusted accordingly.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Mon, 1 Apr 2019 20:01:08 +0000 (16:01 -0400)]
Adjust bt_component_borrow_class to match declaration
Rename it to bt_component_borrow_class_const and constify return type,
to match declaration in graph/component-const.h.
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Simon Marchi [Mon, 1 Apr 2019 16:17:43 +0000 (12:17 -0400)]
Small fixes to include/
* Function bt_component_filter_borrow_from_self is declared, but does not
actually exist.
* Parameters to bt_self_component_{get,set}_data are named "private",
which is the old naming scheme. Change that to self.
* bt_component_graph is typedef'ed to struct bt_component_graph, which
doesn't actually exist. struct bt_graph exists, and there's already a
typedef for it.
Francis Deslauriers [Tue, 17 Apr 2018 14:27:31 +0000 (10:27 -0400)]
ctf-writer: Split clock and clock-class declarations and definitions
Split the clock_class functions of ctf-writer in its own header file.
clock_class deserves it.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Philippe Proulx [Fri, 8 Mar 2019 16:06:38 +0000 (11:06 -0500)]
Adapt `sink.ctf.fs` to current API
This patch is a complete rewrite of the `sink.ctf.fs` component class to
work with the current library's API.
Changes
=======
A `sink.ctf.fs` component does not use the CTF writer API anymore: it
tracks its own traces and streams, and writes data thanks to the common,
internal ctfser API (also used by CTF writer).
On the metadata side, the component class has its own CTF IR data
structures. They are based on the common CTF source IR data structures,
but they have additional fields and otherwise contain only what's needed
for a `sink.ctf.fs` component to work.
A `sink.ctf.fs` component systematically "protects" structure FC member
and variant FC option names with a leading `_`. This is suggested by
TSDL 1.8. Because of this, any enumeration FC label could be what
selects a following variant FC option, so enumeration FC labels are also
escaped with `_`. This is a temporary measure which matches the
behaviour of `src.ctf.fs` and `src.ctf.lttng-live`. We should solve this
at the API level later.
If the component fails to resolve the length FC of a dynamic array FC or
the selector FC of a variant FC, it falls back to creating its own
length FC or selector FC just above the requesting FC, within the same
structure FC, giving it a name which avoids any clash with other
members. Here's an example (trace IR, as TSDL):
struct {
struct {
integer { size = 8; } len;
} s;
string str[s.len];
}
This layout is perfectly valid from the trace IR's point of view.
However it is known to fail in Babeltrace 1 and Trace Compass (but is
accepted by Babeltrace 2), so the actual TSDL output for this becomes:
struct {
struct {
integer { size = 8; } _len;
} _s;
integer { size = 32; } ___str_len;
string { encoding = UTF8; } _str[___str_len];
}
The same goes for a variant FC's selector FC:
struct {
struct {
enum : integer { size = 8; } {
A = 1,
B = 5,
C = 17 ... 24
} tag;
} s;
variant <s.tag> {
integer { size = 8; } A;
string B;
struct { } C;
} var;
}
becomes:
struct {
struct {
enum : integer { size = 8; } {
"_A" = 1,
"_B" = 5,
"_C" = 17 ... 24,
} _tag;
} _s;
enum : integer { size = 16; } {
"_A" = 0,
"_B" = 1,
"_C" = 2,
} ___var_tag;
variant <___var_tag> {
integer { size = 8; } _A;
string { encoding = UTF8; } _B;
struct { } _C;
} _var;
}
This way we don't lose the original length/selector field value while
still writing the dynamic array/variant field.
A generated dynamic array FC's length FC is a 32-bit unsigned integer FC
(should be enough) while a generated variant FC's selector FC is a
16-bit unsigned enumeration FC.
A `sink.ctf.fs` component writes its events to the appropriate streams
as it receives them, as opposed to CTF writer which flushes a whole
packet when calling bt_ctf_stream_flush(). This means a packet's total
size is not limited to the available memory.
A `sink.ctf.fs` component generates a new trace UUID instead of using
the original trace class's UUID. The logic behind this is that there is
no way to know that the received trace and streams match exactly the
original trace and streams (the source and filters can modify them), so
systematically using the original UUID could lead to two different CTF
traces having the same UUID.
A `sink.ctf.fs` component writes the metadata file of a given CTF trace
and closes it when the upstream message iterator ends or when the trace
is destroyed. The component does not take a trace or trace class
reference: it registers a trace destruction listener so that it can free
the resources associated to a trace (and its streams) as soon as it's
destroyed. Also, it frees the resouces associated to a given stream when
getting a stream end message for it.
The way output directory paths and stream file names are created is
unchanged from the previous version. The `assume-single-trace` parameter
still exists.
There are three new parameters:
`ignore-discarded-events` (boolean):
Ignore discarded events messages. This can be useful because there
are limitations (see below) regarding where discarded events
messages can be in the message flow, and what their beginning and
end times can be.
`ignore-discarded-packets` (boolean):
Ignore discarded packets messages. This can be useful because there
are limitations (see below) regarding where discarded packets
messages can be in the message flow, and what their beginning and
end times can be.
`quiet` (boolean):
Do not print anything. When the component is not quiet, it prints a
message to the standard output every time it creates a complete CTF
trace with the absolute path to it.
Known limitations
=================
This component class is not complete, in that it does not support all
the features of Babeltrace 2. For most cases, there should be no problem
using it to write streams created by a `src.ctf.fs` or
`src.ctf.lttng-live` component.
As of this patch, the known limitations are:
* Only names which are valid in TSDL are supported. This applies to:
* Trace class environment keys.
* Structure FC member names.
* Variant FC option names.
* Clock class names.
* The value type of a trace class environment entry must be integer or
string.
* Unknown stream clocks are not supported.
* Stream activity messages are ignored.
* Discarded events and packets messages must occur between packets.
* There must not be more than one discarded events message or more than
one discarded packets message between packets.
* The time range of a discarded events message must be from the last
packet's end time to the next packet's end time, except when it occurs
before the first packet of a given stream, where its beginning time
must be the first packet's beginning time.
* The time range of a discarded packets message must be from the last
packet's end time to the next packet's beginning time.
All the constraints above are checked as soon as possible and, when one
is not satisfied, the component fails with a detailed error.
The very strict and very CTF-ish limitations regarding discarded events
and packets messages are the reason why I added the aforementioned
`ignore-discarded-events` and `ignore-discarded-packets` parameters.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Thu, 11 Apr 2019 13:21:24 +0000 (09:21 -0400)]
ctfser: have dedicated signed/unsigned integer writing functions
Most of the time there's an external check for signedness anyway, so it
is redundant to pass the signedness to the writing function only for it
to evaluate it again.
In `lib/ctf-writer/fields.c`, the signedness check is now peripheral to
the ctfser writing function.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Thu, 11 Apr 2019 17:10:04 +0000 (13:10 -0400)]
src.ctf.fs: keep empty structures
If there's any empty structure in the original trace, keep it (create
an empty trace IR structure field class too).
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 10 Apr 2019 03:03:20 +0000 (23:03 -0400)]
lib: add structure FC member and variant FC option objects
This patch adds the structure field class member and variant field
class option objects to the field class API.
The purpose of those objects is to be able to operate at the structure
FC member or variant FC option level in the future, for example when we
add custom user attributes which you can attach to individual field
classes, but also to members/options themselves.
Behind the scenes, the `bt_field_class_structure_member *` and
`bt_field_class_structure_member *` types are just the existing
`struct bt_named_field_class *` type.
You cannot manually create a structure FC member or variant FC option
object: they are created when calling
bt_field_class_structure_append_member() or
bt_field_class_variant_append_option(). You can then borrow them with
bt_field_class_structure_borrow_member_by_index(),
bt_field_class_structure_borrow_member_by_name(),
bt_field_class_variant_borrow_option_by_index(),
bt_field_class_variant_borrow_option_by_name(), and their const
equivalents.
When you append a member to a structure FC or an option to a variant FC,
the function does not freeze the member/option itself, but only its
contained field class. The member/option object is frozen when the
container field class (structure/variant) is frozen itself. This means,
in the future, that you can borrow a structure FC member or a variant FC
option object and then modify it (set an optional property, for
example).
Structure FC member and variant FC option objects are unique (for the
moment): they are merely a proxy offering accessors to the properties of
a container field class's elements, like enumeration field class mapping
range objects.
The only drawback I can see is that you need three function calls to get
the name and borrow the field class of a structure FC member or variant
FC option instead of a single one, but we can reintroduce a single
accessor later if it's really an issue.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 10 Apr 2019 01:22:03 +0000 (21:22 -0400)]
Fix: pretty/print.c: print comma and space after trace name
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 9 Apr 2019 19:39:32 +0000 (15:39 -0400)]
Fix: update_field_class_in_ir(): do not have special int fields in trace IR
The logic in update_field_class_in_ir() for the integer field class case
was wrong. This patch fixes this, so that we only translate the CTF IR
integer field class to a trace IR integer field class if another field
class depends on it (dynamic array, variant) or if it does not update a
clock AND it does not have a special meaning.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Sun, 7 Apr 2019 02:16:36 +0000 (22:16 -0400)]
Fix: msg-iter.c: accept no packet total and content sizes
This changes an assertion in set_current_packet_content_sizes() to check
that either both the packet and content sizes are set (one can be set
from the other) or both are unset.
It's okay that both total and content sizes are unset: the data decoder
is already implemented to end the packet when the medium returns
`BT_MSG_ITER_MEDIUM_STATUS_EOF`.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 29 Mar 2019 22:05:54 +0000 (18:05 -0400)]
Fix: handle_conversion_specifier_std(): use `sizeof(PRI*64) - 1`
`sizeof(PRIu64)`, for example, includes the terminating null character,
so what we want here is this size minus one.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Thu, 28 Mar 2019 18:37:41 +0000 (14:37 -0400)]
lib-logging-internal.h: fix minor documentation mistakes
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 27 Mar 2019 22:30:10 +0000 (18:30 -0400)]
Fix: decoder.c: remove forgotten TODO section
This was probably there for debugging purposes in the first place, but
was unintentionally left there.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 27 Mar 2019 22:28:40 +0000 (18:28 -0400)]
`ctf` plugin: infer packet's total size from packet's content size
If a packet context's `packet_size` field is missing, but the
`content_size` field exists, use the packet's content size as the
packet's total size. I believe this is valid CTF.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 27 Mar 2019 22:11:56 +0000 (18:11 -0400)]
Fix: msg-iter.c: create packet message with correct function
Issue
=====
With CTF metadata having a stream class with:
* A packet context not having `timestamp_begin` or `timestamp_end`
fields.
* An event header having a field mapped to a clock class.
The result is a Babeltrace trace IR stream class with a default clock
class.
However the decoding process fails because, without `timestamp_begin`
and `timestamp_end` packet context fields, the current packet's
beginning and end times are never set, thus we call
bt_message_packet_beginning_create() and bt_message_packet_end_create()
to create the messages. This is a precondition break because when a
stream class has a default clock class, all the packet beginning, packet
end, and event messages must have default clock snapshots.
Solution
========
In create_msg_packet_beginning(), if the current packet's beginning time
property is not set, use the last packet's end time or 0 if this is
not set either.
In create_msg_packet_end(), if the current packet's end time property is
not set, use the current clock value, and set it as the current packet's
end time property so that it becomes the last packet's end time property
in bt_msg_iter_switch_packet().
Known drawbacks
===============
With this fix, the first packet beginning message's clock snapshot's raw
value is 0. This is probably not ideal, as we cannot know if there was
any stream activity between 0 and the first event's time. The real fix
should probably be to use the first event's time, but this would require
more work as we would need to decode the first event's header before
calling create_msg_packet_beginning().
It's not a problem to compute the stream intersection, as no messages
are created during the execution of the `trace-info` query, so both
properties, as returned by bt_msg_iter_get_packet_properties(), are not
set. However there's a bug in the `trace-info` query as of this patch:
it does not check if the properties are set or not.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 27 Mar 2019 00:44:59 +0000 (20:44 -0400)]
Fix: src.ctf.fs: do not use trace IR objects in queries
Issue
=====
We try to create a `bt_stream` object from a translated CTF IR stream
class object (to `bt_stream_class`) systematically in
create_ds_file_groups(), but this path is also taken by the `trace-info`
query which needs DS file groups, but does not need trace IR objects
(and cannot have them because there's no self component object in this
case).
Also there's a bug in build_index_from_stream_file() where the index is
not built because of a wrong status code check.
The result is that `trace-info` query does not work.
Solution
========
Only use CTF IR objects in the queries. Therefore, only create a
`bt_stream` object for the DS file group if the CTF IR stream class was
translated (this only happens when there's an available self component).
Fix the status code check issue in build_index_from_stream_file().
Known drawbacks
===============
None.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Sun, 24 Mar 2019 22:56:48 +0000 (18:56 -0400)]
flt.utils.muxer: validate clock classes on stream beginning messages
This patch makes the clock class validation occur only when getting a
stream beginning message or a message iterator inactivity message.
Because the library guarantees that we always get a stream beginning
message before getting any other message for the same stream, and
because a stream class needs a default clock class for an
`flt.utils.muxer` message iterator to work, we can do the validation a
single time at the beginning of a given stream. This is instead of
checking this for every single message.
This patch also does the "stream's default clock is always known"
validation at the same location (stream beginning message) instead of
checking the clock snapshot state for every message.
I'm adding the `MUXER_MSG_ITER_CLOCK_CLASS_EXPECTATION_NONE` clock class
expectation type which is used when the first stream-associated message
has a stream with no default clock. This is possible with a single
upstream message iterator which provides messages with a single stream
of which the class has no default clock class.
The outcome is less function calls, and I can consistently measure an
approximate 6% performance improvement when reading a typical
four-stream, 1.4-GiB CTF trace created by LTTng.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 20 Mar 2019 20:59:28 +0000 (16:59 -0400)]
Fix: add_component_with_init_method_data(): freeze parameters
In add_component_with_init_method_data(), an internal function called by
all the bt_graph_add_*_component*() functions, the function must freeze
the parameters value object (developer mode) in order to prevent further
changes of this object (and its children), since at this point the
object becomes shared with other users.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Francis Deslauriers [Wed, 20 Mar 2019 13:24:34 +0000 (09:24 -0400)]
plugins/ctf/common: remove unused borrow_named_int_field_class() function
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 20 Mar 2019 13:19:12 +0000 (09:19 -0400)]
flt.utils.trimmer: fix: variables used uninitialized
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 20 Mar 2019 13:11:34 +0000 (09:11 -0400)]
lib: remove unused port_connection_iterators_are_finalized() function
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 20 Mar 2019 13:09:48 +0000 (09:09 -0400)]
lib: remove unused precondition utility function
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Wed, 20 Mar 2019 13:05:14 +0000 (09:05 -0400)]
flt.lttng-utils.debug-info: fix: set but not used warnings
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Philippe Proulx [Tue, 19 Mar 2019 18:51:38 +0000 (14:51 -0400)]
lib: mark graph as faulty when adding a comp. or connecting ports fails
This patch makes the graph's state become
`BT_GRAPH_CONFIGURATION_STATE_FAULTY` when there's a configuration
error, that is, when one of the bt_graph_add_*_component*() functions or
the bt_graph_connect_ports() fail.
When a graph is faulty, it becomes impossible to use it in any way,
either to continue configuring it or to run it.
This new state makes it invalid to do anything with a graph when it
failed one of its configuration steps, so that each step must succeed in
order to get a valid, useable, configured graph. This fixes the issue
where in bt_graph_connect_ports(), one of the "port connected" method
would succeed but the other would fail: in this scenario, the connection
ends immediately (disconnecting each port), but one component still
believes that its port was successfully connected (as there's no "port
disconnected" method, because we don't need it for anything else). With
this patch, the component's "port connected" is still called and
successful, but the component will never be used anyway, so there's less
danger for errors. bt_port_is_connected() will indeed return `BT_FALSE`
for this port in the component's finalization method, for example.
The alternative would be to add an optional component method to indicate
that a port which was just connected is in fact not connected because
the other component's "port connected" method failed. This is possible,
but what can a graph's user do with a failing bt_graph_connect_ports()
anyway? If the intention was to connect two ports, and this operation
fails, then there's probably no plan B: retrying the exact same port
connection would probably fail again.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 19 Mar 2019 18:39:17 +0000 (14:39 -0400)]
bt_port_output_message_iterator_create(): check that graph has no sinks
The graph must not have any sink component when calling
bt_port_output_message_iterator_create() because the purpose of an
output port message iterator is to iterator the messages of a source or
a filter component.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Tue, 19 Mar 2019 18:34:05 +0000 (14:34 -0400)]
lib: remove bt_connection_is_ended()
This function is not needed anymore because connections cannot end.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 15 Mar 2019 22:03:01 +0000 (18:03 -0400)]
lib: metadata: add missing non-const borrowing accessors
Add a few missing non-const borrowing accessors in the metadata part of
trace IR. They are not strictly needed for anything written in C
(because since you own the non-const object, you typically don't need to
read a property that you set yourself), but they will help make some
binding wrapper APIs symmetrical.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Francis Deslauriers [Tue, 19 Feb 2019 18:55:40 +0000 (13:55 -0500)]
flt.lttng-utils.debug-info: adapt debug-info component to API changes
General idea
------------
Given that:
* debug-info filter component needs to add new fields to the
events common context,
* Metadata and data objects are immutable after being added to
the trace class or trace respectively.
The debug-info component needs output a new augmented trace containing
the new fields along side all the received input data. It thus needs to
act both as a sink AND a source of graph messages in the sense that it
consumes and creates graph messages.
For every message received, the corresponding output objects(e.g.
stream) are created in an output trace. Those objects are stored in
hashmaps corresponding to their types for later use. These hashmaps map
input object to their corresponding output objects.
For example, when handling a packet beginning message, we need to create
a new output packet. To do that, the corresponding output stream is
needed, amongst other things. This output stream was created earlier
when a stream beginning message was received and can be recovered from
the stream hashmap.
For each type of message received from the upstream iterator, we must
create a copy of that message and its corresponding trace IR objects,
and send it downstream. This is true for all but one message type,
inactivity messages can be forwarded directly as they are independent of
the data and metadata IR objects.
Implementation
--------------
This commit adds the concept of mapping between input and output objects
and implements copy functions to create (nearly) identical output
objects based on the input ones.
The entry point of the mapping of objects is the struct trace_id_maps
structure.
The metadata_map and data_map hashtables have input trace class and
input trace objects, respectively, as key and are pointing to struct
trace_ir_metadata_maps and struct trace_ir_data_maps. Each of the IR
object maps contained in these structs have the input object of their
specific types as key.
All input and mapped output objects are identical except for the stream
classes. Event-common-context of the stream classes needs to be modified
to include the debug-info fields if and only if the right fields
(instruction pointer (ip) and vpid) are present in the input stream
class.
Here are the objects kept in the mapping:
- Trace class
- Stream class
- Event class
- Clock class
- Trace
- Stream
- Packet
The memory layout of each processes is kept up to date using the
lttng-ust-statedump and lib-load events. This information is then used
to resolve source locations based on ip and vpid values found in the
event-common-context of events.
Future Improvements
---------------------
Decoupling the trace-ir-mapping library from the debug-info component so
it can be used in other filter components.
Other fixes
-----------
- Don't resolve debug-info on build id mismatch
- When parsing the build id of a statedump event, confirm that it
matches the one present on disk for that file.
- Prevent NULL pointer access of arg_debug_info_field_name
- Typo in the lttng_ust_statedump:debug_link event name.
- Typo in the crc field name of the lttng_ust_statedump:debug_link
event.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Tue, 26 Feb 2019 23:45:02 +0000 (18:45 -0500)]
lib: replace trace is_static with destruction listeners
General idea
------------
Replace bt_trace is_static infrastructure with the concept of
destruction listener and extend it for the bt_trace_class object.
This allows for downstream components to ask to be notify of the
destruction of those objects. This notification is useful for component
that keep data related trace IR objects that should be deallocated when
the trace and/or trace class are destroyed.
This feature is mostly useful to handle sources which could create and
release lots of trace class and trace objects. In this case, when the
source message iterator is done with the stream objects (emitted all the
relevant stream end messages), it puts the stream, trace, and trace
class references. Then downstream users are notified that specific
external objects are about to be destroyed, so they release/close any
associated resource.
The flt.lttng-utils.debug-info component will make use of this
capability to cleanup the mapping between input and output objects it
needs to manage to do its function.
Implementation
--------------
Both trace class and trace offer the same interface and are implemented
in the exact same way.
Destruction listener callbacks and their associated private pointer are
stored in an array and are all called at the execution of the destroy
functions. Destruction listeners are registered using the following
functions:
bt_trace_status bt_trace_add_destruction_listener(
const bt_trace *trace,
bt_trace_destruction_listener_func listener,
void *data,
uint64_t *listener_id);
bt_trace_class_status bt_trace_class_add_destruction_listener(
const bt_trace_class *trace,
bt_trace_class_destruction_listener_func listener,
void *data,
uint64_t *listener_id);
It's possible to remove a destruction listener using the destruction
listener id received when register it initially used the following
functions:
bt_trace_status bt_trace_remove_destruction_listener(
const bt_trace *trace,
uint64_t listener_id);
bt_trace_class_status bt_trace_class_remove_destruction_listener(
const bt_trace_class *trace_class,
uint64_t listener_id);
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Philippe Proulx [Tue, 12 Mar 2019 18:06:39 +0000 (14:06 -0400)]
Fix: flt.utils.muxer: use return value (clock class)
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Francis Deslauriers [Thu, 7 Mar 2019 19:35:15 +0000 (14:35 -0500)]
Fix: flt.utils.muxer: reset is_ended variable after each iteration
Problem
-------
Whenever an upstream message iterator returns STATUS_END, we remove it
from the array used to order the messages. The variable `is_ended` is
used to record if a upstream message iterator is ended and must be
removed. This variable is only set to false _before_ the loop. If the
first upstream message iterator in the array needs to be removed, all
the following iterators will be mistakenly removed.
Solution
--------
The `is_ended` variable must be reset to false at each iteration so that
only the upstream message iterator that is actually ended is removed from
the array.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Philippe Proulx [Wed, 6 Mar 2019 23:15:06 +0000 (18:15 -0500)]
sink.utils.counter: use a default step of 10,000
1000 makes this way too fast for the terminal.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Wed, 6 Mar 2019 23:14:52 +0000 (18:14 -0500)]
sink.utils.counter: handle all types of messages
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Francis Deslauriers [Thu, 28 Feb 2019 22:48:49 +0000 (17:48 -0500)]
Cleanup: erroneous assert and log messages in stream.c
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Tue, 19 Feb 2019 18:53:48 +0000 (13:53 -0500)]
flt.lttng-utils.debug-info: Fix: dirname(3) might modify input string
Use g_path_get_dirname() to allocate a news string instead.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Fri, 15 Feb 2019 22:01:08 +0000 (17:01 -0500)]
Fix: use of wrong enumeration_mapping_ranges type
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Fri, 15 Feb 2019 21:58:10 +0000 (16:58 -0500)]
Fix: add missing `_const` in name of const functions
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Francis Deslauriers [Fri, 15 Feb 2019 21:52:08 +0000 (16:52 -0500)]
Fix: BT_ASSERT() on variable not yet initialized
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Philippe Proulx [Wed, 6 Mar 2019 19:55:18 +0000 (14:55 -0500)]
common: prepend `bt_common_` to bt_message_iterator_status_to_self()
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Francis Deslauriers [Mon, 17 Dec 2018 21:43:26 +0000 (16:43 -0500)]
Add function converting bt_message_iterator_status to bt_self_*
It's expected that this conversion will be needed often by plugin
developers.
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Philippe Proulx [Tue, 5 Mar 2019 20:10:17 +0000 (15:10 -0500)]
Add a common, internal CTF serialization library; make CTF writer use it
This patch adds the `bt_ctfser` API and implementation, a generic,
common, and internal CTF stream file serialization library. This patch
also makes CTF writer use this library instead of its internal
serialization API (on which `bt_ctfser` is based). The purpose of this
partition is to make the `sink.ctf.fs` component class depend on
`bt_ctfser` without depending on CTF writer (as the trace IR and the CTF
writer IR objects are not compatible anymore) while still reusing code.
After initializing a `struct bt_ctfser` with bt_ctfser_init(), you need
to call bt_ctfser_open_packet() to open a packet, then call one of the
bt_ctfser_write_*() functions to write individual fields or
bt_ctfser_align_offset_in_current_packet() to align the offset within
the current packet. The serialization and alignment functions resize the
current packet as needed, which is why they can fail if resizing is not
possible. Then call bt_ctfser_close_current_packet() with the required
packet size. Repeat for each packet to append to the stream file. When
done, call bt_ctfser_fini(). bt_ctfser_fini() truncates the stream file
as needed to remove any data after the last closed packet.
It is common for the packet context to be written after writing the
last event record because this is when you know the effective content
size. For this, the bt_ctfser_get_offset_in_current_packet_bits()
and bt_ctfser_set_offset_in_current_packet_bits() functions exist to
control the current offset within the current packet. The typical
packet writing algorithm is:
1. Open the packet.
2. Write the packet header.
3. Save the current offset (packet context offset).
4. Write the packet context, potentially with wrong/incomplete fields.
5. Write each event record.
6. Get the current offset which is the packet's content size, and also
decide on a packet's total size (greater than content size).
7. Go to the (saved) packet context offset.
8. Write the packet context with updated fields.
Fast path serialization and alignment functions are `static inline`.
There is minimal verbose logging which always indicate the stream file's
path and its numeric descriptor.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Mon, 4 Mar 2019 21:47:41 +0000 (16:47 -0500)]
common-internal.h: prefix functions with `bt_common_`
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 15 Feb 2019 23:59:29 +0000 (18:59 -0500)]
Adapt `flt.utils.trimmer` to current API
This patch is a 99% rewrite of the `flt.utils.trimmer` component class.
An `flt.utils.trimmer` component still supports the `begin` and `end`
parameters, each one being optional, to configure its bounds. You can
still specify a time without a date for its message iterators to find
the date automatically and apply the time over it. The supported
date/time formats are:
YYYY-MM-DD hh:mm:ss.nnnnnnnnn
YYYY-MM-DD hh:mm:ss
YYYY-MM-DD hh:mm
hh:mm:ss.nnnnnnnnn
hh:mm:ss
ss.nnnnnnnnn
ss
-ss.nnnnnnnnn
-ss
The `gmt` boolean parameter controls whether or not the time strings
express UTC or local times.
An `flt.utils.trimmer` message iterator keeps a state for each stream it
encounters. This is needed to generate some messages that could be
missing at the beginning and at the end of the trimming range for each
stream.
An `flt.utils.trimmer` message iterator initially makes its upstream
message iterator seek the trimming range's beginning time, so that the
seeking operation performs the initial message skipping. This means that
the upstream message iterator must support seeking for a trimmer message
iterator to work with it. The iterator fails gracefully when the
upstream message iterator cannot seek.
Another constraint is that an `flt.utils.trimmer` message iterator only
works with messages having a default clock snapshot. This is obvious
considering the iterator's purpose.
Initial and final stream activity messages are of particular interest.
The implemented algorithm is as such:
Stream beginning message:
Known time:
Inside trimming range:
Accept message as is.
Outside trimming range:
Standard end of trimming range condition.
Unknown time:
Accept as is.
-inf time:
Generate new stream beginning message having the trimming
range's beginning time as its time.
Missing stream beginning message:
Generate new stream beginning message having the trimming
range's beginning time as its time.
Stream end message:
Known time:
Inside trimming range:
Accept message as is.
Outside trimming range:
Standard end of trimming range condition.
Unknown time:
Accept as is.
+inf time:
Generate new stream end message having the trimming range's end
time as its time, and emit it only when reaching the end of the
trimming range.
Missing stream end message:
Generate new stream end message having the trimming range's end time
as its time.
Discarded events and packets messages can also be modified (that is,
discarded and then the iterator creates a new, similar message) to
narrow their time range if it's outside the trimming range. When this
occurs, the message's count property is not copied: we can't know how
many events/packets were discarded in the new time range.
Each `flt.utils.trimmer`¸message iterator has an output message queue.
When the iterator's "next" method is called, it gets the next upstream
message iterator's messages until the output message queue contains
something, handling each received message to drive the per-stream state
machines which can or cannot push messages to the queue (state_trim()
function). Then it pops messages from the queue to fill the outgoing
message array.
When the upstream iterator is ended, the trimmer iterator walks the
current stream states and pushes the appropriate messages to the queue
to correctly end each one, and then removes all the stream states. The
state becomes `TRIMMER_ITERATOR_STATE_ENDING` until the queue is empty,
in which case it becomes `TRIMMER_ITERATOR_STATE_ENDED` of which the
only purpose is to return the `BT_SELF_MESSAGE_ITERATOR_STATUS_END`
status.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Mon, 4 Mar 2019 19:36:51 +0000 (14:36 -0500)]
Fix: sink.text.pretty: do not always update last timestamp
We use print_timestamp_cycles() and print_timestamp_wall() in
print_discarded_elements_msg(), but those must not update the
component's last known timestamp because this is used to compute the
time delta between event messages.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 1 Mar 2019 21:29:35 +0000 (16:29 -0500)]
flt.utils.muxer: do not release upstream iterators when they're ended
This patch makes an `flt.utils.muxer` message iterator NOT release (put
the reference of) one of its upstream message iterator when it's ended
(when bt_self_component_port_input_message_iterator_next() returns
`BT_MESSAGE_ITERATOR_STATUS_END`). This is needed to make it possible
for an `flt.utils.muxer` message iterator to seek its beginning without
creating new upstream message iterators.
Instead, when an upstream message iterator ends, its internal wrapper is
moved to an "ended" array of upstream message iterators. Only "active"
upstream message iterators are considered when actually muxing. When
seeking the beginning, we make all upstream iterators seek (active and
ended), and then put back the ended upstream message iterators into the
active upstream message iterator array.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 1 Mar 2019 21:27:24 +0000 (16:27 -0500)]
Fix: lib: stop auto-seeking when getting the first satisfying message
find_message_ge_ns_from_origin() did push all the remaining messages
when getting the first message >= the requested seeking time, but then
it was continuing its "next" loop.
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 1 Mar 2019 21:26:41 +0000 (16:26 -0500)]
Fix: lib: auto_seek_handle_message(): set `msg` to `NULL` when skipping
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 1 Mar 2019 03:10:40 +0000 (22:10 -0500)]
Fix: print.c: remove extra `)` after `+????????????`
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 1 Mar 2019 02:17:37 +0000 (21:17 -0500)]
Fix: cli: `list-plugins`: print bold code before "component classes"
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Philippe Proulx [Fri, 1 Mar 2019 02:17:00 +0000 (21:17 -0500)]
Fix: cli: cmd_list_plugins(): pass appropriate by-index functions
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
This page took 0.0517 seconds and 4 git commands to generate.