How to Use LTTng

Summary

LTTng, the Linux Trace Toolkit Next Generation (LTTng), is a highly efficient full system tracing solution. It is composed of several components to allow tracing of the kernel, of userspace, trace viewing and analysis, and trace streaming. LTTng provides an integrated interface for both kernel and user-space tracing. A "tracing" group allows non-root users to control tracing and read the generated traces. It is multi-user aware, and allows multiple concurrent tracing sessions.

LTTng allows access to tracepoints, function tracing, CPU PMU counters, kprobes, and kretprobes. It provides the ability to attach "context" information to events in the trace (e.g. any PMU counter, process and thread ID, container-aware virtual PIDs and TIDs, process name, etc). It works on mainline kernels (2.6.38 or higher) without any patches. Since LTTng 2.0 is not provided by upstream kernel; it is required to install separate packages. LTTng focuses on providing a system-wide view (across software stack layers) with detailed combined application and system-level execution traces, without adding too much overhead to the traced system.

LTTng is best suited to finding performance slowdowns or latency issues (e.g. long delays) in production or while doing development when the cause is either unknown or comes from the interaction between various software/hardware components. It can also be used to monitor production systems and desktops (in flight recorder mode) and trigger an execution trace snapshot when an error occurs, which provides detailed reports to system administrators and developers.

lttng-tools provide a session daemon (lttng-sessiond) that acts as a tracing registry. To trace any instrumented applications or the kernel, a registered tracing session is needed beforehand. To interact with the session daemon and a tracing session, you should use the lttng command line UI (lttng).

Compile and Install LTTng

Extract the LTTng package in the kernel source directory and configure the Kernel and the Factory as below.

LTTng Tools (lttng cmd line utility, lttng-sessiond) should be included in the Factory.

 $ make menuconfig 
[*] Include Target Software -> Software Packages -> Development -> Debugging and Testing tools -> ltt-control
[*] Include Target Software -> Software Packages -> Development -> Debugging and Testing tools -> ltt-usertrace
[*] Include Target Software -> Software Packages -> Development -> Debugging and Testing tools -> lttng-modules
[*] Include Target Software -> Software Packages -> Development -> Debugging and Testing tools -> lttng-tools

Kernel to be traced with LTTng should be configured with LTTng and Tracing support. Configure the kernel and re-build the kernel & rfs with the following included:

$ make kernel-menuconfig
[*] Include Kernel Hacking -> Tracers
[*] Include Kernel Hacking -> Tracers -> Kernel Function Tracer
[*] Include Kernel Hacking -> Tracers -> Kernel Function Graph Tracer
[*] Include Kernel Hacking -> Tracers -> Support for tracing block IO actions
[*] Include Device Drivers -> Staging Drivers -> LTTng Kernel Tracer

Once the build is complete, boot the target with the new Kernel image and RFS and enjoy tracing!

Procedure for Kernel Tracing

In order to trace the kernel, you'll need the lttng-modules 2.x compiled and installed. You can start the session daemon by invoking the command lttng-sessiond, or let the lttng command line tool do it for you. The session daemon loads the LTTng tracer modules for you if those modules can be found on your system. If they are not found, the kernel tracing feature will be unavailable.

Once all installation is successfully complete run the following commands on the target to collect traces. The traces can then be viewed on the Host/target.

0) List available kernel events:

# lttng list -k
Spawning a session daemon
Kernel events:
-------------
      timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      lttng_statedump_network_interface (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_migrate_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_process_free (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_process_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_process_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)

1) Create a tracing session. The $HOME/lttng-traces directory will be created containing the session name (here 'mysession') you are working on.

# lttng create mysession
Session mysession created.

If you have multiple sessions, you can change the current session by using:

# lttng set-session myothersession

# lttng set-session mysession

2) Enable all tracepoints and all system call events.

# lttng enable-event -a -k
All kernel events are enabled in channel channel0

3) Start tracing:

# lttng start
Tracing started for session mysession

Tracing is in progress at this point and traces will be written in $HOME/lttng-traces/mysession-<date>-<time>/kernel/

NOTE: It will start tracing for all domain(s).

4) Stop tracing:

# lttng stop
Waiting for data availability.
Tracing stopped for session mysession

NOTE: At this point, you can restart the trace (lttng start), enable/disable events You can also read the trace since the buffers are flushed on stop command.

5) Destroy your session after you are done with tracing:

# lttng destroy

6) Read/View the trace with Babeltrace:

$ babeltrace /path/to/trace/dir

7) In the case you are trying to capture a chunk of trace data over time, lttng can be configured to send the output to a round robin
series of trace log files. Note that "session6" can be any character string. The lttng enable-channel command configures 32 log files each at maximum size
of 20,000,000 bytes, with the oldest file being overwritten once the last file is filled.

 lttng create session6
 lttng set-session session6
 lttng enable-channel -k -C 20000000 -W 32 --overwrite chan0
 lttng enable-event -a -k -c chan0
 lttng start

Timestorm Trace Viewer

The viewer delivered with Timesys Timestorm Version: 4.4.3.201409150920 includes the Eclipse LTT-NG viewer plug in. From TimeStorm, you can configure and control LTTng, collect the trace data, and visualize and analyze the trace data. For help on how to use LTTng tools, please refer to http://wiki.eclipse.org/Linux_Tools_Project/LTTng2/User_Guide

1 From the menu bar select:

Window->Open Perspective->LTTng Kernel, select OK

2 On the left hand pane called Project Explorer, right click to select New->tracing project and enter a name for your project, example mytraceingproject, select the box Use default location, then Finish.

3 On the left hand pane Project Explorer, select the small arrow to the left of mytraceingproject to drop down a list. Right click on Traces [0] and select Import.

4 Fill in the location of the directory name containing the trace files: /home/robert/Datalogic/trace

5 Select the check box next to the folder name trace.

6 Select Finish.

7. Select the small arrow to the left of Traces [ 1 ] to drop down a kernel trace called “trace”. Underneath that “trace” entry are 3 new entries,
“CPU usage”
“LTTng Kernel Analysis”
“Tmf Statistics Analysis”

8. Right click on trace and select open to get the data displayed visually.

9. On to your detailed analysis.

Additional Resources