How to develop and debug LKM (drivers) on a remote target

If you want to add code to a Linux kernel, the most basic way to do that is to add some source files to the kernel source tree and recompile the kernel. In fact, the kernel configuration process consists mainly of choosing which files to include in the kernel to be compiled.

But you can also add code to the Linux kernel while it is running. A chunk of code that you add in this way is called a loadable kernel module. These modules can do lots of things, but they typically are one of three things: 1) device drivers; 2) filesystem drivers; 3) system calls. The kernel isolates certain functions, including these, especially well so they don't have to be intricately wired into the rest of the kernel. (Reference: https://tldp.org/HOWTO/Module-HOWTO/x73.html)

A loadable kernel module (LKM) is a special type of C/C++ project and this document describes How to develop and debug LKM on a remote target.

Before you create and debug a LKM, first create a Kernel project by following the How to create and debug Linux kernel project on a remote target document. To develop and debug LKM in TimeStorm, please refer to the following sections:

  1. Creating a LKM project
  2. Building Kernel Module Project
  3. Deploying the Kernel Module
  4. Debugging Kernel Module

1. Creating a Loadable Kernel Module (LKM) Project:

To create a new loadable kernel module project click File --> New --> Project --> C/C++, click Next, and choose Kernel Module as the project type as shown below.

You can start developing a kernel module project from two of the templates included with TimeStorm, or you can start from an empty project. Enter a name for the project and click next.

Select the build configurations to be created for the project as shown below and click Next.

Select the SDK Type and choose SDK you want to use with the kernel module as shown below and click next. Be sure to select the toolchain that is used with the kernel project / kernel sources you will be choosing on the subsequent page.

Kernel module development requires the location of the sources and a kernel image. You can choose a kernel project that is already created in your workspace (recommended) or you can choose an external directory that contains the kernel sources as shown below.

Click Finish to create the kernel module project.

2. Building Kernel Module Project:

A loadable kernel module project is built the same way as other C/C++ projects. To build the kernel module, choose a build option from Project menu or from the context menu that appears when you right click on the project name as shown below.

3. Deploying the Kernel Module:

To deploy a kernel module on a running target, you will create and run a kernel module launch configuration. To create a kernel module launch configuration, select the kernel module project and click TimeStorm Kernel/Driver > Run Loadable Kernel Module as shown below.

After clicking on the “Run Loadable Kernel Module” it will open “TimeStorm Kernel/Driver Configurations” as shown below.

1. Main Tab:

In the Main tab shown below. Enter a name for the launch configuration. The module to be deployed to the target and the project name are filled in. You can change them if you want to make any modifications.

2. Arguments tab:

The kernel module is deployed to the target by using a mod_install.sh script. This script is filled in as the command to execute on the arguments tab. The kernel module is the argument to this script, which is also filled in the program arguments text field.

3. Target Tab:

The Target tab shown below, allows you to select the hardware target used when you run the application.

In the Target tab, use the drop-down list to select a hardware target that you have already created. If you have not yet created a hardware target or want to change the settings for an existing target, click ‘Manage targets’ to open the Hardware Targets management utility.

4. Download Files tab

In the ‘Download Files’ tab shown below, you can select the files that TimeStorm transfers to your target. The application that has to be downloaded is already listed. You can use this panel to download additional files to your target, along with your application.

The application specified in the File field is listed as [Target Program]. This is the application that will be debugged. You can change where it is downloaded by selecting it and clicking Edit. If the program links in shared libraries, then these libraries will be downloaded to the target as well and the Files field is listed as [Target Program and Libraries]. These libraries will be downloaded to the same location as the target program. To download the program and libraries to separate locations, remove the default entry and re-add each as a separate entry.

Upload files to the target check-box option: This option is used to copy the files to target by unchecking the ‘Upload files to the target’ check-box configuration won’t copy the files to the remote target.

In case, if you want to transfer only a few files (not all the files in the project to the remote target) then check [Upload files to the target] and remove default settings (project) by clicking on the remove button in the Download Files tab. Then add files which you want to upload to the remote target by clicking on the Add files button in the Download Files tab.

This panel contains the following buttons:

Add File – Use the ‘Add File’ button to specify additional files to transfer, along with the application. These files are transferred every time you transfer the application file.

Edit – To change where the application (or any other file in this list) is installed, select it and click ‘Edit’ to change its destination directory.

Remove – To eliminate a file from the items to download, select it in the list and click the ‘Remove’ button.

Download Now – The ‘Download Now’ button copies files immediately, without running the application.

Restore Default – The ‘Restore Default’ button resets the panel to its initial state (download only the application and its libraries).

NOTE: Downloaded files overwrite any identically named files on the target without giving a warning. TimeStorm will not create destination directories on the target, so be sure to create them before launching the configuration.

5. Common tab:

The ‘Common’ tab shown below, sets options for sharing this run configuration among multiple projects.

The options with the Common tab include:

Local File — By default, debug configurations are saved with the workspace state files, so they can only be used with projects in the current workspace. (This setting corresponds to the ‘Local file’ radio button on this panel.) However, you can make the configuration available for use in other workspaces by choosing the ‘Shared file’ radio button.

Shared File — When you select this option, the debug configuration is saved as a .launch file that can be imported into another TimeStorm workspace. Optionally, you can specify a different location for the file so that it is more easily accessible to multiple projects.

Display in Favorites Menu — You can select whether to include this Run configuration on the main ‘Run’ menus. For example, if you select the ‘Run’ checkbox, the Run configuration always appears in the ‘Run History’ submenu.

You can choose to include the run configuration in either menu, both menus, or neither menu.

After setting the values in all the tabs, click the Run button at the bottom of the Run configuration dialog. This will connect your host to your target, download the application to the target and execute the application on the target. The commands executed on the target and the program output progress are displayed in the console view.

4. Debugging Kernel Module

Debugging a kernel module involves:

  1. Configure and Build Kernel Project
  2. Boot the board
  3. Kernel Debugger Configuration
  4. Kernel Module Debug Launch Configuration

1. Configure and Build Kernel Project:

To configure the kernel, refer to How to create and debug Linux kernel project on a remote target documents --> Configuring the kernel section and set these options:

  • In “General Setup”, turn on “Prompt for development and/or incomplete code/drivers” (CONFIG_EXPERIMENTAL)
  • In “Kernel Hacking”, turn on “Kernel Debugging” , “Compile the kernel with debug info”, and “KGDB: kernel debugger”
  • In “Kernel Hacking” --> “KGDB: kernel debugger”, turn on “KGDB: use kgdb over the serial console”.
  • In “General Setup” --> “Configure standard kernel features (expert users)” , turn on “Load all symbols for debugging/ksymoops” and “Include all symbols in kallsyms”

Refer to How to create and debug Linux kernel project on a remote target document --> Building and deploying the kernel section to know how to build the kernel and build the Kernel.

2. Boot the board

Boot the board using the kernel image built in the previous section and following the Getting Started Guide for the board. No additional boot arguments are required.

3. Kernel Debugger Configuration

Refer How to create and debug Linux kernel project on a remote target document --> Debugging the kernel section to create a debug configuration for the kernel project associated with the kernel module project. Just create and save the debug launch configuration and DO NOT launch the debugger. This Configuration will be referred in the kernel module debug launch configuration explained in the next section.

4. Kernel Module Debug Launch Configuration

To open kernel module debug launch configuration Right Click on the Project → TimeStorm Kernel/Driver → Debug Loadable Kernel Module.

It will open kernel module debug launch configuration which is similar to kernel module launch configuration (As mentioned above Deploying the Kernel Module section ) with an additional debugger configuration tab as shown below.

To let the kernel module debugger know which kernel debugger to use, select the kernel debugger configuration you created in the previous section from the drop down list. Selecting the kernel debugger configuration will identify and fill in the kernel project, kernel path and the debugger path.

Debugging a kernel module is different depending whether you want to debug the init section (you have a breakpoint in the init section of the kernel module) or the functional section (you have a breakpoint in the kernel module outside the init section).

Debug the init section

To debug the init section, select the “Start Debugger, then deploy module” radio button. You can leave the target connection retry count and the target connection retry delay to be the default values.

Debug the functional section

To debug the functional section, select the “Deploy module, then start debugger” radio button.

Click the Debug button to debug the kernel module and the debugger would stop at the breakpoint set in the init section or functional section.

Stepping commands: The TimeStorm Platform helps developers debug by providing buttons in the toolbar and key binding shortcuts to control program execution.

Important Notes:

  1. Kernel module debugging works for one session. To debug the kernel module again, reboot the board.
  2. Before rebooting the board, make sure you terminate all currently running kernel module debug sessions. This will release the serial console connection to the board used by the debugger.
  3. If you experience timeout errors during the debug session, uncheck the Window --> Preferences --> C/C++ > Debug > GDB > Command timeout(ms) checkbox.