Recommended Factory Workflow

Table of Contents

Introduction

Timesys builds its Linux starting points using the "Factory" build engine. This same engine is available to LinuxLink subscribers to create their own custom platforms. The engine is very flexible, allowing a myriad of options to accomplish this. The goal of this document is to specify a recommended procedure and workflow for using the Factory build engine to develop an embedded linux project as a team. The following diagram outlines the basic workflow:

The workflow starts with the section in red at the top. This section is generally not a team-based activity. Each person who performs these tasks will start from the beginning independently. The tasks also have an expected serial order. Once this initial smoke test period is complete, a Team Environment is set up. At this point all members of the project will be working from a single code-base shared via source control. This source control system can also be used to perform nightly builds and the like. Once this environment is setup, there are a number of independent tasks that can be performed (or not performed) in any order or in parallel. These tasks are represented within the blue box at the bottom. For each of these, some code is checked out from source control, local building and testing occurs, and then the modified or added code is checked back in. Most of these tasks modify only a portion of the platform. To perform each task, the relevant part of the platform is isolated from the remainder of the platform for the edit/build/debug cycle. E.g. The entire platform need not be built just to test out a new package or build an application.

Obtain Starting point, Create Factory Build, Download, Smoke Test/Boot built Platform

Usually the first step is to either obtain a pre-built starting point or create a custom Factory build from your online LinuxLink dashboard. In both cases, the output files are very similar, and are defined in FactoryOutput. One of these files, {board_name}-development-environment.sh , contains everything need to smoke test the board. The procedure for booting Linux is specific to each board, and is covered in the Getting Started Guide for that board, which is available both as a PDF and on the LinuxLink website.

Rebuild starting point from source

Once you've verified that the built components will boot on your board, the next step is to reproduce this build locally. First, download and unpack the downloaded Factory archive.

tar xzf factory.tar.gz
cd factory

Optionally download and unpack the source archive. This step is required if you will not have an network connection at build time.

tar xzf sources.tar.gz

Copy in .config unless the Factory builder archive already contains a customized version (such as from the web Factory). Type make. The kernel image will be under build_<target>/images/. A tar containing the root filesystem will be under build_<target>/images/rfs/rootfs.tar.gz. If the starting point is configured to produce other types of root filesystem images, then they will also be under build_<target>/images/rfs/. Given these images, the user can boot the target board using the same procedure as when they initially booted the downloaded Platform.

See also:

Establish a team-build environment, aka setup build system (with git as an example)

  • Check the Factory into your source control system. Check all of the files that came in the Factory archive. Exclude build_*, src, and include/config/tswo as these are directories that are created during the build. Also exclude .config. The .config file should be ignored so that each developer can use this for the current task they are working on. Follow the instructions from your source control system. For example, here is how to do it with git: 
    git init .
git add *
git commit -a
  • If you are using git for version control, Timesys recommends that you clone http://git.timesys.com/factory.git and create your own integration branch. This makes it much easier to pull from Timesys, merge the weekly release, or cherry-pick specific commits from subsequent releases into your branch. Refer to HOWTO Clone a Timesys Git Repository for help in configuring access authentication. 
    git clone http://git.timesys.com/factory.git
cd factory
git checkout -b mycompany_branch
...
# later, use git to download all changes in the newer releases
git fetch origin
...
# review log of changes in the Timesys master branch
git log origin
# to rebase your branch on top of newer Timesys changes
git rebase origin
# or cherry pick the change that you are interested in (repeat as necessary)
git cherry-pick [sha1 of commit]
  • By default the Factory will download source code from the Timesys website on demand. Generally you will want to switch this to a local location. This could be a local directory, or an http server at your local site. Everyone on your team will need access to this, so a server or a shared directory makes the most sense. You can seed this directory with the contents of the src/dl directory from a local build or from the sources archive from a web starting point. Here is an example that uploads the src directory to server named "corpshare" :
    scp -r src/dl/* corpshare.mycompany.com:/var/www/factory/src/

    Here is an example of uncompressing the source archive from the web download in a web server's Factory/src directory:
    cd /var/www/factory/src && tar xzf /path/to/sources.tar.gz

    Once this is seeded, change your workorder to point to this server for its Global Source Location (under Work Order Configuration in menuconfig). All of the other source locations are set to be relative to this path by default, so all source and patches should now be downloaded from your local server. You can put a file:// URL instead of http to use a local directory as long as the two locations do not point to the same place. E.g. file:///var/www/factory/src/
  • If you are using a LinuxLink Factory installer, or your team's build configuration is to fetch a toolchain, the default toolchain configuration is to fetch the toolchain archive from: 
    file://$(BASE_DIR)/.toolchain.tgz

    Change the path to point to the toolchain archive in your local repository or local file repository.
    This option is configured via make menuconfig at:
    • Toolchain Configuration
      • Toolchain Location
  • Now the source and patches will be pulled from your local location. However, configuration files will be pulled from local directories within the Factory engine itself. You can see any files that will be used this way by searching your configuration for file URLS: 
    grep 'file:' .config

    You will see a list of URLs that refer to local files. Depending on your current package selection, not all of these option are necessarily relevant. The URLs that start with file://$(BASE_DIR) refer to files relative to the Factory's root directory. You may continue to use these URLs if you want to use the Timesys default configurations. If not, you may change them to point to alternate files, or to files on the local source location that you setup in the previous step. You are not required to use file: URLs, http or other protocols will also work. Each of these options will be available under the options for the package when you use menuconfig.
  • The engine is now checked in, but there is no default platform configuration (workorder) for the team to share. Since developers will need to have their own private .config to perform various development tasks, the shared configuration for the platform should be checked in under a different name. A simple convention is to use platform_config file name for this. For example, assuming the config you want to use is currently stored in .config: 
    cp .config platform_config
git add platform_config
git commit platform_config
  • Your Factory engine is now configured to be shared among team members. Each team member can check-out a copy from git, set the shared configuration, and build a complete platform. In addition, your nightly build system can check-out from this location and perform a build on your build server. Here is an simple example of how to re-build the platform from an git check-out: 
    git clone https://<URL to internal git service>>/factory factory
cd factory
cp platform_config .config
make 2>&1 | tee build.log

    If a developer wants to make a change to the shared config, they should copy platform_config to .config, make any changes, then move the changes back to the original file and commit. E.g.: 
    git pull
cp platform_config .config
make menuconfig
make
# Test any changes you made using menuconfig
cp .config platform_config
git commit platform_config

Using a custom kernel

By default, the Factory builds all of its software, including the kernel, based on a "pristine" upstream tarball and a list of patches to apply. This is convenient when you are working with stock source and patches from Timesys or even other vendors. However, often significant work is done on the kernel as part of embedded projects. In these cases it may be more convenient to work with a source tree, rather than tar files and patches. The Factory supports this work flow through the "external" kernel source option. The already created "src" directory is a convenient place to keep our kernel source. Unpack the kernel source here, apply any patches, and copy down or make a config. Example:

tar xjf src/dl/k/kernel/kernel-2.6.37/linux-2.6.37.tar.bz2 -C src
patch -d src/linux-2.6.37 -p1 < ~/my-special.patch
cp target/configs/kernel/qemu_versatile/config-2.6.37 src/linux-2.6.37/.config

Now run menuconfig to edit your workorder. Under Target Software->Kernel, change the option to build an external kernel. This should enable an option to set the Kernel Source Path. This option must be set to an absolute path to your kernel source directory. In order to avoid embedding absolute paths in the workorder, it is recommend to use the BASE_DIR variable. In our example, it would be set to: $(BASE_DIR)/src/linux-2.6.37 . The remainder of the options, including version should be set correctly so that the build engine is able to handle your kernel properly. Before building for the first time, take the opportunity to add your kernel source to your source control and remove any data from old kernel builds. The Factory will use your kernel directory to store its build stamps as well. These should not be checked in, so set an ignore pattern for them in .gitignore.

Subsequent builds will use the kernel source and .config from your directory. You may edit the kernel source or .config file (don't forget to set ARCH= if you use menuconfig to edit the kernel config). For example:

make -C src/linux-2.6.37 menuconfig ARCH=arm

Once you've made changes, the easiest way to force the Factory engine to re-build or re-install the kernel is to use the 'make kernel-restage' make target. This sets the kernel to rebuild, and copies the updated image, modules, etc to the output locations.

Changing package selections and other platform options

The basic procedure for changing your platform's configuration is described above in Establish a Build Environment and summarized below:

  1. Check out from git.
  2. Copy platform_config to .config.
  3. Use menuconfig to edit .config.
  4. Build platform and test changes.
  5. Cycle back to menuconfig step until happy with changes.
  6. Copy .config back to platform_config and check-in.

There are a number of things to keep in mind during the edit (menuconfig), build, test cycle. Many of these are already covered in FactoryBestPractices. You can also check out FactoryTroubleshooting. In addition, some common tasks and things to watch out follow.

Autoconf and building a platform

Most of the open source packages in Factory use the "autoconf" system. This is the "./configure" script that runs prior to building the package. When building a package, autoconf will automatically detect what other packages are installed and enable/disable features as needed. It is possible to pass arguments to the configure script to override these automatic choices. However, as much as possible, the default Factory configuration will not do this. This allows the build from source technique to reach its full potential. Each package has a minimal dependency set that allows for the smaller footprint, but also less capabilities, as some capabilities rely on additional packages. However, if those capabilities are desired the additional packages can be selected and autoconf will automatically detect them and enable the additional features. There are a few things to keep in mind:

  1. The order that the packages are installed makes all the difference when using autoconf in this way. A package's headers and libs must already be installed into the toolchain for autoconf to detect them. If the packages are not being installed in the desired order, you can affect this order by adding a dependency within the recipe for the package.
  2. If a feature in a package is not being enabled, even though the required dependency is present in the toolchain, then you can explicitly pass --enable-<feature> to configure to enable that feature. You can edit the configure arguments for each package within menuconfig for the workorder.

Here are example lines for a dependency as defined in the Timesys SDL recipe:

ifeq (y,$(strip $(TSWO_SOFTWARE_tslib)))
sdl-unpack: tslib-host-install
sdl_CONFIGURE_OPTIONS+=--with-tslib=yes
endif

This ensures that tslib is installed prior to sdl when tslib is selected. The ifeq is necessary to allow sdl to build when tslib is not selected. See the sections below on editing package recipes and editing package options for more details on these operations. Also, notice the additional configure argument specified for sdl that forces the use of tslib in case the automatic detection does not work.

Adding a package

Select the desired package in within the menuconfig configuration. Any required dependencies will be automatically selected and built. Any packages required at runtime will automatically be added to the RFS. The source and patches will be fetched from the local mirror that is set in the workorder (as it was created when setting up the team environment). You will need to add sources from the main TimeSys mirror to your local mirror. If the source is not present on the local mirror, the fetch step of the build will end in a 404 error, similar to the following:

-- fetching fuse 2.7.2 -- 1227025609 [Tue, 18 Nov 2008 11:26:49 -0500]
http://corpshare.company.com/factory/src/f/fuse/fuse-2.7.2//fuse-2.7.2.tar.gz:
11:26:49 ERROR 404: Not Found.
make: *** [/home/user/factory/src/dl/f/fuse/fuse-2.7.2] Error 1

The source for a package can be fetched to the local mirror with a single wget command. For example (run this on the mirror):

wget -r -np -nH --cut-dirs 1 -R "index.html*" http://repository.timesys.com/buildsources/f/fuse/fuse-2.7.2/

The source is always stored in: 

http://repository.timesys.com/buildsources/<first_letter>/<name>/<name>-<version>/

When you add a package, you should simply be able to perform an incremental build. See the "Incremental Build" section of FactoryBestPractices.

Removing a package

This is already covered in the "I want to change my RFS application software selection" section of FactoryBestPractices.

Changing package or RFS options

Each package comes with a series of options that can be set via menuconfig. The most common options are to change the configure options (passed as arguments to ./configure), build options (passed as environment to make), or the list of patches. Again, you should not need to rebuild everything, but you should distclean the package and the RFS before performing the build. See the "I want to change my RFS application software selection" section of FactoryBestPractices. Also, configure options are referenced in the Autoconf section above, and patches are covered in "patching a package" below.

Adding a patch

To add a patch to a package, start by running the package's patch target (to extract the source code and apply any existing patches).

make busybox-patch

In order to create a patch, we need an "original" copy of any changed source files. For our example, make an original copy before editing each file. E.g.:

cp build_armv5l-timesys-linux-gnueabi/busybox-1.12.1/busybox-1.12.1/libbb/appletlib.c{,.orig}
vi build_armv5l-timesys-linux-gnueabi/busybox-1.12.1/busybox-1.12.1/libbb/appletlib.c

In the example, vi is used as an editor to change some of the usage messages in appletlib.c. Any editor may be used. Once we've made the changes, we can build busybox in place.

make busybox-package

Like with the changed config, the resulting package will include your change and can be used for testing. Also like that scenario, all changes so far are temporary. To make the change permanent, we must generate a patch and upload that patch to our source location. In the example, the diff is generated by comparing the .orig file to the changed file. The resulting patch is uploaded to the corpshare server that was established to hold source in the original step.

cd build_armv5l-timesys-linux-gnueabi/busybox-1.12.1/
diff -u busybox-1.12.1/libbb/appletlib.c{.orig,} > mycompany_bbusage.diff
scp mycompany_bbusage.diff engservices.mycompany.com:/var/www/factory/src/b/busybox/busybox-1.12.1/
cd ../..

Finally, use menuconfig to edit the busybox package options. Edit the list of patches. There may already be patches on the list that have been specified by Timesys. Generally these patches should be left in place and the name of the new patch appended to the end of the list. Once this change has been made, you can distclean the package and do a rebuild to verify that the change is complete.

make menuconfig
make busybox-distclean busybox-package

Tips

  • Editing source files and generating patches for a package are examples of working within the package source directory that is created during the build. It is important to remember that this is a temporary location, and any changes must be made permanent via patches, config files, or changes to the platform_config file.
  • Once a package has been built, the engine will not re-build it, even if you edit the source code or change the configuration. To force the engine to re-built a package, use the -restage make target for that package (in our case, 'make busybox-restage').
  • Don't forget that any changes made to .config must be copied to platform_config to be checked in.

Customizing Package Recipes

Each package that is selectable via the menuconfig interface comes with a "recipe", which is instructions on how to build this package. These come in two files that work together. Both of these files are stored in the engine under the target/software/ directory. Within that area, each package has its own directory (organized under Category directories). For example, the libxml2 recipe is in:

target/software/Application/libxml2/Config.in
target/software/Application/libxml2/libxml2.mk

The Config.in file defines (in KConfig format) the options that are visible within menuconfig for the package, as well as dependency information and some other unchangeable options. The .mk file is a makefile that uses the configuration options and builds the package accordingly. Generally, it is not necessary to know the details of the recipe. However, there are always cases where things don't go as expected, and understanding these recipes is key to understanding those cases. In addition, there may be specific customizations that are desired, but not currently support in the package recipe. In these cases, editing the package recipe is required.

All of the package recipes are based on a single common series of targets. Each contains a call to the COMMON_SETUP macro:

$(eval $(call COMMON_SETUP,libxml2))

This call uses the configuration options and sets up a series of default targets for building the package. These targets follow the standard procedure for building most open-source packages:

  • unpack
  • patch
  • configure (autoconf's ./configure, qmake, cmake, etc)
  • build (make)
  • install (make install)

This list is further extended as follows. A fetch stage is added at the beginning, which downloads the source and patches from the shared server. The install stage is broken into separate host-install and rfs-install stages. The host-install stage installs any libraries or headers into the toolchain. The rfs-install stage installs the packages files into the RFS. We also add an optional package stage which puts all of the packages output files into a tar or other package type. For each of these phases, a package-specific target is defined. For example:

  • libxml2-fetch
  • libxml2-unpack
  • libxml2-patch
  • libxml2-configure
  • libxml2-build
  • libxml2-host-install
  • libxml2-rfs-install
  • libxml2-package

Each of these is a "PHONY" target in make. This means that they do not directly create files. Instead, each is a front-end for a series of stamp files that control each phase and divide it into further sub-phases. The fetch target is controlled by the download directory (i.e. fetch is performed only if the download directory does not exist). The others break down as follows:

  • .stamp_unpacked
  • .stamp_patched
  • .stamp_pre_configured
  • .stamp_configured
  • .stamp_post_configured
  • .stamp_pre_built
  • .stamp_built
  • .stamp_post_built
  • .stamp_pre_host_installed
  • .stamp_host_installed
  • .stamp_post_host_installed
  • .stamp_pre_rfs_installed
  • .stamp_rfs_installed
  • .stamp_post_rfs_installed
  • .stamp_pre_packaged
  • .stamp_packaged
  • .stamp_post_packaged

These stamps are automatically defined to follow commands for the default method of building an autoconf-based package. The common rules are defined in include/common.mk . However, the easiest way to understand them is to look at the new package template in target/software/.template_mk . In the template, there are comments for each stamp that indicate the default commands that are used for the stamp. If these commands do not fit the package, then any or all of them may be overridden in the makefile for the package. Simply define a target for the stamp and it will override the default target. For example, busybox does not use autoconf. Therefore, the .stamp*_configured targets have been re-defined to interact with busybox's config system rather than using ./configure .

From looking at the target you can see that a number of make variables are automatically defined and can be used in the commands. Some, like TARGET_ALIASES, BUILD_PATH, and TSWO_RFS_INSTALL_DIR are common to all packages. Others, are specific to each package. Generally, these are defined based on the configuration values set via menuconfig and contain the package name. For example, for a package named "mypackage", mypackage_VERSION, mypacakge_URL, mypackage_CONFIGURE_OPTIONS, and mypackage_BUILD_OPTIONS would be defined, among others. In addition to these standards, any configuration value defined in the Config.in will be available to make as a variable with the same name. Also, a few package-specific variables not related to config options will be defined. E.g. mypacakge_DIR, mypackage_DDIR, and mypackage_PACKAGE_INSTALL_DIR. For more information on the variables and their meanings, see the "Common make variables" section in HackingTheFactory.

In addition to the rules for building the package, each makefile contains lines similar to the following:

ifeq (y,$(TSWO_SOFTWARE_libxml2))
SOFTWARE+=libxml2
endif

All packages listed in the SOFTWARE variable will be built into the root filesystem when running make. These lines add the package to that list, if the package has been selected (set to "y") in the menuconfig interface.

Developing custom applications

Using the platform toolchain to build a custom application

Building and customizing open source packages and applications provide much of the power of embedded Linux. Of course, usually custom and possibly proprietary applications are layered on top of this base open-source platform. In order to develop these applications the compiler, linker, libraries, and headers from the platform are necessary. All of these items are already present in the "final" toolchain from a Factory build, or via the build output development-environment.sh shareable installer. Once you have the toolchain in place, simply put its "bin" directory on the PATH (e.g. build_armv5l-timesys-linux-gnueabi/toolchain/bin/). Then change your application's Makefiles to use the properly named toolchain. Often this is done by setting CROSS_COMPILE or CC. (E.g. CC=armv5l-timesys-linux-gnueabi-gcc). The exact steps will depend on your application's build system. Libraries and headers from the platform will already be installed within the default search locations for the toolchain and will be available to the application. Once you've built the application, you may copy executables and other parts of the application to a built root filesystem and test them.

Integrating custom applications into Factory build

It is possible to indefinitely build custom applications directly using the toolchain and simply copy the application into the RFS (as described above). However, in many cases it is convenient to integrated your custom application directly into the platform build. This way, the custom application is seen as "just another package" and built along with the rest of the platform. To achieve this we will create a custom recipe, following the guidelines outlined in "Customizing Package Recipes" section, above. For a custom application, the Config.in file will be skipped entirely. The custom application will not be configurable or selectable via the menuconfig interface, but it will always be included in the RFS. The recipe will consist of just a Makefile that is stored in the target/software/Custom/ directory and ends with the extension .mk . Files following this pattern are automatically included when running make for the platform. Like with the packages, the most basic definition is just to call the setup macro and add yourself to the software list. Because there is no config, we must define a few basic definitions and then call the "CUSTOM_SETUP" macro:

myapp_EXTERNAL_SRC=$(BASE_DIR)/src/myapp
myapp_VERSION=1.0

$(eval $(call CUSTOM_SETUP,myapp))

SOFTWARE+=myapp

By setting the *_EXTERNAL_SRC variable in the recipe, the default rules will use an existing directory for the source, rather trying to download, unpack, patch, etc. This is very similar to the external source option that we used for the kernel in the "Using a custom kernel" section, above. The default rules will cd into the given directory, then run ./configure, make, make install. This should work well if autoconf is used for the application. However this example assumes that autoconf is not used. An empty rule will be defined to override the default behavior.

$(myapp_DIR)/.stamp_configured:
    @touch $@

Another common modification is adding a dependency. Often custom applications will require headers or libraries from other packages. These dependencies must be declared in the makefile. A dependency ensures that headers and libraries are available at build time. A runtime dependency ensures that the package will always be installed. For the example, the application will have both a dependency and runtime dependency on cracklib. Each can be declared using TSWO_myapp_DEPENDS and TSWO_myapp_RUNTIME_DEPENDS, respectively. Here is the final recipe for the simple app:

myapp_EXTERNAL_SRC=$(BASE_DIR)/src/myapp
myapp_VERSION=1.0
TSWO_myapp_DEPENDS=cracklib
TSWO_myapp_RUNTIME_DEPENDS=cracklib

$(eval $(call CUSTOM_SETUP,myapp))

SOFTWARE+=myapp

$(myapp_DIR)/.stamp_configured:
    @touch $@

It is also possible to do this as without having the external source directory setup. If you have a tarball and one or more patches, you can create a simple makefile like the following:

TSWO_myapp_VERSION="0.6" 
TSWO_myapp_URL="file://$(BASE_DIR)/src/mysrc/myapp-$(TSWO_myapp_VERSION)" 
TSWO_myapp_SOURCE="myapp-$(TSWO_myapp_VERSION).tar.gz" 
TSWO_myapp_PATCHES="myapp-$(TSWO_myapp_VERSION)-foo.patch" 
# optional
#TSWO_myapp_DEPENDS="" 
#TSWO_myapp_RUNTIME_DEPENDS="" 
#TSWO_myapp_CONFIGURE_OPTIONS="" 
#TSWO_myapp_BUILD_OPTIONS="" 
#TSWO_myapp_WORKINGDIR="" if not [name]-[version]

$(eval $(call COMMON_SETUP,myapp))
SOFTWARE+=myapp

$(myapp_DIR)/.stamp_configured:
    @touch $@
$(myapp_DIR)/.stamp_built:
    cd $(myapp_DDIR) && \
        make
    @touch $@

This assumes that the package can be built by calling "make" with CC, AR, LD, etc. set in the environment. Similarly it assumes that it can be installed by calling "make install" with DESTDIR in the environment set to the root directory for installation. If these assumptions do not match the application, then the rules for these stamp files can be overridden as well. See the "Customizing Package Recipes" area above, for a summary of the stamps that can be overridden and the environment variables that can be used.