DornerWorks

Building a Real-Time Petalinux Kernel for AMD Devices

Posted on March 21, 2023 by Taylor Prins

The world of technology is constantly evolving, and real-time computing is becoming an increasingly critical aspect of many applications. In the field of defense, accuracy and speed are of the utmost importance. Real-time computing is an essential requirement for many applications, where the accuracy and speed of operations are critical. In aerospace and defense, real-time computing is especially important as the systems must respond to threats quickly and accurately. A delay of even a few seconds can result in significant damage and loss of life. This is why the defense industry is constantly seeking out new ways to improve the performance and reliability of their systems.

With real-time computing, it is possible to achieve high levels of precision and accuracy in a short amount of time, making it a valuable tool in many areas, from air-traffic control to medical systems and beyond. A major defense contractor wanted to meet this goal, but also wanted to leverage their existing software stack, based on Linux, which is not a Real-Time Operating System (RTOS).

To solve this problem, DornerWorks engineers applied the Linux real-time patch to a AMD Petalinux kernel, along with several other steps, to provide better real-time behavior for their application.

Building the Kernel

The kernel version to be used was v2021.1. First, the AMD source code of the kernel was cloned at the ‘xlnx_rebase_v5.10_2021.1’ tag. The PREEMPT_RT patch version must be matched specifically to the Linux kernel version being used. Since the petalinux 2021.1 kernel being build uses the 5.10 linux kernel, the correct patch was obtained here: https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.10/older/patch-5.10-rt17.patch.gz

Other patch versions are available if using a different linux kernel version here:
https://wiki.linuxfoundation.org/realtime/preempt_rt_versions

To apply the patch the following command was used from within the linux-xlnx folder:

zcat //patch-5.10-rt17.patch.gz | patch -p1

The ZCU102 development board uses a zynqmp processor. To build the defconfig for the kernel, the following command was used:

make ARCH=arm64 xilinx_zynqmp_defconfig

Beyond the PREEMT_RT patch itself, several kernel configuration changes are also needed. Once the defconfig was built, `make menuconfig` was used to open the kernel configuration GUI. The following changes were necessary:

  • General setup —> Preemption Model —> Fully Preemptible Kernel (Real-Time)
  • Kernel features —> Timer frequency —> 1000 HZ
  • CPU Power Management —> CPU Frequency scaling —> (Disable)

The first step sets the preemption model necessary for real-time.

In the second step, the timer interrupt handler interrupts the kernel at a certain rate, which is determined by the HZ constant. The higher the frequency, the more precise the timer resolution. For instance, a timer granularity of 100 Hz results in a maximum resolution of 10ms, while 250Hz equals 4ms and 1000Hz gives the best-case resolution of 1ms. To achieve the best possible scenario for interrupt handling, the 1000Hz option is selected.

The third step disables any cpu scaling to ensure that the desired real-time effects are maintained. Larger latencies could be seen if cpu scaling was left enabled.

With the kernel changes made, the kernel can now be built using the following:

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

Once the kernel finishes building, the kernel image can be found in here: ./arch/arm64/boot/Image

Benchmarking the Kernel

Paired with the newly created PREEMPT_RT petalinux 2021.1 kernel was a petalinux 2021.1 rootfs (built prior using petalinux tools) that contained both the ‘rt-test’ and ‘stress-ng’ application packages. Adding these applications to the project are accomplished by following the petalinux tools guide here: https://docs.xilinx.com/r/2021.1-English/ug1144-petalinux-tools-reference-guide/Adding-an-Existing-Recipe-into-the-Root-File-System. These software packages allowed for the use of ‘cyclictest’ and ‘stress’ applications for benchmarking. The benchmarking procedure was the following for each test:

  1. Boot the kernel and rootfs in a ramdisk environment (ie… not using persistent storage).
  2. Start the stress application in the background as follows:

    stress --cpu 4 --io 4 --vm 2 --vm-bytes 128M &

  3. Start a cyclictest iteration and pipe its data to a file for 1 million cycles as follows:

    cyclictest --clock=1 --mlockall --nanosleep --priority=80 --verbose --loops=1000000 > testdata.dat

Once each test was finished, the output data file was copied to a development machine for post processing.

Benchmarking Results

The gathered data was clear and concise that the real-time kernel was in fact working properly. This can be seen in the vastly lower worst-case latencies observed during the cyclictest runs. Comparison of the data can be seen below.

Baseline (without PREEMPT_RT)

Figure 1: Cyclictest data of Petalinux 2021.1 w/o PREEMPT_RT
Figure 2: Cyclictest data of Petalinux 2021.1 w/o PREEMPT_RT – Zoomed

RT_PREEMPT patch applied

Figure 3: Cyclictest data of Petalinux 2021.1 w/ PREEMPT_RT patch applied
Figure 4: Cyclictest data of Petalinux 2021.1 w/ PREEMPT_RT patch applied – Zoomed

A foundation for real-time systems innovation

Though the ZCU102 is referenced in this project, none of the steps are specifically for that board. Since these directions are only for the kernel config and build, the steps to implement a real-time Linux kernel on a custom board should be the same, with some exceptions.

This project has provided valuable insights and a baseline comparison for future customers who want to run real-time applications using Linux. Developing real-time systems that stand out and deliver the accuracy and precision your customers require can be difficult, but new technologies shouldn’t hold you back. DornerWorks can help you leverage Linux as a license free soft real-time operating environment for your timing sensitive applications. Schedule a meeting with our team and we will help you turn your ideas into reality.

Taylor Prins
by Taylor Prins
Embedded Engineer
Taylor Prins is an embedded engineer at DornerWorks.