DornerWorks

Achieving Theoretical Maximum Performance with High-Accuracy Time-Synchronization over Ethernet

Posted on November 26, 2019 by David Verbree

The precision of modern networked devices is becoming more and more important as time-critical applications in aerospace, autonomous automotive, and other industries continue to develop.

There’s literally no time to waste. At least, there shouldn’t be, and leveraging the Precision Time Protocol (PTP) is making that possible.

PTP is a mechanism of synchronizing time between end-points over standard Ethernet. It was first standardized in IEEE 1588-2002 [1]. Since then, new versions and variants have been and are currently being developed to support protocols that demand greater accuracy or performance guarantees for time-synchronization.

Scheduling Examples

Audio Video Bridging (AVB) is a collection of standards that provides deterministic communication across the network by reserving bandwidth for data streams. A variant of PTP, called the generalized PTP (gPTP) was ratified as IEEE 802.1AS-2011 [3] to provide accuracy guarantees and performance enhancements that are needed to support AVB.

Time-Sensitive Networking (TSN) is another collection of emerging standards currently being drafted to augment AVB by accurately scheduling stream transmissions. It requires even greater time-synchronization accuracy guarantees across end-points. Therefore, a revision to IEEE 802.1AS (IEEE 802.1AS-REV [4]) is currently be drafted to support TSN.

In the development of DornerWorks’ own implementation of IEEE 802.1AS-REV end-point, we worked hard to maximize time-synchronization accuracy and performance well beyond the standard.

Here are some of the challenges we ran into and what we did to overcome them.

Time-synchronization is important to networked systems, and the Precision Time Protocol (PTP) makes that possible over Ethernet.

Clock Accuracy

According to the IEEE 802.1AS standards, gPTP implementations must obtain < 1 μs end-to-end synchronization accuracy for 7 or fewer hops during steady state conditions. Currently, the IEEE 802.1AS-REV draft is considering 1-μs end-to-end synchronization accuracy for 64 or fewer hops for industrial applications which is approximately 15.6 ns per hop.

This has been our goal as well.

Typically, the gPTP protocol is implemented in complicated software that requires a hard processor, DDR memory, and operating system services. The software running in an operating system may experience substantial and non-deterministic delays in retrieving the time-counter register value used for hardware timestamping of PTP packets due to context switches, task preemption, interrupt preemption, system call overhead, MDIO interface latency, etc.

We overcame these challenges using several unique approaches. The entire gPTP protocol was implemented in custom logic (FPGA) making all packet parsing, packet transmission, calculations, and clock adjustments highly deterministic. All calculations are made using 16-B math representing EPOCH time with a resolution down to the 2(-16) nanoseconds to prevent any loss of precision.

The clock increment base value (e.g. 8 ns) is scaled up 224 to allow very fine adjustments that are applied to the local clock at the link rate frequency (125 MHz for a 1-Gb link). It is updated with every sync message received from the GM (up to 10/s) using a P-I loop to minimize the master-local offset. This compensates for most frequency differences or phase offsets between the device and the grandmaster as well as any indeterminism introduced via PHY interfaces or the PHY itself. The proportional and integral gain constants may be modified either statically or dynamically to minimize convergence time or maximize clock stability and accuracy.

Clock Distribution

An accurately synchronized local clock is of little use if it cannot be distributed accurately, especially to software processes running on the same device. It is relatively easy to synchronize software applications to system time derived from a real-Time Clock (RTC) or CPU clock. However, it is not as easy to synchronize software to a synchronized network time that is only accessible through a physical network device (e.g. NIC, PHY, or MAC) without incurring high and non-deterministic latency as previously described.

We developed a general-purpose solution that is ideal for distribution to other custom logic as well as for routing to a hard processor via an interrupt controller. Included within the Dornerworks AVB-capable MAC IP core are eight user-configurable triggers that are externally available. Each trigger is configurable with the start time in EPOCH time down with 1-ns resolution, the period in nanoseconds up to 1s for recurring triggers, and the number of iterations to trigger on. Each trigger is asserted when the synchronized network time is within 8 ns of user-configured trigger time.

When routed to an interrupt controller, the user triggers can synchronize software in multiple ways. They can directly drive the system tick and/or OS scheduler to ensure a process awakes at a synchronized time interval. They can also be configured as general-purpose periodic timers that an application can register interrupt handlers for. Whether routed to an interrupt controller or to down-stream logic, they provide an effective way to clock distributed hardware or processes nearly in lock-step.

Driverless vehicles, unmanned aircraft, and other autonomous systems are highly dependent on time-synchronous processes.

Usability

The most notable feature of this approach is that time-synchronization is done automatically behind the scenes. System integrators and software developers do not need to allocate any additional resources (CPU processing time, DDR memory, etc.) to support time synchronization.

It is entirely transparent, the way time-synchronization should be.

The Results

We instantiated two of our end-points on Zed Boards with quad-port FMC Ethernet cards connected to an AVB-capable switch and configured each to provide a 1-PPS output at the top of the second of synchronized time. We connected both 1-PPS outputs to a 1-GHz oscilloscope to verify synchronization accuracy.

The difference between the rising-edge of both signals was less than 8-ns and stable. This is the theoretical maximum accuracy achievable with a 1-Gbps clock frequency of 125 MHz.

Achieving accuracy this precise is becoming more and more important to time-critical applications as developers start to implement AVB and TSN technology into their products. From driverless vehicles to complex sensor applications, DornerWorks is helping companies develop those projects now, readying them for even greater opportunities in the future.

References

[1] “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems,” IEEE Std 1588-2002, pp. i–144, 2002.
[2] “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems,” IEEE Std 1588-2008 (Revision of IEEE Std 1588-2002), pp. 1–300, Jul. 2008.
[3] “IEEE Standard for Local and Metropolitan Area Networks – Timing and Synchronization for Time-Sensitive Applications in Bridged Local Area Networks,” IEEE Std 802.1AS-2011, pp. 1–292, Mar. 2011.
[4] “IEEE Draft Standard for Local and Metropolitan Area Networks – Timing and Synchronization for Time-Sensitive Applications,” IEEE P802.1AS-Rev/D6.0 December 2017, pp. 1–496, Jan. 2018.

David Verbree
by David Verbree
Embedded Engineer
David Verbree is an embedded engineer at DornerWorks.