Timecounters for NetBSD

scalable time keeping infrastucture

Even Multiprocessor can keep good time
System: ASUS A8V, Athlon 64 X2 4400+, NetBSD simonb-timecounters, TSC timecounter, MP mode


Timecounters provide a scalable MP capable high precision time keeping infrastructure. They where developed by Poul-Henning Kamp for the FreeBSD project. For more information have a look at his paper:
       Timecounters: Efficient and precise timekeeping in SMP kernels.

Poul-Henning Kamp

The FreeBSD Project


The FreeBSD timecounters are an architecture-independent

implementation of a binary timescale using whatever hardware support

is at hand for tracking time. The binary timescale converts using

simple multiplication to canonical timescales based on micro- or

nano-seconds and can interface seamlessly to the NTP PLL/FLL

facilities for clock synchronisation. Timecounters are implemented

using lock-less stable-storage based primitives which scale

efficiently in SMP systems. The math and implementation behind

timecounters will be detailed as well as the mechanisms used for


Possible Future Developments:

Graphics before and after:

The following graphs show time offsets and clock frequency corrections applied by ntp, before and after timecounters.  Pay particular attention to the difference in vertical scale (up to a factor of 21000 for the SOEKRIS case) between the before and after graphs.

No Timecounters: Multiprocessor Stratum 1 operation before on the A8V

Timekeeping on multiprocessor before time counters

Timecounters: Stratum 1 (Meinberg GPS167) - Sample of a busy A8V + Athlon 64 X2 doing 2*setiathome + build.sh -j 3 release:
Loopfilter graph
At this state the synchronisation results are usually strongly influenced by temperature changes and interrupt latencies.

No Timecounters: SOEKRIS4801 with NetBSD-current:
Timekeeping on SOEKRIS without Timecounters

Timecounters: Sample of an idle SOEKRIS4801 Stratum 1 Server (Meinberg GPS167)
Loopfilter graph

Timecounters: Now if we could get rid of the up to 8us spikes I am seeing it would look more like this:
Loopfilter graph

Converting a port to Timecounters

Ideally all ports should be converted to timecounters so that the old microtime implementation can be completely retired and we can get rid of the dual code (__HAVE_TIMECOUNTER). The requirements for timecounter support are minimal.

Requirements for timecounter implementations:
  • fixed frequency incrementing counter wrapping at a power of two.
  • counter must not wrap within 2/hz - if the counter is wrapping too fast consider increasing hz (e.g. ELAN SC520 boards need a HZ of 150 to utilize the counter in that architecture)
  • if no hardware counters exist for microtime interpolation then a port may quickly be converted by fixing the setting of the clock and defining __HAVE_TIMECOUNTER as a default timecounter "clockinterrupt" based on clock ticks is always registered.
  • count down counters must be converted to up counting ones (result = MAX_COUNT - current_count).
  • counters not wrapping at a power of two need to be converted. see also arch/i386/isa/clock.c for examples.
  • you need mutex(IPL_SPLHIGH) guards when you need to manipulate/access state that is updated by clock interrupts or by counter reading. This includes chip access (address, data phases) and local state manipulation for emulation of power of two timers.
  • for MP operation you need to insure that the returned counter values are consistent across cpus. either a single and atomically readable hw counter is available..