Timecounters for NetBSD

scalable time keeping infrastucture

System: ASUS A8V, Athlon 64 X2 4400+, NetBSD simonb-timecounters, TSC timecounter, MP mode

Overview:

Timecounters provide a scalable MP capable high precision time keeping infrastructure. They where developed by Poul-Henning Kamp for the FreeBSD project. For more information have a look at his paper:

       Timecounters: Efficient and precise timekeeping in SMP kernels.

                              Poul-Henning Kamp

                             The FreeBSD Project

                                   ABSTRACT



          The FreeBSD timecounters are an architecture-independent

    implementation of a binary timescale using whatever hardware support

    is at hand for tracking time. The binary timescale converts using

    simple multiplication to canonical timescales based on micro- or

    nano-seconds and can interface seamlessly to the NTP PLL/FLL

    facilities for clock synchronisation.  Timecounters are implemented

    using lock-less stable-storage based primitives which scale

    efficiently in SMP systems. The math and implementation behind

    timecounters will be detailed as well as the mechanisms used for

    synchronisation.

Possible Future Developments:

Cleanup Timescale usage: Some services need wall clock time others need monotonic time. There are many areas in the kernel where not the right scale is uses (e. g. timeout are calculated using wall clock time though the should actuall use monotonic time)
timeout functions for select/poll/kevent/nanosleep have been converted

overhaul all kernel internal timekeeping/-storage to use struct bintime and just convert at the API boundaries.
implement higher precision timeouts when single shot hw timers are available (could get also rid of periodic interrupt and maybe even reduce power consumption

Graphics before and after:

The following graphs show time offsets and clock frequency corrections applied by ntp, before and after timecounters. Pay particular attention to the difference in vertical scale (up to a factor of 21000 for the SOEKRIS case) between the before and after graphs.

No Timecounters: Multiprocessor Stratum 1 operation before on the A8V

Timekeeping on multiprocessor before time counters

Timecounters: Stratum 1 (Meinberg GPS167) - Sample of a busy A8V + Athlon 64 X2 doing 2*setiathome + build.sh -j 3 release:

At this state the synchronisation results are usually strongly influenced by temperature changes and interrupt latencies.

No Timecounters: SOEKRIS4801 with NetBSD-current:

Timekeeping on SOEKRIS without Timecounters

Timecounters: Sample of an idle SOEKRIS4801 Stratum 1 Server (Meinberg GPS167)

Timecounters: Now if we could get rid of the up to 8us spikes I am seeing it would look more like this:

Converting a port to Timecounters

Ideally all ports should be converted to timecounters so that the old microtime implementation can be completely retired and we can get rid of the dual code (__HAVE_TIMECOUNTER). The requirements for timecounter support are minimal.

Requirements for timecounter implementations:

fixed frequency incrementing counter wrapping at a power of two.
counter must not wrap within 2/hz - if the counter is wrapping too fast consider increasing hz (e.g. ELAN SC520 boards need a HZ of 150 to utilize the counter in that architecture)
if no hardware counters exist for microtime interpolation then a port may quickly be converted by fixing the setting of the clock and defining __HAVE_TIMECOUNTER as a default timecounter "clockinterrupt" based on clock ticks is always registered.
count down counters must be converted to up counting ones (result = MAX_COUNT - current_count).
counters not wrapping at a power of two need to be converted. see also arch/i386/isa/clock.c for examples.
you need mutex(IPL_SPLHIGH) guards when you need to manipulate/access state that is updated by clock interrupts or by counter reading. This includes chip access (address, data phases) and local state manipulation for emulation of power of two timers.
for MP operation you need to insure that the returned counter values are consistent across cpus. either a single and atomically readable hw counter is available..