sys: realtime: Add native real-time library and API

[bjarki-andreasen]

Introduction

This PR adds native APIs for setting, getting and working with real-time as unix timestamps and struct tm like datetimes.

Overview

The following diagram presents an overview of the system realtime library.

The system realtime clock, which is the base for sys_realtime_get_/set_ APIs, is a settable complimentary clock to the sys_clock, which is the base for k_uptime(). The system realtime clock can be implemented using any high accuracy clock. This PR provides an implementation which uses the sys_clock as the base clock. Similarly, the implementation of the conversion functions sys_realtime_timestamp_to_datetime and opposite can be selected at build time. This PR provides a native implementation of said APIs.

System realtime API

The top level API exposed to the user provides the following APIs

struct sys_datetime {
	int tm_sec;	/**< Seconds [0, 59] */
	int tm_min;	/**< Minutes [0, 59] */
	int tm_hour;	/**< Hours [0, 23] */
	int tm_mday;	/**< Day of the month [1, 31] */
	int tm_mon;	/**< Month [0, 11] */
	int tm_year;	/**< Year - 1900 */
	int tm_wday;	/**< Day of the week [0, 6] (Sunday = 0) (Unknown = -1) */
	int tm_yday;	/**< Day of the year [0, 365] (Unknown = -1) */
	int tm_isdst;	/**< Daylight saving time flag [-1] (Unknown = -1) */
	int tm_nsec;	/**< Nanoseconds [0, 999999999] (Unknown = 0) */
};

/** Get universal coordinated time unix timestamp in milliseconds */
__syscall int sys_realtime_get_timestamp(int64_t *timestamp_ms);
/** Set universal coordinated time unix timestamp in milliseconds */
__syscall int sys_realtime_set_timestamp(const int64_t *timestamp_ms);

/** Get universal coordinated time datetime */
__syscall int sys_realtime_get_datetime(struct sys_datetime *datetime);
/** Set universal coordinated time datetime */
__syscall int sys_realtime_set_datetime(const struct sys_datetime *datetime);

/** Convert universal coordinated time datetime to unix timestamp in milliseconds */
int sys_realtime_datetime_to_timestamp(int64_t *timestamp_ms, const struct sys_datetime *datetime);
/** Convert universal coordinated time unix timestamp in milliseconds to datetime */
int sys_realtime_timestamp_to_datetime(struct sys_datetime *datetime, const int64_t *timestamp_ms);

Thje get/set functions are syscalls, making them available to userspace, the conversion functions don't touch any kernel objects so they are not implemented as syscalls.

Configuring initial time source

Setting realtime automatically on boot from a hardware RTC can be configured by using the chosen zephyr,rtc.

/ {
	chosen {
		zephyr,rtc = &rtc;
	};
};

fixes: #35333

[bjarki-andreasen]

Update

Added counter support

Using a low-power counter / real-time counter as timekeeper is now supported!

Added test suite for time conversion functions

The test suite tests conversions from timestamps all the way back from year -1 to year 2399

[bjarki-andreasen]

Note to self, namespace SYS_TIME

[cfriedt]

Looks ok to me. In terms of timestamp, is that seconds since the epoch?

[bjarki-andreasen]

@cfriedt yep :) we could up the resolution to ms as well, we got the bits for it

[ghost]

IIRC, you argued that rtc.h is closely coupled to a specific type of hardware - which I consider essential as otherwise it shouldn't be in the driver subsystem.

Therefore please call this sys_rtc_time... or any other _rtc_ prefix or infix and place it outside of the generic sys/time.h (here and elsewhere in this PR where you're not in the direct realm of the RTC driver API but reference drivers/rtc.h.

The time namespace is so crowded already in Zephyr that generic namespaces like sys_time should not be occupied by specific libraries.

Alternatively use generic POSIX tm and refer to it in the docs. I know that the two concepts are compatible but the conversion should then be done in the implementation IMHO.

In any case the API lacks parameter documentation, it seems.

[bjarki-andreasen]

Its the other way around really :)

The RTC API itself is closely coupled to time.h, struct rtc_time is a 1-1 match to struct tm. The reason for this choice was to keep compatibility with POSIX, while not being dependent on it. This is the reason for using it over struct tm in the general API, not because of RTC drivers. We could move the struct out of rtc.h, have it in sys/time.h instead to make this clearer, actually, we should so that :)

[ghost]

Yep, you can keep them separate in both directions and only have a specific implementation refer to both at the same time, but if you intend to implement within a POSIX namespace (or at least a namespace that can easily be confused with <sys/time.h>, then you should steer clear from any imports deep into a single driver subsystem from a generic header file. If this was in sys/rtc_time.h or ...rtc/time.h (haven't thought much about the specific locations), it would be much clearer which kind of dependencies you imply here.

[ghost]

Same here, the configuration should either be done in the RTC driver realm or refer to generic time concepts independent of a specific hardware type. This API is not general enough to represent general POSIX time.

[bjarki-andreasen]

I will make an attempt to use this API as the basis for posix time, then we can discuss further. I will probably meet some of the hurdles you foresee :)

[ghost]

Yes, then please keep this experimental/unstable API encapsulated in the appropriate subsystem until it keeps the promise of being generic enough to be placed in a generic namespace. You can always move it later one once you make it represent general POSIX time.

[ghost]

Please make it clear that we're dealing with Unix timestamps here, as the timestamp name is also highly fraught in Zepyhr already.

[bjarki-andreasen]

I will update the comments to specify unix time

[ghost]

nit: reversed x-mas - here and elsewhere

[bjarki-andreasen]

I will reverse the reversed names :)

[ghost]

Full disclosure: There's no official consent about this in the project AFAIK, I just used to get comments in that direction myself and got to like it. But I think no one really has a strong opinion about this (myself definitely not).

[ghost]

You seem to be making implicit rounding assumptions here, please document these assumptions and name the function accordingly. Also document your assumptions about overflow (if any), as well as any other concept relevant to counter-to-time-conversion (see the well-defined concepts and nomenclature in #19030 (comment)).

[bjarki-andreasen]

Converting between ticks and seconds is quite trivial, in physics terms:

frequency = ticks / second
ticks / (ticks / second) = second

One may describe the fact that the ticks will always be rounded down to the nearest second as an assumption, but this is simply the expected behavior of a UTC timestamp, I would hardly describe this as an assumption.

Regarding roll-over, the counter may roll over, the implementation expects it to. When it does, the counter will start from 0, and the total ticks are added to an accumulated ticks since the counter was started. There are no assumptions made here that are not defined by the counter API.

[ghost]

I'm aware of the formula.

Have a look at how the well-conceived timepoint and timeout concepts in the kernel, the Linux kernel clocksource and my own counter-to-time conversion API proposal are all extremely careful to keep the "tick-version" well apart from the "rounded version" (potentially non-deterministic, accruing rounding errors over subsequent conversions, ...). They all go to great lengths in explaining in which situations the different "worlds" actually make sense and can be used legitimately. It's usually better to stick to the tick representation internally and make all conversions "ephemeral" with a very well defined epoch, rounding approach, syntonization requirements, etc. You should not give up the original "wrapped" tick-value "truth" prematurely, but if you have to then the chosen approach should be very well defined.

These things have been discussed many times before in the project, that's why I invited @andyross to this PR and pointed you to the Clock Architecture PR which documents that discussion quite well, I hope.

Some (not all) of these requirements go away as soon as you conceive your API as something specific to a specific driver universe.

[ghost]

@andyross Would be great if you could give this a quick review wrt tick-to-counter conversion, duplication with other concepts and encapsulation/openness towards other system timepoint representations.

[ghost]

counter != time-to-counter conversion - I suggest you name this properly as a conversion library (with very specific underlying assumptions) - again to not occupy generic namespaces with specific implementations.

[bjarki-andreasen]

The name of the source file just needs to be specific enough to separate it from other files in the same folder, the name is not exposed anywhere outside the folder, so it is not occupying any generic namespace.

[ghost]

Don't agree. We have been there and it has been highly confusing to users. We are speaking about highly visible header files. I vote for code that is not only "formally" correct but also helps users to not be trapped by misleading names.

[ghost]

The specific kind of counter and conversion assumptions should be decoupled from sys_time. Either make it clear where future decoupling can be done (even if not required now) or name the function more specifically and place it somewhere close to the specific type of counter API you're expecting here.

[bjarki-andreasen]

I don't agree with this comment. The counter API has no relevance to the sys_time API, neither does the struct rtc_time have any relevance to the RTC API. Implementations of the sys_time API may use either, but the entire point is to abstract this away from the user, who simply wants to get/set and convert time.

I understand that the fact that we are reusing the struct rtc_time makes it seem like this is directly related to the rtc device driver API, it is not, sys_time works just fine on systems that don't have an RTC...

[ghost]

If you want to keep up the API as being generic, then please rename the struct such that the name speaks of itself and remove the rtc_time definition from the driver subsystem rtc.h to a more general place. This is not my recommendation, though, see above, as it will be very hard to reach the level of openness to cover the kinds of use cases we already know about. Like it is now it's unnecessarily confusing in a place where people could expect well-known concepts and names.

[bjarki-andreasen]

How about naming it sys_realtime? Its a library handling interactions with and conversions between real-time clocks and real-time counters?

[cfriedt]

It would be kind of great to have 1 header file for converting between zephyr time representations.

Naming things is hard, but if it can be added to an existing header, it's much easier.

[ghost]

Yep, you can keep them separate in both directions and only have a specific implementation refer to both at the same time, but if you intend to implement within a POSIX namespace (or at least a namespace that can easily be confused with <sys/time.h>, then you should steer clear from any imports deep into a single driver subsystem from a generic header file. If this was in sys/rtc_time.h or ...rtc/time.h (haven't thought much about the specific locations), it would be much clearer which kind of dependencies you imply here.

[ghost]

Yes, then please keep this experimental/unstable API encapsulated in the appropriate subsystem until it keeps the promise of being generic enough to be placed in a generic namespace. You can always move it later one once you make it represent general POSIX time.

[ghost]

Full disclosure: There's no official consent about this in the project AFAIK, I just used to get comments in that direction myself and got to like it. But I think no one really has a strong opinion about this (myself definitely not).

[ghost]

I'm aware of the formula.

Have a look at how the well-conceived timepoint and timeout concepts in the kernel, the Linux kernel clocksource and my own counter-to-time conversion API proposal are all extremely careful to keep the "tick-version" well apart from the "rounded version" (potentially non-deterministic, accruing rounding errors over subsequent conversions, ...). They all go to great lengths in explaining in which situations the different "worlds" actually make sense and can be used legitimately. It's usually better to stick to the tick representation internally and make all conversions "ephemeral" with a very well defined epoch, rounding approach, syntonization requirements, etc. You should not give up the original "wrapped" tick-value "truth" prematurely, but if you have to then the chosen approach should be very well defined.

These things have been discussed many times before in the project, that's why I invited @andyross to this PR and pointed you to the Clock Architecture PR which documents that discussion quite well, I hope.

Some (not all) of these requirements go away as soon as you conceive your API as something specific to a specific driver universe.

[ghost]

If you want to keep up the API as being generic, then please rename the struct such that the name speaks of itself and remove the rtc_time definition from the driver subsystem rtc.h to a more general place. This is not my recommendation, though, see above, as it will be very hard to reach the level of openness to cover the kinds of use cases we already know about. Like it is now it's unnecessarily confusing in a place where people could expect well-known concepts and names.

[ghost]

It would be kind of great to have 1 header file for converting between zephyr time representations.

Naming things is hard, but if it can be added to an existing header, it's much easier.

I agree, that would be great, but the problem has been underestimated by many in the project already. There are lots of different ways to convert between pairs of time representations, especially when they come from different layers in the clock hierarchy as is the case here: counter layer vs. synchronized time layer - leaving out the syntonization layer in between w/o even mentioning it.

If you don't have good intermediate representations (single point of truth) on those three layers as well as APIs to manipulate them properly, then you get a combinatoric explosion of time types and conversion approaches. Hub and spoke is the only approach that works for proper time conversion IMHO.

Just think of leap second smearing (or whatever of the many approaches) in the UTC clock and the large amount of established syntonization approaches across system borders which are both huge problems for long-running high-precision timing systems as those driven by Zephyr. And there are many more.

Due to our lack of proper time representation in Zephyr, we currently experience that kind of combinatoric explosion. This PR is just one example of many. Linux has gone a long way to converge towards proper intermediate time representations. But our challenge is even bigger, as we cannot sensibly let go of truely deterministic timing even in higher clock layers (syntonization, synchronization), something Linux mostly fails to deliver today.

Wrt this PR: If we cannot avoid another arbitrary experimental time conversion library, then I vote for keeping it somewhere down in one driver subsystem where it originates. At least this would properly document the dependency of POSIX time (which seems to be the implied use case) onto one or two driver subsystems.

In any case I vote against another flawed approach of "general" time conversion clogging our common namespace,

[ghost]

How about naming it sys_realtime?

"Realtime" is still much too overloaded as a namespace. As long as you don't keep the promise of actually dealing with what "realtime" means in other systems, as long as we don't even reach the Linux benchmark for real time (much less deterministic synchronized clocks as required by several RTOS use cases), then this name promises far too much.

Let's face it: This is an RTC driver/hardware (plus maybe counter subsystem) specific conversion library that might even solve an immediate problem in the POSIX layer for a little while but that will have to be replaced as soon as POSIX APIs want to expose something that is similar to what "real time" means on other systems. But it doesn't even remotely integrate the existing time representations in Zephyr not to speak of those that are clearly on the horizon.

I'm totally open to help you start with the nucleus of a well-defined time representation in Zephyr, but this PR is not the way to go to achieve this IMO. We have experimental implementations and specific APIs that go far beyond this PR already, so you'd first have to go beyond those.

[ghost]

@cfriedt Just so I don't pull too much in the wrong direction.

Another idea: What if we conceived this API as being part of the internal POSIX layer implementation (but not more), just remove the dependencies on specific hardware from the header files and hide the RTC (or other current hardware dependencies) in the .c files rather than in the .h files and let the .h files refer to POSIX concepts only. Then this API would again be well encapsulated inside a subsystem - which would also be fine from my pov.

If we ever introduce proper intermediate time representations we can just introduce them in the implementation w/o having to change anything in the header files or outside the POSIX subsystem.

[cfriedt]

I would say, the basis for POSIX time is struct timespec rather than a 64-bit timestamp.

struct rtctime could be converted to a 64 bit ms counter since the epoch and then that could be converted into a struct timespec to set CLOCK_REALTIME for example.

That would provide a kind of coarse granularity "clock recovery" on power-on (in the absence of synchronizing via e.g. NTP or PTP).

Additionally, I think that would make a great SYS_INIT() feature of an rtc subsystem, along with shell utilities and so on. From that perspective, this code would be a consumer of the POSIX API.

Hard to say where to park this, but since it's not part of any standard, I would steer clear of sys/time.h or time.h since they are standard and kind of overloaded already.

[andyross]

Answering the summons, though I'll note that while I know timing hardware and common traps with timer drivers and timing-sensitive app code, I'm really not much of an expert on Gregorian calendar stuff, except to say that I know it's historically difficult to get right.

Basically the worry here is that we're inventing a novel implementation of a library that the rest of the world has messed up repeatedly. :)

One thing that would be good to see is a host-side (or native_posix) unit test that exhaustively runs this and compares it against e.g. the glibc gmtime() implementation. Like, pick random timestamps in the range of "all the times we'd expect this code to ever be used" and verify that it never differs over a few hundred billion iterations or whatever.

[andyross]

Not really understanding why the defaults are so limited here. The rtc_time struct is basically just a Unix struct tm and the year field is a 32 bit signed int with the zero at 1900.

[andyross]

So, this isn't something called only when e.g. CONFIG_ASSERT=y, it's on the path to every call. And this is actually a lot of code. What is it trying to protect agaist? If the user passes an incorrectly constructed date, it's IMHO perfectly legal to hand them back a nonsense timestamp. Likewise if the RTC driver gives you a broken struct, it's not your problem to solve here.

[andyross]

API nitpick: why does this function exist? What does it mean to "initialize" a date that doesn't actually (by construction!) refer to an actual date. This literally produces an invalid struct per the validate() call above. ;)

As I see it this should just be removed. You initialize a struct rtc_time by getting one from a driver or by converting a timestamp.

[cfriedt]

What about "rtcutil" as a prefix instead of "sys_realtime"? Just a thought

[ghost]

I propose s/sys{_realtime}?/rtc/g, then place the API in rtc.h, see similar conversion APIs in other driver subsystems as counter and sensor. Let it mature there and expose it more prominently once it has proven useful to at least one more subsystem. This feels much more natural and resolves the encapsulation and dependency issue nicely.

If the POSIX layer or an application want to introduce a direct dependency to the RTC driver subsys (rtc.h) as one of several possible realtime sources then it can still do so - but then it's not a forced dependency and one that is well visible as being bound to a specific hardware type by looking at the names.

If that's not fine, then let's rediscuss in the architecture WG. But please document the rtc.h dependency in your diagram above, so we can properly see what's actually going on in that API.

[bjarki-andreasen]

I propose we wait until I have had time to address the existing comments, before discussing further :) I'm working on it ;)

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[madscientist159]

Is this still moving forward? Right now we're having to do yet another roll-your-own solution to handle the fact that the only concept of time Zephyr has is its own uptime. Uptime alone doesn't really work for network enabled applications or even for system event logging, and something better than an application-specific offset + uptime workaround would be greatly appreciated.

[bjarki-andreasen]

Its quite dormant since I am spending most of my time with power management currently, I think the solution in this PR is nice, the only thing I remember needing to add was handling day of week and day of year conversions (if anyone wants that), would you like to help finalize this?

[bjarki-andreasen]

oh, and also, there was a posix clock test which was failing for some target run in Renode, 99% certain it was just some CI instability, but that was the last thing I was dealing with before switching tasks :)