Skip to content

Serde: Impl Avro type for timestamp-nanos logical type #872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
2 tasks done
ManuelRin opened this issue Feb 27, 2025 · 3 comments · May be fixed by #877
Open
2 tasks done

Serde: Impl Avro type for timestamp-nanos logical type #872

ManuelRin opened this issue Feb 27, 2025 · 3 comments · May be fixed by #877
Labels
area/serde Serialization & Deserialization (plugins) good first issue Up for grabs scope/backend Related to backend changes status/triage/completed Automatic triage completed type/enhancement En enhancement/improvement to an already existing feature

Comments

@ManuelRin
Copy link

Issue submitter TODO list

  • I've searched for an already existing issues here
  • I'm running a supported version of the application which is listed here and the feature is not present there

Is your proposal related to a problem?

The UI doesn't currently support decoding an Avro timestamp-nanos logical type to JSON when it's displayed in the UI. It does support other (older) timestamp types such as timestamp-micros though. The timestamp-nanos logical type fields are currently displayed as their underlying long (nanoseconds since the Unix epoch).

In the latest specification Avro 1.12.0 the new logical type timestamp-nanos (and local-timestamp-nanos) was added via [AVRO-3884].

Details:

Describe the feature you're interested in

It would be great to see timestamp-nanos values correctly decoded to a point in time, like timestamp-micros is presented, but with the necessary precision of course. Same should be done for the local-timestamp-nanos logical type (local time instead of UTC).

According to JsonAvroConversion.java as of today, there is support for timestamp-micros but not yet support for timestamp-nanos which would likely work very similar.

TIMESTAMP_MICROS("timestamp-micros",
(node, schema) -> {
if (node.isIntegralNumber()) {
// TimeConversions.TimestampMicrosConversion for impl
long microsFromEpoch = node.longValue();
long epochSeconds = microsFromEpoch / (1_000_000L);
long nanoAdjustment = (microsFromEpoch % (1_000_000L)) * 1_000L;
return Instant.ofEpochSecond(epochSeconds, nanoAdjustment);
} else if (node.isTextual()) {
return Instant.parse(node.asText());
} else {
throw new JsonAvroConversionException(
"node '%s' can't be converted to timestamp-millis logical type"
.formatted(node));
}
},
(obj, schema) -> new TextNode(obj.toString()),
new SimpleFieldSchema(
new SimpleJsonType(
JsonType.Type.STRING,
Map.of(FORMAT, new TextNode(DATE_TIME))))
),

Describe alternatives you've considered

Alternative to this feature is to take the presented integer value of the underlying long and offset it from the Unix epoch 'by hand'.

Version you're running

4cf17a0 (1.1.0)

Additional context

Unit tests should be added as well, code search for 'timestamp-micros' provides the relevant locations where tests for 'timestamp-nanos' would also be appropriate.

@ManuelRin ManuelRin added status/triage Issues pending maintainers triage type/feature A brand new feature labels Feb 27, 2025
@kapybro kapybro bot added status/triage/manual Manual triage in progress status/triage/completed Automatic triage completed and removed status/triage Issues pending maintainers triage labels Feb 27, 2025
Copy link

Hi ManuelRin! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues.
Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.

@Haarolean Haarolean added good first issue Up for grabs type/enhancement En enhancement/improvement to an already existing feature scope/backend Related to backend changes area/serde Serialization & Deserialization (plugins) and removed type/feature A brand new feature status/triage/manual Manual triage in progress labels Feb 28, 2025
@Haarolean Haarolean changed the title Avro type conversion for timestamp-nanos logical type Serde: Impl Avro type for timestamp-nanos logical type Feb 28, 2025
@Haarolean Haarolean moved this to Todo in Up for grabs Feb 28, 2025
@Haarolean
Copy link
Member

@wernerdv Hey, wanna tackle this? It should be straightforward considering we already have the code for a similar millis type.

@wernerdv
Copy link
Contributor

@Haarolean Yes, I will prepare a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/serde Serialization & Deserialization (plugins) good first issue Up for grabs scope/backend Related to backend changes status/triage/completed Automatic triage completed type/enhancement En enhancement/improvement to an already existing feature
Projects
Status: Todo
Development

Successfully merging a pull request may close this issue.

3 participants