Skip to content

RUST-1129 Make the dependency on chrono optional and provide equivalent time interop #352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Apr 15, 2022
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .evergreen/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -181,9 +181,9 @@ axes:
- id: "extra-rust-versions"
values:
- id: "min"
display_name: "1.48 (minimum supported version)"
display_name: "1.53 (minimum supported version)"
variables:
RUST_VERSION: "1.48.0"
RUST_VERSION: "1.53.0"
- id: "nightly"
display_name: "nightly"
variables:
Expand Down
2 changes: 1 addition & 1 deletion .evergreen/run-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ set -o errexit

. ~/.cargo/env
RUST_BACKTRACE=1 cargo test
RUST_BACKTRACE=1 cargo test --features chrono-0_4,uuid-0_8
RUST_BACKTRACE=1 cargo test --all-features

cd serde-tests
RUST_BACKTRACE=1 cargo test
7 changes: 5 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,11 @@ exclude = [
[features]
default = []
# if enabled, include API for interfacing with chrono 0.4
chrono-0_4 = []
chrono-0_4 = ["chrono"]
# if enabled, include API for interfacing with uuid 0.8
uuid-0_8 = []
# if enabled, include API for interfacing with time 0.3
time-0_3 = []
# if enabled, include serde_with interop.
# should be used in conjunction with chrono-0_4 or uuid-0_8.
# it's commented out here because Cargo implicitly adds a feature flag for
Expand All @@ -47,7 +49,7 @@ name = "bson"

[dependencies]
ahash = "0.7.2"
chrono = { version = "0.4.15", features = ["std"], default-features = false }
chrono = { version = "0.4.15", features = ["std"], default-features = false, optional = true }
rand = "0.8"
serde = { version = "1.0", features = ["derive"] }
serde_json = { version = "1.0", features = ["preserve_order"] }
Expand All @@ -58,6 +60,7 @@ lazy_static = "1.4.0"
uuid = { version = "0.8.1", features = ["serde", "v4"] }
serde_bytes = "0.11.5"
serde_with = { version = "1", optional = true }
time = { version = "0.3.9", features = ["formatting", "parsing", "macros", "large-dates"] }

[dev-dependencies]
assert_matches = "1.2"
Expand Down
17 changes: 12 additions & 5 deletions src/bson.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ use std::{
fmt::{self, Debug, Display, Formatter},
};

use chrono::Datelike;
use serde_json::{json, Value};

pub use crate::document::Document;
Expand Down Expand Up @@ -311,6 +310,14 @@ impl From<oid::ObjectId> for Bson {
}
}

#[cfg(feature = "time-0_3")]
#[cfg_attr(docsrs, doc(cfg(feature = "time-0_3")))]
impl From<time::OffsetDateTime> for Bson {
fn from(a: time::OffsetDateTime) -> Bson {
Bson::DateTime(crate::DateTime::from(a))
}
}

#[cfg(feature = "chrono-0_4")]
#[cfg_attr(docsrs, doc(cfg(feature = "chrono-0_4")))]
impl<T: chrono::TimeZone> From<chrono::DateTime<T>> for Bson {
Expand Down Expand Up @@ -430,7 +437,7 @@ impl Bson {
})
}
Bson::ObjectId(v) => json!({"$oid": v.to_hex()}),
Bson::DateTime(v) if v.timestamp_millis() >= 0 && v.to_chrono().year() <= 99999 => {
Bson::DateTime(v) if v.timestamp_millis() >= 0 && v.to_time().year() <= 99999 => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if the year constraint here is correct - it's different than the one below.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from my reading of the extJSON spec we should drop a 9 here to match below... see datetime entries in this table.

I was wondering how we would pass the corresponding corpus test, but it looks like that test case does not have a relaxed extJSON version specified so we likely make no assertions on it -- maybe it's omitted because it's the same as the canonical, but IMO it seems like test runners should probably confirm their relaxed output in that case matches canonical. perhaps we should add an assertion to our corpus runner so that for a test case with no expected relaxed output, we check that the relaxed output is equal to the expected canonical output?

if that just works, I'd propose we file a DRIVERS ticket that suggests amending the corpus spec to require this assertion.

thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like that doesn't just work, unfortunately. I filed RUST-1261 to follow up on that.

json!({
"$date": v.to_rfc3339_string(),
})
Expand Down Expand Up @@ -592,7 +599,7 @@ impl Bson {
Bson::DateTime(v) if rawbson => doc! {
"$date": v.timestamp_millis(),
},
Bson::DateTime(v) if v.timestamp_millis() >= 0 && v.to_chrono().year() <= 9999 => {
Bson::DateTime(v) if v.timestamp_millis() >= 0 && v.to_time().year() <= 9999 => {
doc! {
"$date": v.to_rfc3339_string(),
}
Expand Down Expand Up @@ -776,8 +783,8 @@ impl Bson {
}

if let Ok(date) = doc.get_str("$date") {
if let Ok(date) = chrono::DateTime::parse_from_rfc3339(date) {
return Bson::DateTime(crate::DateTime::from_chrono(date));
if let Ok(dt) = crate::DateTime::parse_rfc3339_str(date) {
return Bson::DateTime(dt);
}
}
}
Expand Down
185 changes: 150 additions & 35 deletions src/datetime.rs
Original file line number Diff line number Diff line change
@@ -1,16 +1,25 @@
use std::{
convert::TryInto,
error,
fmt::{self, Display},
result,
time::{Duration, SystemTime},
};

#[cfg(all(feature = "serde_with", feature = "chrono-0_4"))]
use serde::{Deserialize, Deserializer, Serialize};
#[cfg(all(feature = "serde_with", feature = "chrono-0_4"))]
use serde_with::{DeserializeAs, SerializeAs};
use time::format_description::well_known::Rfc3339;

#[cfg(feature = "chrono-0_4")]
use chrono::{LocalResult, TimeZone, Utc};
#[cfg(all(
feature = "serde_with",
any(feature = "chrono-0_4", feature = "time-0_3")
))]
use serde::{Deserialize, Deserializer, Serialize};
#[cfg(all(
feature = "serde_with",
any(feature = "chrono-0_4", feature = "time-0_3")
))]
use serde_with::{DeserializeAs, SerializeAs};

/// Struct representing a BSON datetime.
/// Note: BSON datetimes have millisecond precision.
Expand Down Expand Up @@ -114,11 +123,6 @@ impl crate::DateTime {
Self::from_system_time(SystemTime::now())
}

#[cfg(not(feature = "chrono-0_4"))]
pub(crate) fn from_chrono<T: chrono::TimeZone>(dt: chrono::DateTime<T>) -> Self {
Self::from_millis(dt.timestamp_millis())
}

/// Convert the given `chrono::DateTime` into a `bson::DateTime`, truncating it to millisecond
/// precision.
#[cfg(feature = "chrono-0_4")]
Expand All @@ -127,7 +131,23 @@ impl crate::DateTime {
Self::from_millis(dt.timestamp_millis())
}

fn to_chrono_private(self) -> chrono::DateTime<Utc> {
/// Convert this [`DateTime`] to a [`chrono::DateTime<Utc>`].
///
/// Note: Not every BSON datetime can be represented as a [`chrono::DateTime`]. For such dates,
/// [`chrono::MIN_DATETIME`] or [`chrono::MAX_DATETIME`] will be returned, whichever is closer.
///
/// ```
/// let bson_dt = bson::DateTime::now();
/// let chrono_dt = bson_dt.to_chrono();
/// assert_eq!(bson_dt.timestamp_millis(), chrono_dt.timestamp_millis());
///
/// let big = bson::DateTime::from_millis(i64::MAX);
/// let chrono_big = big.to_chrono();
/// assert_eq!(chrono_big, chrono::MAX_DATETIME)
/// ```
#[cfg(feature = "chrono-0_4")]
#[cfg_attr(docsrs, doc(cfg(feature = "chrono-0_4")))]
pub fn to_chrono(self) -> chrono::DateTime<Utc> {
match Utc.timestamp_millis_opt(self.0) {
LocalResult::Single(dt) => dt,
_ => {
Expand All @@ -140,30 +160,78 @@ impl crate::DateTime {
}
}

#[cfg(not(feature = "chrono-0_4"))]
fn from_time_private(dt: time::OffsetDateTime) -> Self {
let millis = dt.unix_timestamp_nanos() / 1_000_000;
match millis.try_into() {
Ok(ts) => Self::from_millis(ts),
_ => {
if millis > 0 {
Self::MAX
} else {
Self::MIN
}
}
}
}

#[cfg(not(feature = "time-0_3"))]
#[allow(unused)]
pub(crate) fn to_chrono(self) -> chrono::DateTime<Utc> {
self.to_chrono_private()
pub(crate) fn from_time(dt: time::OffsetDateTime) -> Self {
Self::from_time_private(dt)
}

/// Convert this [`DateTime`] to a [`chrono::DateTime<Utc>`].
/// Convert the given `time::OffsetDateTime` into a `bson::DateTime`, truncating it to
/// millisecond precision.
///
/// Note: Not every BSON datetime can be represented as a [`chrono::DateTime`]. For such dates,
/// [`chrono::MIN_DATETIME`] or [`chrono::MAX_DATETIME`] will be returned, whichever is closer.
/// If the provided time is too far in the future or too far in the past to be represented
/// by a BSON datetime, either [`DateTime::MAX`] or [`DateTime::MIN`] will be
/// returned, whichever is closer.
#[cfg(feature = "time-0_3")]
pub fn from_time(dt: time::OffsetDateTime) -> Self {
Self::from_time_private(dt)
}

fn to_time_private(self) -> time::OffsetDateTime {
match self.to_time_opt() {
Some(dt) => dt,
None => if self.0 < 0 {
time::PrimitiveDateTime::MIN
} else {
time::PrimitiveDateTime::MAX
}
.assume_utc(),
}
}

pub(crate) fn to_time_opt(self) -> Option<time::OffsetDateTime> {
time::OffsetDateTime::UNIX_EPOCH.checked_add(time::Duration::milliseconds(self.0))
}

#[cfg(not(feature = "time-0_3"))]
#[allow(unused)]
pub(crate) fn to_time(self) -> time::OffsetDateTime {
self.to_time_private()
}

/// Convert this [`DateTime`] to a [`time::OffsetDateTime`].
///
/// Note: Not every BSON datetime can be represented as a [`time::OffsetDateTime`]. For such
/// dates, [`time::PrimitiveDateTime::MIN`] or [`time::PrimitiveDateTime::MAX`] will be
/// returned, whichever is closer.
///
/// ```
/// let bson_dt = bson::DateTime::now();
/// let chrono_dt = bson_dt.to_chrono();
/// assert_eq!(bson_dt.timestamp_millis(), chrono_dt.timestamp_millis());
/// let time_dt = bson_dt.to_time();
/// assert_eq!(bson_dt.timestamp_millis() / 1000, time_dt.unix_timestamp());
///
/// let big = bson::DateTime::from_millis(i64::MAX);
/// let chrono_big = big.to_chrono();
/// assert_eq!(chrono_big, chrono::MAX_DATETIME)
/// let big = bson::DateTime::from_millis(i64::MIN);
/// let time_big = big.to_time();
/// assert_eq!(time_big, time::PrimitiveDateTime::MIN.assume_utc())
/// ```
#[cfg(feature = "chrono-0_4")]
#[cfg_attr(docsrs, doc(cfg(feature = "chrono-0_4")))]
pub fn to_chrono(self) -> chrono::DateTime<Utc> {
self.to_chrono_private()
#[cfg(feature = "time-0_3")]
#[cfg_attr(docsrs, doc(cfg(feature = "time-0_3")))]
pub fn to_time(self) -> time::OffsetDateTime {
self.to_time_private()
}

/// Convert the given [`std::time::SystemTime`] to a [`DateTime`].
Expand Down Expand Up @@ -210,26 +278,30 @@ impl crate::DateTime {

/// Convert this [`DateTime`] to an RFC 3339 formatted string.
pub fn to_rfc3339_string(self) -> String {
self.to_chrono()
.to_rfc3339_opts(chrono::SecondsFormat::AutoSi, true)
use std::result::Result;
match self.to_time().format(&Rfc3339) {
Result::Ok(s) => s,
Result::Err(e) => e.to_string(),
Copy link
Contributor Author

@abr-egn abr-egn Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not thrilled about handling the error this way but I couldn't think of a better option that wouldn't be a breaking change.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

given that the chrono equivalent never errored, any idea what can lead this method to fail / how likely we are to encounter this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the source, the time method fails if the year is outside [0,9999]; the chrono equivalent doesn't have that check.

The time method has some other potential failure cases as well since it's a more general method, but for the specific purpose of formatting an OffsetDateTime I don't think they can happen.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm, should we consider deprecating this method in favor of a new version that propagates the error correctly? seems like it might help cut down on the chances anyone hits this weirdness

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The time crate actually has a large-dates flag which expands this to +-99,999, which we may want to consider enabling here, as MongoDB technically supports any date that can be represented as an i64 from the Unix epoch. Should we enable that flag, we'll want to let the maintainer know of our use case here so that the feature flag stays around in the future.

Also, rather than erroring, it may be more appropriate to panic here. Could we test some inputs that error in time and see if they panic in chrono? If they do it might be okay for us to panic here as well. Another option would be to cap the date to time's maximum value and use that for the date here. I think we do that in a few places already with chrono's maximum date. If none of those work, then I agree we probably should deprecate this method and introduce a new one, since I think including error information as a String could be pretty cumbersome or dangerous.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that solution SGTM. since we use the flag already it probably seems worth chiming in on that discussion linked above to let them know why

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And as a last check, are we okay with the risk of breaking user code with this? It's technically possible that a user has a super large date in their DB somewhere that used to format okay but will panic now via this method or fail to serialize via the serde helper.

Copy link
Contributor Author

@abr-egn abr-egn Apr 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the risk is okay here - we're being loud about this change with the deprecation warning, and I think we should also call it out in the patch notes. It's also worth considering that the RFC itself is explicit about only applying to years in the 0000-9999 range and that the year field is fixed at four digits, so the chrono implementation is producing output that's probably not going to be accepted elsewhere.

That said, I'd be interested to hear if @kmahar or @isabelatkinson have other opinions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice, I didn't know that about the RFC, that definitely makes this change seem fine to me then.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree it seems fine 👍

}
}

/// Convert the given RFC 3339 formatted string to a [`DateTime`], truncating it to millisecond
/// precision.
pub fn parse_rfc3339_str(s: impl AsRef<str>) -> Result<Self> {
let date = chrono::DateTime::<chrono::FixedOffset>::parse_from_rfc3339(s.as_ref())
.map_err(|e| Error::InvalidTimestamp {
let odt = time::OffsetDateTime::parse(s.as_ref(), &Rfc3339).map_err(|e| {
Error::InvalidTimestamp {
message: e.to_string(),
})?;
Ok(Self::from_chrono(date))
}
})?;
Ok(Self::from_time(odt))
}
}

impl fmt::Debug for crate::DateTime {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut tup = f.debug_tuple("DateTime");
match Utc.timestamp_millis_opt(self.0) {
LocalResult::Single(ref dt) => tup.field(dt),
match self.to_time_opt() {
Some(dt) => tup.field(&dt),
_ => tup.field(&self.0),
};
tup.finish()
Expand All @@ -238,8 +310,8 @@ impl fmt::Debug for crate::DateTime {

impl Display for crate::DateTime {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match Utc.timestamp_millis_opt(self.0) {
LocalResult::Single(ref dt) => Display::fmt(dt, f),
match self.to_time_opt() {
Some(dt) => Display::fmt(&dt, f),
_ => Display::fmt(&self.0, f),
}
}
Expand Down Expand Up @@ -300,6 +372,49 @@ impl SerializeAs<chrono::DateTime<Utc>> for crate::DateTime {
}
}

#[cfg(feature = "time-0_3")]
#[cfg_attr(docsrs, doc(cfg(feature = "time-0_3")))]
impl From<crate::DateTime> for time::OffsetDateTime {
fn from(bson_dt: DateTime) -> Self {
bson_dt.to_time()
}
}

#[cfg(feature = "time-0_3")]
#[cfg_attr(docsrs, doc(cfg(feature = "time-0_3")))]
impl From<time::OffsetDateTime> for crate::DateTime {
fn from(x: time::OffsetDateTime) -> Self {
Self::from_time(x)
}
}

#[cfg(all(feature = "time-0_3", feature = "serde_with"))]
#[cfg_attr(docsrs, doc(cfg(all(feature = "time-0_3", feature = "serde_with"))))]
impl<'de> DeserializeAs<'de, time::OffsetDateTime> for crate::DateTime {
fn deserialize_as<D>(deserializer: D) -> std::result::Result<time::OffsetDateTime, D::Error>
where
D: Deserializer<'de>,
{
let dt = DateTime::deserialize(deserializer)?;
Ok(dt.to_time())
}
}

#[cfg(all(feature = "time-0_3", feature = "serde_with"))]
#[cfg_attr(docsrs, doc(cfg(all(feature = "time-0_3", feature = "chrono-0_4"))))]
impl SerializeAs<time::OffsetDateTime> for crate::DateTime {
fn serialize_as<S>(
source: &time::OffsetDateTime,
serializer: S,
) -> std::result::Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
let dt = DateTime::from_time(*source);
dt.serialize(serializer)
}
}

/// Errors that can occur during [`DateTime`] construction and generation.
#[derive(Clone, Debug)]
#[non_exhaustive]
Expand Down
18 changes: 7 additions & 11 deletions src/extjson/models.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
//! A module defining serde models for the extended JSON representations of the various BSON types.

use chrono::Utc;
use serde::{
de::{Error, Unexpected},
Deserialize,
Expand Down Expand Up @@ -256,16 +255,13 @@ impl DateTime {
Ok(crate::DateTime::from_millis(date))
}
DateTimeBody::Relaxed(date) => {
let datetime: chrono::DateTime<Utc> =
chrono::DateTime::parse_from_rfc3339(date.as_str())
.map_err(|_| {
extjson::de::Error::invalid_value(
Unexpected::Str(date.as_str()),
&"rfc3339 formatted utc datetime",
)
})?
.into();
Ok(crate::DateTime::from_chrono(datetime))
let datetime = crate::DateTime::parse_rfc3339_str(date.as_str()).map_err(|_| {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and elsewhere I changed it from the "parse into external datetime, convert to bson datetime" pattern to just directly using the methods bson::DateTime provides.

extjson::de::Error::invalid_value(
Unexpected::Str(date.as_str()),
&"rfc3339 formatted utc datetime",
)
})?;
Ok(datetime)
}
}
}
Expand Down
Loading