Serializing Optional not enabled by default since 2.16.0

[cor3000]

Search before asking

• I searched in the issues and found nothing similar.

Describe the bug

in version 2.15.4 the following code below succeeded

giving the output

{"test":{"empty":false,"present":true}}
{"test":{"empty":false,"present":true}}

(Obviously this value doesn't make sense, but it doesn't fail with a hard exception, breaking existing projects)

when using version a higher version e.g. 2.17.0 the following Exception occurs instead (in both cases the jackson-datatype-jdk8.jar of the same version is on the classpath)

Exception in thread "main" com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Java 8 optional type `java.util.Optional` not supported by default: add Module "com.fasterxml.jackson.datatype:jackson-datatype-jdk8" to enable handling (through reference chain: java.util.ImmutableCollections$Map1["test"])
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
	at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1330)
	at com.fasterxml.jackson.databind.ser.impl.UnsupportedTypeSerializer.serialize(UnsupportedTypeSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:502)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:341)
	at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4799)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:4040)
	at Jackson2_17_Test_ObjectMapper2.main(Jackson2_17_Test_ObjectMapper2.java:24)

I didn't find this default behavior change in the release notes, so I wonder whether this is an intended change.
As a workaround I can register the Jdk8Module manually (commented out in the code above), to produce the same results as in 2.15.4

Version Information

2.17.0

Reproduction

Run the example using the maven dependencies

• com.fasterxml.jackson.core:jackson-databind:2.15.4
• com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.15.4

after switching to 2.16.0, 2.16.1, 2.17.0 the same code below fails

• com.fasterxml.jackson.core:jackson-databind:2.17.0
• com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.17.0

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.json.JsonMapper;
import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;

import java.util.Map;
import java.util.Optional;

public class Jackson2_17_Test_Optional {

    public static void main(String[] args) throws JsonProcessingException {

        Map<String, Optional<String>> object = Map.of(
                "test", Optional.of("optional value"));

        JsonMapper jsonMapper = JsonMapper.builder()
//                .addModule(new Jdk8Module()) // workaround, register jdk8 module manually
                .build();
        String value = jsonMapper.writeValueAsString(object);
        System.out.println(value);

        ObjectMapper objectMapper = new ObjectMapper();
//        objectMapper.registerModule(new Jdk8Module()); // workaround, register jdk8 module manually
        String value2 = objectMapper.writeValueAsString(object);
        System.out.println(value2);
    }
}

Expected behavior

giving the output as in 2.15.4, not throwing an exception

{"test":{"empty":false,"present":true}}
{"test":{"empty":false,"present":true}}

Additional context

even though we have a manual workaround it's difficult to track and adjust all usages by external libraries.

my apologies in case I missed the related documentation and this is the new expected behavior since 2.16.0

[cor3000]

I found this was changed by #4082
so I guess that's the intended behavior now... if that's the case, please confirm and close the issue

[cowtowncoder]

@cor3000 Correct: this is the new intended behavior, and exception indicates the root cause. It is unfortunate issue referred does not fully explain logic (or have good title) to make it easier to find the change.

But the goal was to prevent accidental serialization of Optional using structure that was never meant to be used by databind (but happens to "work" due to existence of boolean isEmpty() method).

So registering Java 8 types module is the fix, not workaround. Or, alternatively, registering custom serializer.

[schiemon]

Thank you, @cowtowncoder, for the clarification! Is there a way to avoid having to register the Jdk8Module for every ObjectMapper instance and instead register it globally?

[cowtowncoder]

@schiemon if you mean "set some static thing to force all ObjectMapper to add JdkModule" then no.

[JooHyukKim]

Yeah, globally (statically) setting such thing is not desirable and won't be in the future either.
Try doing such thing at application-level like factory method for generating mapper with such?

[schiemon]

@cowtowncoder yes, this exactly what I had in mind.
@JooHyukKim This would be possible but imagine a giant codebase prior to this decision which has not wrapped those ObjectMappers. Now, for the patch upgrade, in such codebase one has to touch every occurance of it which I aim to avoid.

[cowtowncoder]

@schiemon I can see how that can be problematic. Unfortunately Jackson is so widely used by various frameworks and libraries that any global configuration (at Jackson level, not by application) is potential problem -- meaning that if global configuration existed, it would effect things users wouldn't plan to. So the choice of having no (*) globally configurable entries is conscious.
And in case of modules I am pretty sure we'll never want to expose global set of modules to add. But users can definitely add global singletons for producing either ObjectMappers, or their Builders, that do do that. Takes planning of course.

(*) there are couple of things related to security (max name length) for which there are global static settings one can define as baseline -- but all overridable.

[JooHyukKim]

@JooHyukKim This would be possible but imagine a giant codebase prior to this decision which has not wrapped those ObjectMappers. Now, for the patch upgrade, in such codebase one has to touch every occurance of it which I aim to avoid.

Yeah, sorry to hear that. I don't know how gigantic you're talking about, but I have experience working with such systems as well. Somethings are better off modularized or isolated. But! Let's consider this as techincal debt we are willing to pay off :-) willing because it could be a sign that system getting bigger means ur business grew as well.

Hopefully but there may be programming patterns/approaches to find!
Cheers
@schiemon

[schiemon]

Thank you again for your detailed answers, @JooHyukKim and @cowtowncoder. I now have a much better understanding of the problem from your perspective as library maintainers.

I do have one remaining question, though. Since it is closely related to the original question, I will ask it here. However, feel free to let me know if I should open a separate issue for this.

Let’s say I register the Jdk8Module for all ObjectMappers in this ominous giant codebase. Prior to this registration, entire object trees were effectively "hidden" under an Optional property. These trees were not serialized but were instead represented as {"empty":false,"present":true}. After the upgrade and registration, these sub-trees are now being serialized, which has significant implications for performance and correctness.

Is there a way in the newer version to revert to the previous behavior to ensure that mappers are producing the same output? For instance, could something be configured at the mapper level to achieve this?

This is getting long but I think the following is also important to mention in this context. I noticed a difference in behaviour between older versions when handling Optionals with a mapper where mapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.NONE) is set. With the small setup here I recognized following behaviour:

2.12.7
{"parentGetter":{}}

2.13.0
{"parentGetter":{"empty":false,"present":true}}

When I am removing the visibility setting, both output the same string ({"parentGetter":{"empty":false,"present":true}}). I would be very grateful if you could explain where this behaviour comes from so that I can consider it when trying to adapt the mappers!

[cowtowncoder]

@schiemon I do not know why/how settings as applied in 2.12.x (which seem correct) would work differently in 2.13.x. These are relatively old versions by now so nothing much could be done to change them.

But one possibility is that something in handling of "is-getter" ("isPresent()" etc) vs "regular getter" ("getValue()" etc) was not properly handled any longer; that is, visibility setting was not applied to "is-setters" and defaults were.
I would recommend trying with later minor versions since if there was such a visibility bug, it would likely have been found and fixed. I do not remember specific issue off-hand, however. Will see if I can spot anything in release notes.

Finally, while there is no centralized way to revert serialization of Optional<T> values back to no-module-registered case, it is still possible to just register custom JsonSerializer that can output old style output (whether it should be { } or Object with "empty" and "present").
That might be the best way, although requires configuration of ObjectMappers too.

[cowtowncoder]

Also tests for attempts to serialize Optional, without module registered, will will result in exception is tested via:

src/test/java/com/fasterxml/jackson/databind/interop/OptionalJava8Fallbacks4082Test.java

[seadbrane]

Following up on this as we have a quite a bit of code that is dependent on the old behavior of serializing { "value" : ..., "present" : true } and also expecting to be able to deserialize back to pojos and collections. This looks to be fairly large non-backward incompatible change with no easy way to get back to the old behavior, since previously this generally just worked.

To clarify, I am running into 2 major issues trying to create custom serializers/deserializers to get back the prior behavior.

1. Creating a custom JsonSerializer is not an issue; however, the only custom JsonDeserializer implementation I have that is even close to working is very complex and still not covering all cases.
2. Finding all the ObjectMapper's this now needs to be added to is non-trivial.

Let me know if it is better to move this to a new issue?

[cowtowncoder]

@seadbrane Yes, a new issue outlining desired changes to help work around the problem would make sense.

FWTW I did not think it was ever possible to automatically deserialize Optionals back (that is, with default ObjectMapper) and was thinking this was just about serialization.

[seadbrane]

Correct - not with default ObjectMapper, but it is possible in 2.12 with only the following configuration on the ObjectMapper, and it supports full deserialization of complex types and collections.

  ObjectMapper mapper = new ObjectMapper();
  mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
  mapper.setVisibility(mapper.getSerializationConfig().getDefaultVisibilityChecker()
            .withFieldVisibility(JsonAutoDetect.Visibility.ANY));

Sample code.

import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Optional;

import org.junit.Assert;
import org.junit.jupiter.api.Test;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationFeature;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.annotation.JsonInclude;

import com.fasterxml.jackson.annotation.JsonAutoDetect;


public class OptionalTest {

    public static class Sequence {
        private int start;
        private int end;
        private Optional<Boolean> enabled;
        @Override
        public int hashCode() {
            return Objects.hash(enabled, end, start);
        }
        @Override
        public boolean equals(Object obj) {
            if (this == obj) {
                return true;
            }
            if (obj == null) {
                return false;
            }
            if (getClass() != obj.getClass()) {
                return false;
            }
            Sequence other = (Sequence) obj;
            return Objects.equals(enabled, other.enabled) && end == other.end && start == other.start;
        }
    }

    public static class Pojo {
        private Optional<String> stringVal;
        private Optional<Sequence> sequence;
        private Optional<List<Sequence>> sequenceList;
        @Override
        public int hashCode() {
            return Objects.hash(sequence, sequenceList, stringVal);
        }
        @Override
        public boolean equals(Object obj) {
            if (this == obj) {
                return true;
            }
            if (obj == null) {
                return false;
            }
            if (getClass() != obj.getClass()) {
                return false;
            }
            Pojo other = (Pojo) obj;
            return Objects.equals(sequence, other.sequence) && Objects.equals(sequenceList, other.sequenceList)
                    && Objects.equals(stringVal, other.stringVal);
        }
    }

    private Pojo create() {
        Pojo p = new Pojo();
        p.stringVal = Optional.of("testValue");
        p.sequence = Optional.of(sequence(1, 10));
        List<Sequence> l = new ArrayList<>();
        l.add(sequence(10, 100));
        l.add(sequence(20, 200));
        l.add(sequence(30, 300));
        p.sequenceList = Optional.of(l);
        return p;
    }

    private Sequence sequence(int start, int end) {
        Sequence s = new Sequence();
        s.start = start;
        s.end = end;
        s.enabled = Optional.of(false);
        return s;
    }


    @Test
    public void roundTrip() throws JsonProcessingException {
        ObjectMapper mapper = new ObjectMapper();
        mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
        mapper.setVisibility(mapper.getSerializationConfig().getDefaultVisibilityChecker()
            .withFieldVisibility(JsonAutoDetect.Visibility.ANY));

        Pojo p = create();
        String value = mapper.writeValueAsString(p);
        Assert.assertEquals("{\"stringVal\":{\"value\":\"testValue\",\"present\":true},"
                + "\"sequence\":{\"value\":{\"start\":1,\"end\":10,\"enabled\":{\"value\":false,\"present\":true}},"
                + "\"present\":true},\"sequenceList\":{\"value\":["
                + "{\"start\":10,\"end\":100,\"enabled\":{\"value\":false,\"present\":true}},"
                + "{\"start\":20,\"end\":200,\"enabled\":{\"value\":false,\"present\":true}},"
                + "{\"start\":30,\"end\":300,\"enabled\":{\"value\":false,\"present\":true}}],"
                + "\"present\":true}}", value);
        Pojo r = mapper.readValue(value, Pojo.class);
        Assert.assertEquals(p, r);
    }
}

[cowtowncoder]

Ok so with configuration. One possibility here would be to add a new MapperFeature to prevent blocking of Optional (de)serialization, to let default as-POJO handling work.

[seadbrane]

What about adding it as an option in Jdk8Module including the ability to serialize to the existing { "value": ..., "present" : true }?

This is what the error suggests you should do - even though it will not result in the same serialized output today, and this would also help minimize/prevent potential conflicts between the Optional serializers/deserlizers that are already in this module.

[cowtowncoder]

@seadbrane (I assume you meant "deserialize from").
Hmmmh. I'd rather not. In my opinion this is a fallback for legacy use and not to an actual feature recommended.

[seadbrane]

I meant serialize - as in support both serialization and deserliazation of the legacy format - not by default, but as an "feature" that would preserve the legacy behavior.
Suggesting this path for 2 reasons. First, given the error message that you get after upgrading to 2.16+, folks infer that adding JDK8 module is all they need to get back to the prior behavior. Second, if a custom module is built it has to be registered after the Jdk8Module. Adding an option to this module seems like it would be a good way to communicate these nuances, while also being consistent with the recommended solution.

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Java 8 optional type `java.util.Optional<java.lang.String>` not 
supported by default: add Module "com.fasterxml.jackson.datatype:jackson-datatype-jdk8" to enable handling (through 
reference chain: 

[cowtowncoder]

The earlier behavior was just a side-effect of POJO handling, not intentional support.
And the default serialization without visibility changes really only produced something like:

{ "present": false }

on serialization, failing to deserialize.

I am open to allowing handling without module registration again, but not to adding specific alternative representations by Java 8 datatypes module.

To that end a new MapperFeature that merely prevents 2.16 change (via (ISSUE) ) would be ok.

Removing (ISSUE) is not something I'd like to do because serialization as per vanilla ObjectMapper settings was and is still useless.

[seadbrane]

To clarify, are you saying that the new MapperFeature would handle deserialization of data in the legacy format, but if it still needed to be written in that format - then folks would need to create a custom serializer?

[cowtowncoder]

No. What I am suggesting as minimal thing is adding a MapperFeature to prevent failure due to missing module, and proceeding with pre-2.16 behavior, using Bean[De]Serializer.
Which can serialize Optional always, and apparently deserialize with visibility configurations you outlined.

Use of custom (de)serializers works regardless of any new settings: 2.16 only added exception if there is no (de)serializer provided by modules or user registering ones.

[seadbrane]

Ok, thanks. That sounds like it will work.

[cowtowncoder]

Will need to file an issue, and find time to implement (not a big thing but... there's a LOT of work to be done besides :) ).

[cowtowncoder]

Ok: new issue is #5006 and I am merging fix for it via #5023.

So you can prevent failure by disabling MapperFeature.REQUIRE_HANDLERS_FOR_JAVA8_OPTIONALS to be added in 2.19.0 (API change can't go in patch)

[seadbrane]

Thank you!

[cowtowncoder]

You are welcome, @seadbrane !