Is it possible to deserialize unknown keys?

[MMairinger]

I have currently an issue where I need to be able to deserialize JSON that has a defined structure but also allows for arbitary extensions. So I need to be able to work with unknown keys and unknown types. Now I would like to somehow store those unknown keys with their values and later, when needed serialize them into their original form again.

My first thought was to create some map property in which all the unknown keys and their values are put. When serializing that map I probably need to use a JsonTransformingSerializer since I want to keep the original structure and without that the map would be serialized as object.

The problem of how to store unknown keys and their values remain. I searched a lot but I could not find an answer to that. Does there exist a feature that deals with unknown keys or do I need to write a custom deserializer for that and what would an example of that look like dealing with unknown keys? Or do I need to approach this problem in a different way?

At the moment this is the approach I am experimenting with:
My test class:
@Serializable data class testSerializationVC (val knownKey: String, val elements: JsonObject)
My test json:
jsonToParse = """{"knownKey":"knownKey", "knownKey":"knownKey", "elements": {"arbitraryValue1":"json", "arbitraryValue2": 1, "arbitraryValue3": "arbitraryValue3"}}"""
Created Object:
testSerializationVC(knownKey=knownKey, elements={"arbitraryValue1":"json","arbitraryValue2":1,"arbitraryValue3":"arbitraryValue3"})
Output of serialization:
{"knownKey":"knownKey", "elements":{"arbitraryValue1":"json","arbitraryValue2":1,"arbitraryValue3":"arbitraryValue3"}}

My goal is to parse JSON like:
jsonToParse = """{"knownKey":"knownKey", "arbitraryValue1":"json", "arbitraryValue2": 1, "arbitraryValue3": "arbitraryValue3"}"""
And deserilze the object back into:
{"knownKey":"knownKey", "arbitraryValue1":"json", "arbitraryValue2": 1, "arbitraryValue3": "arbitraryValue3"}

[pdvrieze]

By far the simplest approach to do this would be to have a custom deserializer that does the following:

It first delegates to a serializer of a map of string,JsonElement values. For the known keys you then invoke the appropriate deserializers from the jsonElement and store them in some temporary. The unknown keys go into some map of unknown values. You create the instance of the result class and return it.

Note that it requires you delegate the descriptor to be that of a map (with the correct serialkind)

To serialize you do the same thing in reverse, you first serialize all the properties into a map (you serialize to an in-memory structure, not the encoder passed as function parameter). Then you add all the "unknown" values to the same map. Finally you serialize the map (against the encoder).

In short, you do nested serialization into/from an intermediate representation.

[MMairinger]

It first delegates to a serializer of a map of string,JsonElement values. For the known keys you then invoke the appropriate deserializers from the jsonElement and store them in some temporary. The unknown keys go into some map of unknown values. You create the instance of the result class and return it.

First of all thank you very much for your reply. I just have one question about that explanation. How I understood it was that I have 2 maps. One temporary for the known keys and another one for the unknown keys. Can't the known keys just be directly stored in the appropriate properties of the class I'm de-/serializing and only delegate the unknown keys to the string,JsonElement map serializer?

[pdvrieze]

First of all thank you very much for your reply. I just have one question about that explanation. How I understood it was that I have 2 maps. One temporary for the known keys and another one for the unknown keys. Can't the known keys just be directly stored in the appropriate properties of the class I'm de-/serializing and only delegate the unknown keys to the string,JsonElement map serializer?

The reason for the temporaries (which could be a map with all values) is that you normally serialize through the constructor, not by updating properties of an already constructed object. Of course you have a custom serializer so you can do whatever you want, including updating properties during deserialization.

[matax87]

Thanks to @pdvrieze suggestions I was able to accomplish the same challenge asked from @MMairinger.
Please note that I'm using version 0.20.0. You'll have to use JsonDecoder/JsonEncoder instead of JsonInput/JsonOutput in order to make it work on version 1.0.0.

Example:

@Serializable(with = FiltersConfigSerializer::class)
data class FiltersConfig(
    val name: String? = null,
    val age: Int? = null,
    val from: Date? = null,
    val unknownFilters: Map<String, JsonElement>? = null
)

@Serializable
data class Date(val timestamp: Long)

@Serializer(forClass = FiltersConfig::class)
internal object FiltersConfigSerializer : KSerializer<FiltersConfig> {

    private val stringToJsonElementSerializer = MapSerializer(String.serializer(), JsonElement.serializer())

    override val descriptor: SerialDescriptor = stringToJsonElementSerializer.descriptor

    override fun deserialize(decoder: Decoder): FiltersConfig {
        // Decoder -> JsonInput
        require(decoder is JsonInput) // this class can be decoded only by Json
        val json = decoder.json
        val filtersMap = decoder.decode(stringToJsonElementSerializer)

        val name = filtersMap["name"]?.let {
            json.fromJson(String.serializer(), it)
        }
        val age = filtersMap["age"]?.let {
            json.fromJson(Int.serializer(), it)
        }
        val from = filtersMap["from"]?.let {
            json.fromJson(Date.serializer(), it)
        }
        val knownKeys =
            setOf("name", "age", "from")
        val unknownFilters = filtersMap.filter { (key, _) -> ! knownKeys.contains(key) }

        return FiltersConfig(name, age, from, unknownFilters)
    }

    override fun serialize(encoder: Encoder, value: FiltersConfig) {
        // Encoder -> JsonOutput
        require(encoder is JsonOutput) // This class can be encoded only by Json
        val json = encoder.json
        val map: MutableMap<String, JsonElement> = mutableMapOf()

        value.name?.let { map["name"] = json.toJson(String.serializer(), it) }
        value.age?.let { map["age"] = json.toJson(Int.serializer(), it) }
        value.from?.let { map["from"] = json.toJson(Date.serializer(), it) }
        value.unknownFilters?.let { map.putAll(it) }

        encoder.encode(stringToJsonElementSerializer, map)
    }
}

[sandwwraith]

Regarding original question, it is also possible to write JsonTransformingSerializer to transform JSON from """{"knownKey":"knownKey", "arbitraryValue1":"json", "arbitraryValue2": 1, "arbitraryValue3": "arbitraryValue3"}""" to """{"knownKey":"knownKey", "knownKey":"knownKey", "elements": {"arbitraryValue1":"json", "arbitraryValue2": 1, "arbitraryValue3": "arbitraryValue3"}}""" and then use the original serializer.

[MMairinger]

It's been a while and would like to quickly elaborate what I did to serialize arbitary values.

I resorted to the use of 2 classes. One that is serializable and consists of the known values and of kotlinx types like JsonObject, JsonArray and so on for properties that allow arbitary keys in it. The second class would consists of more precise standard types and your own types. For example a property that was in the first class a JsonObject would be now in the second class a Map<String, Any?> which is not serializable by default but it doesn't have to since I don't use this class for serialization/deserialization.

I then used methods that convert the first class into an object of the second class. For that I methods that would convert JsonArrays, JsonObjects and JsonPrimitive to Lists<...>, Map<...> and for JsonPrimitives I got their content type with JsonPrimitive.booleanOrNull and so on.

When serializing my second class I converted it back to the first kind of class which is de-/serializable and for that I used the buildJsonObject, buildJsonArray methods and so on.

I hope this will help others in future who try to encomplish a similar task as I had to. For me personally this issue is solved.