provide a generic serializer

[bobzhang]

cc @ryyppy @jfrolich
There is a bug in v8, https://bugs.chromium.org/p/v8/issues/detail?id=10595
so this may not be that useful (it is not uncommon to have deeply nested objects, take list for example)
Since JSON does not have undefined, we encode it a s a magic value, and when de-serializing, we convert it back to undefined.

[ryyppy]

why does serializeExn return an string option?

[bobzhang]

JSON.stringify(Symbol()) returns undefined, maybe we just raise in this case to simplify things

[ryyppy]

Just to describe this PR for my own understanding, this is how you could e.g. use the serialization functions for NextJS limitations where I have option values for nullability reasons:

type user = {
  name: string,
  age: option(int), 
};

type props = { userStr: string };

let default = (props: props) => {
  // user = `{ name: "Patrick", age: None }`
  let user = Js.Json.deserializeExn(props.userStr);

 // We can now use the age like an option value
 let ageEl = Belt.Option.getWithDefault(React.null, (age) => age->Belt.Int.toString->React.string);
 <div>
  {user.name->React.string}  {ageEl}
 </div>
};

let getStaticProps = (_ctx) => {
  let mockUser = { name: "Patrick", age: None};

  let props = {
    // userStr = `{ "name": "Patrick" }`
    userStr: Js.Json.serializeExn(mockUser)
  };

  {
    "props": props
  }
};

So these functions would essentially be a (better?) integrated part of the platform to serialize any data to JSON compliant structures?

[bobzhang]

I would say for prototype or not highly invested project (personal projects).
To serialize data in a high quality versioned style, you need a schema like protobuf.

deserialize is not type safe, since its return type can be any depending on user annotations, what's the better name, unsafeDeserializeExn?

[ryyppy]

hmm, i am not entirely sure if this is the right way yet.

E.g. I don't see a direct benefit of using a magic value to represent the case of a non-existing option value, since it actually does also work to just strip away the value entirely (as I proposed in my latest blog post).

I think this topic needs further proper discussion with detailed edge-cases / a user-land prototype before there should be another function in Js.Json?

[bobzhang]

Do you have docs about how next.js makes use of the JSON object? Like what’s the semantics of the JSON object? Does it need to be deserialized back Patrick Stapfer <[email protected]>于2020年6月5日 周五下午8:36写道：

hmm, i am not entirely sure if this is the right way yet. E.g. I don't see a direct benefit of using a magic value to represent the case of a non-existing option value, since it actually does also work to just strip away the value entirely (as I proposed in my latest blog post <https://dev.to/ryyppy/reason-records-nextjs-undefined-and-getstaticprops-5d46> ). I think this topic needs further proper discussion with detailed edge-cases / a user-land prototype before there should be another function in Js.Json? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4443 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFWMK47S3VMLNRYUMMLGKTRVDRGXANCNFSM4NTI4DLQ> .

-- Regards -- Hongbo Zhang

[bloodyowl]

since it actually does also work to just strip away the value entirely

@ryyppy That works for records, but for arrays we still need a "magic" value to encode None

JSON.stringify({foo: undefined, bar: 1})
// "{\"bar\":1}"
JSON.stringify([undefined, 1, undefined])
// "[null, 1, null]"

[bloodyowl]

Is there a reason why you don't use the second JSON.parse argument like the following?

JSON.parse(s, function(_key, value) {
  if(value && value.RE_PRIVATE_NONE === true) {
    return undefined
  };
  return value
});

[bobzhang]

Unfortunately this does not work, you can try it

[bloodyowl]

Tried on Chrome and Safari and it does seem to work? The only difference I see is that it makes the arrays sparse, but that doesn't seem to be an issue given how BS's runtime interacts with them?

[bloodyowl]

Oh yeah, that'd break Js.Array.*. Nevermind 😄

[bloodyowl]

Something like this would work, I'm not sure if that'd optimise better in browser VMs

let pairs = [];
let parsed = JSON.parse(s, function(key, value) {
  if(value && value.RE_PRIVATE_NONE === true) {
    pairs.push([this, key]);
    return undefined
  };
  return value
});
pairs.forEach(([obj, key]) => obj[key] = undefined)
parsed;

[bobzhang]

it does not work for me, do you have a working example? is this semantics documented somewhere?

Even if it works, it seems the IC is still changed

[bobzhang]

let p = s => {var parents = [];
var obj = JSON.parse(s, function(key, value) {
  if(value && value.RE_PRIVATE_NONE === true) {
    parents.push({par:this, key});
  };
  return value
}); for(let {par,key} of parents){par[key]=undefined}; return obj }

This looks good to me

[bloodyowl]

Documented here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse#Using_the_reviver_parameter

I made a rough benchmark in Chrome and Safari and it seems that your original solution is consistently faster, so the only benefit left would be runtime size. About IC it seems like returning undefined from the reviver function will necessarily call some delete, but I'm wondering if using null as a placeholder until replacing "by hand" with undefined would help.

[bobzhang]

It maybe that JSON.parse without reviver is specialised.
About the bundle size, my implementation is quite small, if it is indeed faster, we can keep it.
It is good to know an alternative solution!

[bobzhang]

I am thinking we can do the same thing with stringify without using callback, it may be faster

[bobzhang]

@bloodyowl This looks interesting, is `this` semantics in the callback API documented somewhere? I can not find any docs about this

…

On Sat, Jun 6, 2020 at 8:18 PM Matthias Le Brun ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In jscomp/others/js_json.ml <#4443 (comment)> : > +let deserializeExn (s: string) : 'a = + patch (parseExn s) Something like this would work, I'm not sure if that'd optimise better in browser VMs let pairs = [];let parsed = JSON.parse(s, function(key, value) { if(value && value.RE_PRIVATE_NONE === true) { pairs.push([this, key]); return undefined }; return value});pairs.forEach(([obj, key]) => obj[key] = undefined)parsed; — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4443 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFWMK5ZZHW6CYENVYCNA3DRVIXY7ANCNFSM4NTI4DLQ> .

-- Regards -- Hongbo Zhang

[bobzhang]

ping @ryyppy , what's the semantics of next.js toJSON protocol? Is there any docs about it, I expect next.js would require more than serialized to JSON? does it need be deserialized back?

[ryyppy]

@bobzhang NextJS currently has two APIs for hydrating React components via props

• getStaticProps
• getServerSideProps

The docs are vague on what the shape of props actually is, but it turned out to be plain JSON... but the React component doesn't show that there is any JSON serialization going on.

Here the docs example in JS:

function Blog({ posts }) {
  // Render posts (where `posts` is already an array of posts)....
}

// This function gets called at build time
export async function getStaticProps() {
  // Call an external API endpoint to get posts
  const res = await fetch('https://.../posts')
  const posts = await res.json()

  // By returning { props: posts }, the Blog component
  // will receive `posts` as a prop at build time
  return {
    props: {
      posts,
    },
  }
}

export default Blog

In this example, you don't see any serialization going on, since Next assumes that you are passing plain JS objects without any functions / this context etc. It's doing the transformation for you already in the background, so you can just use the data as is in the component... that's why in development mode, Next will warn you if you are returning data that is not JSON compliant.

Regarding the protocol: Next is literally using JSON.stringify for serialization and JSON.parse for reconstructing the value

[bobzhang]

@ryyppy thanks for the point. What will you happen if you pass an arbitrary json object to next.js.
Take your input for example:
instead of

 {
  diet_kind: undefined
};

{ diet_kind : {RE_PRIVATE_NONE:true}}

This is a valid json object, how will next.js interpret it?

[ryyppy]

This would be serialized correctly by Next but we'd need to type the props as some JS.Json.t object and convert it in the component body to the actual value

[bobzhang]

I mean what would be the rendered output by Next.js? Patrick Stapfer <[email protected]>于2020年6月10日 周三下午1:48写道：

This would be serialized correctly by Next but we'd need to type the props as some JS.Json.t object and convert it in the component body to the actual value — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4443 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFWMK646Q7L3FKGHD4IKIDRV4NCNANCNFSM4NTI4DLQ> .

-- Regards -- Hongbo Zhang

[ryyppy]

in the html output: "{ \"diet_kind\" : {\"RE_PRIVATE_NONE\":true}}"

when mounted in JS: { diet_kind : {RE_PRIVATE_NONE:true}}

[bobzhang]

Yeah, I mean is this your desired output? I ask this since I am unclear about the context what next.js is trying to do Patrick Stapfer <[email protected]>于2020年6月10日 周三下午1:57写道：

in the html output: ˋ"{ "diet_kind" : {"RE_PRIVATE_NONE":true}}"ˋ when mounted in JS: ˋ{ diet_kind : {RE_PRIVATE_NONE:true}}ˋ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4443 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFWMK63MNSUOVUNQ4LJMKTRV4OFJANCNFSM4NTI4DLQ> .

-- Regards -- Hongbo Zhang

[ryyppy]

Yes, the example above is the desired output. The reason why Next does this is bc it pre renders the react component data as a JSON string in the static html output and as soon as the user loaded the website it will parse the string data and pass it to the component

[bobzhang]

@ryyppy so our solution will work for you. Instead of a serializing/deserization to string, you need a RTT function to plain json object

[bloodyowl]

@ryyppy Maybe we can open an issue in next.js and ask them if they'd be okay to let users provide a custom (de)serialiser in next.config.js

[ryyppy]

@bobzhang yep
@bloodyowl do you think a config would be useful? i feel like having props typed as json and then doing this one single conversion ourselves is less confusing (at least until we figured out how tedious this actually is)

[bloodyowl]

I think that in terms of API, it'd be simpler if we were to tell people:

Here are two lines to copy once in your next.config.js

serializer: require("bs-platform/.../json").serializeExn
deserializer: require("bs-platform/.../json").deserializeExn

vs telling them to call the function in both places for each component.

[bobzhang]

updated the interface of val serializeExn : 'a -> string

@ryyppy I planned to add a pair of functions for 'a -> Json.t, but you can do this on user side, using JSON.parse. It seems as long as we preserve s -> serialize -> deserialize = s, it will be useful in your cases.

For the record, the patching on serialisation is not done since you don't want to modify the existing

[bobzhang]

depends on user demand, we may add support for Date, bigint in the future

[ryyppy]

thanks @bobzhang .. will upgrade reasonml.org to v8 to use this when the release is out.

@bloodyowl yeah, might be easier for some ppl... just wanted to have a chance to testflight this feature set before asking for new features in Next 😄

[bobzhang]

this is the nodejs convention

const buf = Buffer.from([0x1, 0x2, 0x3, 0x4, 0x5]);
const json = JSON.stringify(buf);

console.log(json);
// Prints: {"type":"Buffer","data":[1,2,3,4,5]}

Maybe we should follow the same convention?