KEP-4222: CBOR encoding alpha requirement tracking

[benluddy]

What would you like to be added?

Requirements for https://kep.k8s.io/4222:

As we check these off, please add links to the tests.

• Test: Decoding a map containing duplicate keys into a Go map produces an error: KEP-4222: Add duplicate key and field case-sensitivity CBOR decode tests. #123436
• Test: Decoding a map containing duplicate keys into a Go struct produces an error: KEP-4222: Add duplicate key and field case-sensitivity CBOR decode tests. #123436
• Test: Decoding a map into a Go struct matches json field tag names case-sensitively: KEP-4222: Add duplicate key and field case-sensitivity CBOR decode tests. #123436
• Test: When decoding a map into a Go struct, a case-insensitive match between a key and a json field tag name is treated the same as no match: KEP-4222: Add duplicate key and field case-sensitivity CBOR decode tests. #123436
• Test: Encoding a struct with duplicate field names (json tag names) does not result in a map containing duplicate keys: KEP-4222: Add tests for CBOR encoder handling of duplicate field names/tags. #123626 (discussed at March 6 2024 api-machinery biweekly and agreed on matching the encoding/json behavior, with addition of parallel test cases for the json serializer)
• Test: Roundtripping preserves the distinction between integers and floating-point numbers
  • subtests for (u)int{64,32,16,8} and float{64,32} in TestRoundtrip of KEP-4222: Add decode and roundtrip tests for CBOR marshaling. #123267
  • Still need tests for the distinction being maintained when decoding into map[string]interface{} KEP-4222: Make CBOR roundtrip tests pass through interface{}. #123536
  • missing case covering uint64(math.MaxUint64) because it would overflow int64. Kube JSON cannot roundtrip this, either: uint64(math.MaxUint64) -> JSON 18446744073709551615 -> float64(1.8446744073709552e+19) -> JSON 18446744073709552000 -> json: cannot unmarshal number 18446744073709552000 into Go value of type uint64: this was discussed in the apimachinery biweekly on march 6 and agreed to reject rather than decode to float64: https://docs.google.com/document/d/1x9RNaaysyO0gXHIr1y50QFbiL1x8OWnk2v3XnrdkT5Y/edit#heading=h.xzja5vbnkhk2
• Test: Decoding a text string containing an invalid UTF-8 sequence produces an error
  • subtest in TestDecode of KEP-4222: Add decode and roundtrip tests for CBOR marshaling. #123267
• Test: Pooled buffers should not grow and be retained forever: KEP-4222: Don't pool large CBOR encode buffers #125676
• Test: Decoding into a Go interface{} stores only either nil concrete values or concrete values of type bool, string, int64, float64, []interface{}, or map[string]interface{} (no special treatment of tagged content producing time.Time, math/big.Int, etc.): KEP-4222: Add unit tests for decoding CBOR into interface{} type #125068
  • fixme: "decoding cbor data tagged with 0 produces time.Time instead of RFC3339 timestamp string": KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
  • fixme: "decoding cbor data tagged with 1 produces time.Time instead of RFC3339 timestamp string"
    • tagged unsigned integer: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged float16: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged float32: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged float64: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged unsigned integer, year beyond 9999: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged negative integer: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged negative float16: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged negative float32: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged negative float64: KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
    • tagged Inf: KEP-4222: Reject NaN or infinite floating-point values in the CBOR serializer. #125423
    • tagged -Inf: KEP-4222: Reject NaN or infinite floating-point values in the CBOR serializer. #125423
    • tagged NaN: KEP-4222: Reject NaN or infinite floating-point values in the CBOR serializer. #125423
  • fixme: "decoding cbor data tagged with 2 produces big.Int instead of []byte": KEP-4222: Reject math/big.Int on encode and bignum tags on decode for CBOR. #125420
  • fixme: "decoding cbor data tagged with 3 produces big.Int instead of []byte": KEP-4222: Reject math/big.Int on encode and bignum tags on decode for CBOR. #125420
  • fixme: "cbor data with simple value 23 should throw an error": KEP-4222: Reject CBOR simple values other than true, false, and null. #125421
  • fixme: "unrecognized simple values should be rejected": KEP-4222: Reject CBOR simple values other than true, false, and null. #125421
• Test: Conformance to CBOR specification
  • All CBOR examples in RFC 8949 Appendix A roundtripped in KEP-4222: Add roundtrip tests for all CBOR examples in RFC 8949 Appendix A. #123268
  • fixme: "decoding bigint to interface{} must not produce math/big.Int": KEP-4222: Reject math/big.Int on encode and bignum tags on decode for CBOR. #125420
  • fixme: "decoding negative bigint to interface{} must not produce math/big.Int": KEP-4222: Reject math/big.Int on encode and bignum tags on decode for CBOR. #125420
  • fixme: "the undefined simple value should not successfully decode as nil": KEP-4222: Reject CBOR simple values other than true, false, and null. #125421
  • fixme: "simple values other than false, true, and null should be rejected": KEP-4222: Reject CBOR simple values other than true, false, and null. #125421
  • fixme: "simple values other than false, true, and null should be rejected": KEP-4222: Reject CBOR simple values other than true, false, and null. #125421
  • fixme: "decoding of tagged time into interface{} must produce RFC3339 timestamp compatible with JSON, not time.Time": KEP-4222: Decode CBOR time tags to interface{} as RFC 3339 timestamps. #125424
• Test: Conformance tests demonstrated to run against a non-Go Kubernetes client language: KEP-4222: Add link to Python spec conformance demo. enhancements#4923
• Test: Go strings that are not valid UTF-8 sequences can be roundtripped through CBOR without error
  • subtest in TestRoundtrip of KEP-4222: Add decode and roundtrip tests for CBOR marshaling. #123267
• Test: Decoding a map into a Go struct produces a strict decoding error if the map contains a key that does not correspond to JSON tag name of one of the struct's fields
  • Subtest of TestDecode in KEP-4222: Add CBOR Serializer implementation. #122881
• Test: Roundtripping preserves the distinction between absent, present-but-null, and present-and-empty for slices and maps
  • subtest in TestRoundtrip of KEP-4222: Add decode and roundtrip tests for CBOR marshaling. #123267
  • Still need tests for the distinction being maintained when decoding into map[string]interface{}: KEP-4222: Make CBOR roundtrip tests pass through interface{}. #123536
• Test: runtime.RawExtension: Re-encoding preserves the original raw bytes: KEP-4222: Automatically transcode RawExtension between unstructured protocols. #125629
• Test: runtime.RawExtension: Decoding JSON-in-CBOR, JSON-in-Protobuf, CBOR-in-JSON, and CBOR-in-Protobuf is supported: KEP-4222: Automatically transcode RawExtension between unstructured protocols. #125629
• Test: Fuzz: For all native types, native-to-JSON-to-unstructured and native-to-CBOR-to-unstructured is identical: KEP-4222: Add roundtrip tests to Unstructured via CBOR and JSON. #122832
  • cover aggregator types: KEP-4222: Cover aggregator and apiextension types in unstructured roundtrip test. #125570
  • cover apiextensions types: KEP-4222: Cover aggregator and apiextension types in unstructured roundtrip test. #125570
  • todo: enable for types using RawExtension: KEP-4222: Automatically transcode RawExtension between unstructured protocols. #125629
  • todo: enable for types using []byte: KEP-4222: Enable JSON-compatible base64 encoding of []byte for CBOR. #125419
• Test: Fuzz: Decoding does not allocate an inordinate number of bytes: KEP-4222: Add CBOR fuzz test for unreasonable allocations during decode. #124761
• Test: Fuzz: Roundtrip JSON-to-CBOR-to-JSON and CBOR-to-JSON-to-CBOR: KEP-4222: Add fuzz test for roundtrip unstructured objects to JSON/CBOR #127862
• Test: Fuzz: Roundtrip through implementations in at least some of the non-Go client languages: KEP-4222: Add link to Python spec conformance demo. enhancements#4923
• Test: Integration: Response content-type negotiation works and honors indicated preference (Protobuf > CBOR > JSON): KEP-4222: Test response content negotiation for each CBOR enablement state. #128267
• Test: Integration: Get, list, watch, update, delete, deletecollection, apply, and scale support CBOR using dynamic and generated clients for all native types
  • Streaming support in Serializer implementation: KEP-4222: Implement runtime.Framer for CBOR Sequences. #126105
  • Tests using dynamic client for all resources: KEP-4222: Add CBOR variant of admission webhook integration test. #128243
  • Tests using generated client: KEP-4222: Support CBOR for generated clients based on client-go feature gating. #128323
  • Patch handler accepts CBOR apply: KEP-4222: Support CBOR encoding for apply requests. #128273
  • Apply integration test coverage (dynamic + generated): KEP-4222: Support CBOR encoding for apply requests. #128273
    • Dynamic client supports CBOR apply: KEP-4222: Support CBOR encoding for apply requests. #128273
    • Generated clients support CBOR apply: KEP-4222: Support CBOR encoding for apply requests. #128273
  • TODO: Dynamic client watch integration test coverage: KEP-4222: Add integration test for CBOR-enabled dynamic client watches. #128491
• Alpha: Test: Integration: Request content-type falls back to JSON for life of RESTClient: KEP-4222: Fall back to JSON request encoding after CBOR 415. #128497
• Custom Resource Storage: KEP-4222: Wire CBOR CR storage behind test-only feature gate. #128322
  • Test: Integration: Custom resources storage encoding is CBOR with API server feature gate enabled
  • Test: Integration: Custom resources storage encoding is JSON with API server feature gate disabled
  • Test: Integration: Mixed CBOR and JSON encodings in storage for a single custom resource can be retrieved with feature gate disabled
• Support nondeterministic encoding for responses: KEP-4222: Support nondeterministic encode for the CBOR serializer. #125678
  • Implement NondeterministicEncoder on serializer: KEP-4222: Support nondeterministic encode for the CBOR serializer. #125678
• Test: Integration: Client feature gate can force clients otherwise configured with a CBOR request encoding to use JSON: KEP-4222: Support CBOR for generated clients based on client-go feature gating. #128323
• Test: Integration: Client feature gate can change the default request encoding to CBOR for clients with no explicitly configured encoding: KEP-4222: Support CBOR for generated clients based on client-go feature gating. #128323
• Dynamic client supports CBOR behind client-side gates KEP-4222: Add CBOR variant of admission webhook integration test. #128243
• Generated clients support CBOR behind client feature gates: KEP-4222: Support CBOR for generated clients based on client-go feature gating. #128323
• Remove test-only feature gate and client feature gate instances and introduce regular feature gates: KEP-4222: Add CBOR feature gates. #128539
• Serving Codecs wired to apiserver feature gate, integration test wiring removed: KEP-4222: Wire serving codecs to CBOR feature gate. #128503
  • Integration test: storage encoding of all built-in types should be Protobuf Add integration test for per-resource storage encoding. #128590
  • Watch responses use the application/cbor-seq media type: KEP-4222: Use cbor-seq content-type for CBOR watch responses. #128501
• Test: E2E: Request and response content-type negotiation with 1.17 sample API server: KEP-4222: Fix JSON fallback for clients using default content-type and add E2E client test. #128641

TODOs

Tracks debt built up as we merged. If there were some edges that didn't work, we want to avoid losing track of those, while unlocking as much as possible.

• KEP-4222: Add CBOR Serializer implementation. #122881 leaves debt
  • IntDecConvertSignedOrFail errors on overflow, JSON will try to fall back to float64. Discussed in March 6 2024 api-machinery biweekly and decided to continue returning an error for this case instead of imitating JSON's fallback to float64.
  • UnstructuredList implementation of SetUnstructuredContent is not identical to what unstructuredJSONScheme does: KEP-4222: Decode CBOR to UnstructuredList as UnstructuredJSONScheme does. #124775
  • Should Inf/-Inf/NaN be supported? Current Protobuf will encode this, but JSON will produce an error
    • Discussed in March 6 2024 api-machinery biweekly: Match JSON behavior here, error on encode and decode: KEP-4222: Reject NaN or infinite floating-point values in the CBOR serializer. #125423
  • DefaultMapType isn't set properly by DecOptions, set it again here until the bugfix is adopted
  • Missing test case that demonstrates list encode/decode. Probably covered above, but indicating just in case: KEP-4222: Decode CBOR to UnstructuredList as UnstructuredJSONScheme does. #124775
• Handle typed CR clients using custom json Marshaler/Unmarshaler implementations: KEP-4222: Check for and reject unsupported custom marshalers and unmarshalers. #125748
  • Enforce the same check on args to Marshal and Unmarshal KEP-4222: Reject custom marshalers from direct CBOR Marshal and Unmarshal. #128462
• Add Unstructured field accessor(s) that handle coercion of numeric values to a specific type, e.g. "NestedNumberAsFloat": Add NestedNumberAsFloat64 unstructured field accessor. #127838
• Decoder unit test verifies that the "self-described CBOR" tag is accepted without changing the result: KEP-4222: Add CBOR decoder unit test that accepts tag 55799. #124799
• Go byte arrays marshal to CBOR array of integers, not byte string: KEP-4222: Encode byte array to CBOR as array of integer, not byte string. #125418
• Implementations of encoding.BinaryMarshaler or encoding.BinaryUnmarshaler are not used during encoding or decoding: KEP-4222: Disable recognition of Binary(Unm|M)arshaler in CBOR serializer. #125422
• Decode allocation fuzz target covers unstructured typed. built-in go types are excluded because we don't know of a tool to generate fuzzed cbor bytes that will actually be usefully decoded and the coverage would end up checking custom marshallers and not the cbor library itself
• Add beta requirement to opt out of automatic transcoding of RawExtension and FieldsV1 to JSON for Kubernetes components: KEP-4222: Add transcoding opt-out as beta criteria. enhancements#4758

Why is this needed?

The requirements tracked here originate from the KEP alpha criteria.

[benluddy]

/sig api-machinery

[jiahuif]

/assign @benluddy
For this is a TODO list
/triage accepted

[deads2k]

I think we're at alpha!

@benluddy would you do the honor of /close?

[benluddy]

/close

[k8s-ci-robot]

@benluddy: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.