Feature : merge multiple specification files

[wheezil]

Description

While JSON references are useful for certain cases as described in this tutorial, they cannot "include" an entire specification and merge it into the main one. This would be helpful in organizing API specifications into modules, where each module can include a full complement of schemas, paths, etc.

An "import" feature is proposed as a pre-processor before calling the swagger parser, with a section like:

"imports": {
   "collision_handling":"error|first_wins|last_wins",
   "locations":[
      { "location":"URL" },
      { "location":"URL" },
      ...
   ]
}

Of course, both YAML and JSON are supported. It would be enabled by a command-line switch --enable-imports. If enabled, the pre-processor would read each of the import sources, parse it as JSON, then merge the import JSON objects with the main file. It would write the resulting JSON to a temporary file and hand that to the main engine. This raises the possibility of collisions (same keys defined in multiple sources). Collisions are resoloved using a set of rules:

• If both keys reference a map, the maps are merged
• If both keys reference an array, second array is appended to the first
• Otherwise, the collision_handling settings controls behavior:
  • error is the default: an exception is thrown
  • first_wins means that the later definition is ignored and a warning is logged
  • last_wins means that the earlier definition is replaced and a warning is logged

openapi-generator version

N/A

OpenAPI declaration file content or url

common.json

{
    "components": {
        "schemas": {
            "DateTime": {
                "type":"string",
                "format":"date-time"
            },
            "ResponseBase": {
                "properties": {
                    "type":{
                        "type":"string"
                    }
                }
            },
            "ErrorResponse": {
                "allOf":[
                    { "$ref": "#/components/schemas/ResponseBase" }
                ]
            },
            "Request": {
                "properties": {
                    "id": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

spec.json

{
    "openapi": "3.0.0",
    "info": {
        "description": "blah",
        "version": "1.0.0",
        "title": "blah"
    },
    "imports":{
        "collision_handling":"error",
        "locations":[
            { "location":"common.json" }
        ]
    },
    "paths": {
        "/test1": {
            "post": {
                "tags": [
                  "test"
                ],
                "operationId": "testOp1",
                "responses": {
                    "405": {
                        "description": "Invalid input",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/ErrorResponse"
                                }
                            }
                        }
                    },
                    "200": {
                        "description": "successful operation",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/Response"
                                }
                            }
                        }
                    }
                },
                "requestBody": {
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/Request"
                            }
                        }
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "ErrorResponse": {
                "allOf":[
                    {
                        "type":"object",
                        "properties": {
                            "message": {
                                "type": "string"
                            }
                        }
                    }
                ]
            },
            "Request": {
                "properties": {
                    "name": {
                        "type": "string"
                    }
                }
            },
            "Response": {
                "properties": {
                    "email": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

When processed (there are no collisions), would produce:

{
    "openapi": "3.0.0",
    "info": {
        "description": "blah",
        "version": "1.0.0",
        "title": "blah"
    },
    "paths": {
        "/test1": {
            "post": {
                "tags": [
                  "test"
                ],
                "operationId": "testOp1",
                "responses": {
                    "405": {
                        "description": "Invalid input",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/ErrorResponse"
                                }
                            }
                        }
                    },
                    "200": {
                        "description": "successful operation",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/Response"
                                }
                            }
                        }
                    }
                },
                "requestBody": {
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/Request"
                            }
                        }
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "DateTime": {
                "type":"string",
                "format":"date-time"
            },
            "ErrorResponse": {
                "allOf":[
                    { "$ref": "#/components/schemas/ResponseBase" },
                    {
                        "type":"object",
                        "properties": {
                            "message": {
                                "type": "string"
                            }
                        }
                    }
                ]
            },
            "Request": {
                "properties": {
                    "id": {
                        "type": "string"
                    },
                    "timestamp": {
                        "$ref" : "#/components/schemas/DateTime"
                    }
                }
            },
            "Response": {
                "properties": {
                    "email": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

Note that ErrorResponse's allOf list has been concatenated, and Request's properties object has been merged.

Command line used for generation

New option --enable-imports would enable this feature

Steps to reproduce

Related issues/PRs

None in this repo, but in swagger-parser:

swagger-api/swagger-parser#147

Suggest a fix/enhancement

[wheezil]

Comments welcome! Perhaps this simply becomes a talking point and should really be considered as part of the swagger parser instead.

[wheezil]

It was pointed out in the forum that this messes with IDEs and editors. To address that, one could separate the API into swagger-compliant modules, and build an "import only" file to perform the preprocessing. Perhaps this feature should be moved entirely out of the swagger file? There could be a pure merge pre-processor that is outside of the swagger files but otherwise operates as described?

[wheezil]

Alternative idea: instead of putting the imports section into a file, add command-line options like

• --merge-inputs=input1,input2,...
• --merge-collision-handling=error|first_wins|last_wins

This would trigger the pre-processor to merge the input files to a temp file, which becomes the effective -i argument. Either that or -i itself could take a comma-separated list, the presence of which triggers the merge (although , being a valid filename character that would cause ambiguity).

[wing328]

Right, I agree it should be done by the Swagger Parser instead of OpenAPI Generator but good to start the discussion.

Have you tried https://github.com/maxdome/swagger-combine (which does not seem to support OAS v3 at the moment)? Is that what you're looking for?

[wheezil]

swagger-combine requires that each input be a complete API. In particular, it does not support one input defining schemas that another input needs, which is the primary use-case of this issue.

[wheezil]

This is really quite a simple transformation -- much simpler in fact than swagger-combine because it operates entirely at the JSON/YAML level and does not need to parse swagger. Perhaps I should make a new project for this?

[wheezil]

json-merger actually comes close, but it requires that operations be injected deep into one of the source documents.

[wheezil]

Actually... json-merger may do everything I need. It doesn't concatenate arrays by default, but it does merge maps. I think that array concatenation may be somewhat contrived.

[robertmassaioli]

I wrote a quick tool to do this recently. I call it openapi-merge. There is a library and an associated CLI tool:

• https://www.npmjs.com/package/openapi-merge
• https://www.npmjs.com/package/openapi-merge-cli

In order to use the CLI tool you just write a configuration file and then run npx openapi-merge-cli. The configuration file is fairly simple and would look something like this:

{
  "inputs": [
    {
      "inputFile": "./gateway.swagger.json"
    },
    {
      "inputFile": "./jira.swagger.json",
      "pathModification": {
        "stripStart": "/rest",
        "prepend": "/jira"
      }
    },
    {
      "inputFile": "./confluence.swagger.json",
      "disputePrefix": "Confluence",
      "pathModification": {
        "prepend": "/confluence"
      }
    }
  ], 
  "output": "./output.swagger.json"
}

For more details, see the README on the NPM package.