Rewrite GraphQL schema generation and query parsing (close #2801)

[lexi-lambda]

This PR is a work in progress. This description will be updated to fit the usual format once it is closer to done.

Overview

This pull request is a complete rearchitecting of the way we build GraphQL schemas and parse GraphQL queries. It implements some of the ideas in #2801, but the approach is rather different from the one described there (as that approach turned out to be too naïve in various ways).

The core API is built around a new type, Parser, which has the following interface:

data Parser (k :: Kind) (m :: * -> *) (a :: *)

parserType :: Parser k m a -> Type k
runParser :: Parser k m a -> ParserInput k -> m a

instance Functor (Parser k m)

This may raise more question than it answers. How do you do anything with a Parser if it’s only a Functor? What is ParserInput? And what on earth is a Kind?

Before we answer those questions, let’s start with a recap of the Big Idea:

• Currently, schema generation and query parsing are separate code paths, so they can get out of sync. We want a new approach that can do both at once.

• We want to parse, not validate, so there shouldn’t be a separate GraphQL validation pass. Checking consistency against the schema should happen naturally as part of query parsing.

• We need to be able to reflect on the GraphQL schema without even necessarily having a query to execute, since we need to serve GraphQL introspection queries.

The Parser abstraction is designed to solve all of the above problems. It works like this:

• A Parser is something that knows how to parse a piece of a GraphQL AST. (Note that it doesn’t have anything to do with parsing GraphQL query concrete syntax, that step is still handled by graphql-parser-hs.)

• You build Parsers using parser combinators, just like other Haskell parsing libraries. Since a Parser parses a GraphQL AST, the primitive combinators in this library are things like scalar, enum, and object.

• However, unlike a traditional parser combinator library, every Parser is associated with a GraphQL type. For example, if you have a Parser that parses an input object, you can reflectively get information about what fields it contains and what their types are.

Challenges

This idea is simple on the surface, but subtle to implement. Broadly speaking, there are three significant sources of complexity:

1. GraphQL ASTs are much more complicated than something like JSON. Field sets, fragments, nullability, and variables all add rules that need to be captured in the type system.

2. Because our GraphQL schema is generated dynamically, based on information that isn’t known until runtime, we essentially have to generate a program on the fly. This stratifies the schema generation logic into two distinct phases: schema generation and query parsing. Both these phases can potentially fail due to invalid input, and in Haskell, that means we need two monads nested inside one another!

3. The need to be able to reflect on the schema imposes strong constraints on the expressiveness of the combinator language. Especially challenging are mutually-recursive types, since the Parser abstraction requires these actually be represented in Haskell as a cyclic data structure!

Given the above, I think the solution I’ve come up with strikes a good compromise. The types are not so trivial that they are immediately obvious at first glance, but I think they’re actually quite readable once you understand what’s happening. The internal implementation of the combinators is very subtle in places, but I think that’s okay: that should have to change very infrequently. The important thing is what using the combinators looks like, and that, happily, is fairly simple.

API overview

GraphQL schemas

Before talking about parsers, let’s look at the representation of the GraphQL schema itself. GraphQL types are represented in a way that should be at least somewhat recognizable:

data Type k
  = NonNullable (NonNullableType k)
  | Nullable (NonNullableType k)

data NonNullableType k
  = TNamed (Definition (TypeInfo k))
  | TList (Type k)

data TypeInfo k where
  TIScalar :: TypeInfo 'Both
  TIEnum :: NonEmpty (Definition EnumValueInfo) -> TypeInfo 'Both
  TIInputObject :: [Definition (FieldInfo 'Input)] -> TypeInfo 'Input
  TIObject :: [Definition (FieldInfo 'Output)] -> TypeInfo 'Output

Here we have the usual suspects accounted for. What’s most interesting about these definitions is that they’re all indexed by k, which describes their kinds. These are not Haskell kinds, but “GraphQL kinds,” which have the following definition:

data Kind = Both | Input | Output

The GraphQL specification does not actually use the word “kinds” when it discusses the classification of types into input types and output types, but the name is apt. Since GraphQL types are Haskell values, it follows naturally that GraphQL kinds should be Haskell types—everything is shifted one level down—and that is precisely the case. The Kind datatype is used with DataKinds as a type-level index to TypeInfo, and this classification is used to enforce various invariants.

Parsers

As mentioned earlier, the Parser type is at the center of the API, and it has the following signature:

data Parser (k :: Kind) (m :: * -> *) (a :: *)

Here, the k parameter describes the Type associated with the Parser, and it also determines what the input to the parser will be:

runParser :: Parser k m a -> ParserInput k -> m a

type family ParserInput k where
  ParserInput 'Both = Value Variable
  ParserInput 'Input = Value Variable
  ParserInput 'Output = SelectionSet Variable

(The fact that we can treat 'Both the same as 'Input here is something of a happy coincidence that we can use to simplify the types; comments in the code explain in more detail.)

Parsing atoms

Parsers are constructed using some parser combinator functions. The “atomic” parser constructors are scalar and enum:

scalar
  :: MonadParse m
  => Name
  -> Maybe Description
  -> ScalarRepresentation a
  -> Parser 'Both m a

data ScalarRepresentation a where
  SRBoolean :: ScalarRepresentation Bool
  SRInt :: ScalarRepresentation Int32
  SRFloat :: ScalarRepresentation Double
  SRString :: ScalarRepresentation Text

enum
  :: MonadParse m
  => Name
  -> Maybe Description
  -> NonEmpty (Definition EnumValueInfo, a)
  -> Parser 'Both m a

Other combinators accept parsers as input to produce new parsers. For example, parsers for list types and nullable types can be created with list and nullable, respectively:

nullable :: (MonadParse m, 'Input <: k) => Parser k m a -> Parser k m (Maybe a)
list :: (MonadParse m, 'Input <: k) => Parser k m a -> Parser k m [a]

These type signatures introduce a use of the <: operator, which is a subtyping constraint on kinds. The constraint 'Input <: k specifies that k can be either 'Input or 'Both but cannot be 'Output.

Parsing input objects

One of the most important combinators is object, which builds a parser capable of parsing input objects:

object
  :: MonadParse m
  => Name
  -> Maybe Description
  -> FieldsParser 'Input m a
  -> Parser 'Input m a

The object combinator uses a separate FieldsParser abstraction to specify the fields of the input object. A FieldsParser is similar to a Parser, but a key difference is that FieldsParser has an Applicative instance, while Parser only has a Functor instance. This is because any two FieldsParsers can be combined to create a bigger FieldsParser that parses a union of their fields.

Field parsers are created using combinators that accept ordinary Parsers as arguments. The simplest of these is field:

field
  :: (MonadParse m, 'Input <: k)
  => Name
  -> Maybe Description
  -> Parser k m a
  -> FieldsParser 'Input m a

This builds a FieldsParser that parses exactly one field, using the given Parser to parse its value. If the field’s type is nullable, the field will be optional with a default value of null, otherwise the field will be required. If a default value other than null is desired, the fieldWithDefault combinator can be used instead:

fieldWithDefault
  :: (MonadParse m, 'Input <: k)
  => Name
  -> Maybe Description
  -> Value Void -- ^ default value
  -> Parser k m a
  -> FieldsParser 'Input m a

Field parsers are combined using the Applicative instance, which allows them to be used with ApplicativeDo and sequenceA, both of which are very useful for defining composite parsers. For example, a parser for comparison expressions can be defined this way:

let name = getName columnParser <> $$(litName "_comparison_exp")
object name Nothing $ catMaybes <$> sequenceA
  [ field $$(litName "_cast") Nothing (ACast <$> castParser)
  , field $$(litName "_eq") Nothing (AEQ True . mkParameter <$> columnParser)
  , field $$(litName "_ne") Nothing (ANE True . mkParameter <$> columnParser)
  -- etc.
  ]

Parsing selection sets

Finally, there are combinators for parsing selection sets. The selectionSet combinator is essentially the same as object, just for output objects instead of input objects:

selectionSet
  :: MonadParse m
  => Name
  -> Maybe Description
  -> FieldsParser 'Output m a
  -> Parser 'Output m a

Output field parsers are a little more complicated than input field parsers for two reasons:

1. Output fields can have arguments.

2. Output fields that return atomic values like scalars and enums don’t have subselection sets, but fields that return other selection sets do.

Both of these are handled by the selection function, which has the most complicated type of all the combinators:

selection
  :: forall k m a b
   . (MonadParse m, 'Output <: k)
  => Name
  -> Maybe Description
  -> FieldsParser 'Input m a
  -> Parser k m b
  -> FieldsParser 'Output m (Maybe (SelectionResult k a b))

type family SelectionResult k a b = r | r -> k a where
  SelectionResult 'Both   a _ = (Name, a)
  SelectionResult 'Output a b = (Name, a, b)

The extra FieldsParser 'Input argument is fairly straightforward—it parses the field’s arguments—but the SelectionResult return type is a bit trickier. It handles point 2 above by returning an extra result if the subparser is an output parser; if it’s an input parser, it just returns the field’s arguments and only uses the input parser for its type.

This is a bit subtle to describe, but when using it, it mostly just works: the result of selection is just slightly different depending on the type of its subparser.

Tying the knot

As mentioned in the challenges section, one of the trickiest parts of the implementation involves mutually-recursive types. Mutual recursion didn’t cause any trouble when we represented a reference to a type as that type’s Name, but with the Parser abstraction, the reference must be more direct (since the Parser needs to actually call the Parser for the referenced type).

This requirement means that Parsers for mutually-recursive types must be represented as a cyclic data structure. If we were to build that data structure by hand, it would be an enormous pain: we’d have to very carefully thread around (lazy) references to all the parsers we construct. To make things easier, we use a MonadSchema class that mostly handles the details automatically using memoization.

When constructing a Parser for a recursive or mutually-recursive type, the constructor function should be wrapped in memoize or one of its variants:

memoize
  :: (HasCallStack, MonadSchema n m, Ord a, Typeable a, Typeable b, Typeable k)
  => TH.Name
  -> (a -> m (Parser k n b))
  -> (a -> m (Parser k n b))

This type signature looks very scary, but it’s actually not nearly as intimidating as it looks. The HasCallStack constraint is just for error reporting, and the Typeable constraints just get magically solved behind the scenes by the compiler. That means this function morally has a type signature like this:

memoize
  :: (MonadSchema n m, Ord a)
  => TH.Name
  -> (a -> m (Parser k n b))
  -> (a -> m (Parser k n b))

Explaining how to use it is best done through an example. Consider the types of GraphQL boolean expressions, which are recursive:

input type_bool_exp {
  _or: [type_bool_exp!]
  _and: [type_bool_exp!]
  _not: type_bool_exp
  ...
}

A naïve implementation of a Parser for this type might look like this:

boolExp
  :: (MonadError QErr m, MonadParse n)
  => QualifiedTable -> m (Parser 'Input n BoolExp)
boolExp tableName = do
  name <- ...
  tableFieldParsers <- ...
  recur <- boolExp tableName -- direct recursion!
  pure $ BoolAnd <$> P.object name Nothing $ sequenceA
    [ P.fieldOptional $$(G.litName "_or") Nothing (BoolOr <$> P.list recur)
    , P.fieldOptional $$(G.litName "_and") Nothing (BoolAnd <$> P.list recur)
    , P.fieldOptional $$(G.litName "_not") Nothing (BoolNot <$> recur)
    ] ++ tableFieldParsers

Note the direct recursive call to boolExp. Because boolExp is monadic (it does some error checking during schema construction, so it must be), laziness won’t us help here. If we were to run this, we would just loop forever!

One solution would be to use MonadFix to add a little explicit sharing, which would work okay for this simple self-recursive example. Unfortunately, it’s much less nice for mutually-recursive types, since it becomes unclear where to put the call to mfix. To avoid that problem, we can use memoize:

boolExp
  :: (MonadSchema n m, MonadError QErr m)
  => QualifiedTable -> m (Parser 'Input n BoolExp)
boolExp = P.memoize 'boolExp \tableName -> do -- explicit memoization
  name <- ...
  tableFieldParsers <- ...
  recur <- boolExp tableName -- recursion will be memoized
  ...

Now each call to boolExp will be cached, and further calls with the same QualifiedTable argument will return the same Parser. The TH.Name argument to memoize is a bit of a trick: we need to somehow distinguish different parser constructors from each other, and Template Haskell name quotations provide a convenient, statically-known unique key. The actual name doesn’t have any significance, it’s just used for its Ord instance.

Remaining work

The Parser abstraction seems to work nicely, and I think the tricky cases in the interface are now worked out. However, there is still a lot of programming left to be done:

• selectionSet needs to be improved to handle fragments. This shouldn’t be too hard, since we can still use the normalization strategy we currently do.

• All the existing code needs to be ported to use the new abstractions. I do not expect this to be challenging, but I do expect it to take time.

Also, the current interface does not itself include support for union types and interfaces, so it only achieves feature parity with the current approach. I believe adding union types and interfaces should not be challenging, but I have not yet thought about it in detail.

[netlify]

Deploy preview for hasura-docs ready!

Built with commit af72e73

https://deploy-preview-4111--hasura-docs.netlify.app

[0x777]

@lexi-lambda

There are three use cases related to Unions and Interfaces that need to be handled

Allow executing top level fields from multiple sources in the same query.

Currently, when you have a remote schema, you could have a query as follows:

query q ($order_id: Int!) {
  order {
  }
  getPaymentdetail(..) {
  }
}

where order is a Postgres table but getPaymentDetail comes from an external remote schema.

What do we do currently?

We use a hacky approach to determine if all of the root fields belong to the same source, if so, we allow the query otherwise we reject it.

If the source is a remote schema, we simply forward the request to the remote server without any validation.

The current approach is hacky because it doesn't handle fragment spreads on query_root

fragment queryAndPayment on query_root {
  order {
  }
  getPaymentdetail(..) {
  }
}
query q ($order_id: Int!) {
  ...queryAndPayment
}

queries such as these will result in internal errors.

How can we fix this?

I'm not sure, I can share my thoughts on how this can be fixed with our current validation approach:

By the end of the validation phase, we have an AST representing a 'denormalized' GraphQL query (i.e, no fragments and variables). We can extend this AST to support denormalized Unions and Interfaces.

Consider this schema and query:

interface Attachment {
  name: String!
  location: Url!
}

type Picture implements Attachment {
  name: String!
  location: Url!
  thumbnail: Url!
  ...
}

type Document implements Attachment {
  name: String!
  location: Url!
  previewLink: Url!
  ...
}

type Query {
  getAttachments(emailId: ID!): [Attachment!]!
}

Imagine such a query:

fragment EmailAttachmentDetails {
  ... on Picture {
    thumbnail
  }
  ... on Attachment {
    previewLink
  }
}

query {
  getAttachments(emailId: 1) {
    name
    location
    ... EmailAttachmentDetails
  }
}

This can be denormalized as follows:

query {
  getAttachments(emailId: 1) {
    name
    location

  ... on Picture {
    name
    location
    thumbnail
  }

  ... on Attachment {
    name
    location
    previewLink
  }
}

i.e,

1. Get rid of named fragments and only use inline fragments
2. Merge all the common fields into the fragment of the implemented type in the
   correct order.

Consider this query for example:

fragment EmailAttachmentDetails {
  ... on Picture {
    thumbnail
  }
  ... on Attachment {
    previewLink
  }
}

query {
  getAttachments(emailId: 1) {
    name
    ... EmailAttachmentDetails
    location
  }
}

The denormalized query should look like this:

query {
  getAttachments(emailId: 1) {
    name
    location

  ... on Picture {
    name
    thumbnail
    location
  }

  ... on Attachment {
    name
    previewLink
    location
  }
}

The Selection Set of an interface can be represented by this product type:

type InterfaceSelectionSet = (List Field, Map InterfaceImplementation Field)

Similarly, a Union type's selection set can be captured with:

type UnionSelectionSet = Map UnionMember Field

So our current SelSet type changes to

data SelectionSet
  = SelSetObject !(Seq Field)
  | SelSetInterface !InterfaceSelectionSet
  | SelSetUnion !UnionSelectionSet

If we can get rid of named fragments from the query_root it should be straight forward to generate the correct execution plan based on the list of Fields.

Allow specifying permissions on remote schema types

We would like to support specifying permissions on remote schema's types. This is useful in cases where the 'remote schema' is not under our user's control, say for example Stripe's API. In this case our user may want to hide fields of certain types for certain roles (like column permissions on tables). Further, we may want to add argument presets like column presets. Currently the way we enforce permissions is at the validation layer, which means that we'll need to validate interfaces and unions.

If we generate the correct schema, the same approach as above could solve this problem too.

Relay

WIP (I'll need some time to first implement the Global Object Specification) to share any insights on this.

[hasura-bot]

Review app for commit c350585 deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-c3505857

[hasura-bot]

Review app for commit 07f511d deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-07f511de

[hasura-bot]

Review app for commit 4fd8e4d deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-4fd8e4d6

[hasura-bot]

Review app for commit c177d88 deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-c177d886

[hasura-bot]

Review app for commit 07e38e8 deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-07e38e8e

[hasura-bot]

Review app for commit 8c7d38d deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-8c7d38d7

[hasura-bot]

Review app for commit f930962 deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-f9309627

[tirumaraiselvan]

changelog

[hasura-bot]

Review app for commit 69c4c6a deployed to Heroku: https://hge-ci-pull-4111.herokuapp.com
Docker image for server: hasura/graphql-engine:pull4111-69c4c6a9

[hasura-bot]

Review app https://hge-ci-pull-4111.herokuapp.com is deleted

[abooij]

This branch has been archived here.