gatsby-source-graphql plugin re-runs createPage when using fragments

[jnlsn]

Description

If a page query contains a fragment, the graphql source plugin seemingly re-runs the whole create page query multiple times.

Steps to reproduce

This can be recreated from the graphql source example here:
https://github.com/gatsbyjs/gatsby/tree/master/examples/using-gatsby-source-graphql

1. In gatsby-node.js console.log() the blog title.
2. gatsby develop and see that it outputs the titles twice
3. in blog-post.js remove the titleImage fields from the exported query which includes a GatsbyImageSharpFixed fragment.
4. gatsby develop and see that it outputs the titles once.

Expected result

createPage should be called once per page.

Actual result

createPage is called twice per page.

Environment

System:
OS: macOS 10.15.2
CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
Shell: 5.7.1 - /bin/zsh
Binaries:
Node: 10.16.3 - /usr/local/bin/node
Yarn: 1.16.0 - /usr/local/bin/yarn
npm: 6.9.0 - /usr/local/bin/npm
Languages:
Python: 2.7.16 - /usr/bin/python
Browsers:
Chrome: 78.0.3904.108
Firefox: 71.0
Safari: 13.0.4
npmPackages:
gatsby: ^2.0.0 => 2.18.11
gatsby-image: ^2.0.34 => 2.2.36
gatsby-plugin-netlify: ^2.0.0 => 2.1.30
gatsby-plugin-sharp: ^2.0.30 => 2.3.7
gatsby-source-filesystem: ^2.0.27 => 2.1.42
gatsby-source-graphql: ^2.0.0 => 2.1.28
gatsby-transformer-sharp: ^2.1.17 => 2.3.9
npmGlobalPackages:
gatsby-cli: 2.8.8

[vladar]

Looks like double call of createPages is unrelated to fragments. The root cause is the following resolver:

gatsby/examples/using-gatsby-source-graphql/gatsby-node.js

Lines 57 to 62 in 768df7d

	GraphCMS_Asset: {
	imageFile: {
	type: `File`,
	// projection: { url: true },
	resolve(source, args, context, info) {
	return createRemoteFileNode({

As you see a new node is created while the query is running (for a remote file). And gatsby develop re-runs createPages when data in the node store changes. If you try the same example with gatsby build it will run createPages only once.

Unfortunately, this is the expected behavior for gatsby-source-graphql at the moment. See #15906 for some context.

The current workaround is to create remote file nodes separately in the sourceNodes API and then fetch those nodes in the resolver mentioned above.

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

[yaacovCR]

You should be able to use existing createLink option and use an Apollo Link that supports caching, ie https://github.com/prisma-labs/http-link-dataloader

[rburgst]

@vladar can you give more hints on how to integrate this into a source plugin?

[sanderploegsma]

Here's a sample gatsby-node.js that uses the sourceNodes API to create local copies of remote images in Wordpress:

gatsby-node.js

const { createHttpLink } = require("apollo-link-http");
const fetch = require("node-fetch");
const gql = require("graphql-tag");
const { createRemoteFileNode } = require(`gatsby-source-filesystem`);

const uri = process.env.GRAPHQL_ENDPOINT;
const headers = {};

exports.sourceNodes = async (
  { actions, getCache, store, reporter }
) => {
  const { createNode } = actions;
  const link = createHttpLink({
    uri,
    fetch,
    headers
  });
  const query = gql`
    query FetchImages($after: String) {
      mediaItems(first: 100, after: $after) {
        pageInfo {
          hasNextPage
          endCursor
        }
        nodes {
          sourceUrl
        }
      }
    }
  `;

  let hasNextPage;
  let after;
  do {
    const { data } = await makePromise(
      execute(link, { query, variables: { after } })
    );
    await Promise.all(
      data.mediaItems.nodes.map(({ sourceUrl }) => {
        if (!sourceUrl || sourceUrl === "false") return Promise.resolve();
        return createRemoteFileNode({
          url: sourceUrl,
          parentNodeId: null,
          store,
          getCache,
          createNode,
          createNodeId: () => sourceUrl,
          httpHeaders: headers,
          reporter
        });
      })
    );
    hasNextPage = data.mediaItems.pageInfo.hasNextPage;
    after = data.mediaItems.pageInfo.endCursor;
  } while (hasNextPage);
};

exports.createResolvers = ({ createResolvers }) => {
  createResolvers({
    MediaItem: {
      localFile: {
        type: "File",
        resolve: (source, args, context) => context.nodeModel.getNodeById({ id: source.sourceUrl, type: "File" })
      }
    }
  })
}

[rburgst]

@sanderploegsma thanks a lot for the fast response, I tried your approach but am running into problems

either if I create a resolver in the original wpgraphql type, I get

 ERROR #11321  PLUGIN

"gatsby-source-graphql" threw an error while running the sourceNodes lifecycle:

Cannot use GraphQLDirective "@include" from another module or realm.

Ensure that there is only one instance of "graphql" in the node_modules
directory. If different versions of "graphql" are the dependencies of other
relied on modules, use "resolutions" to ensure only one version is installed.

https://yarnpkg.com/en/docs/selective-version-resolutions

Duplicate "graphql" modules cannot be used at the same time since different
versions may have different capabilities and behavior. The data from one
version used in the function from another could produce confusing and
spurious results.

  80 |     if (!sdl) {
  81 |       introspectionSchema = await introspectSchema(link);
> 82 |       sdl = printSchema(introspectionSchema);
     |             ^
  83 |     } else {
  84 |       introspectionSchema = buildSchema(sdl);
  85 |     }

File: node_modules/gatsby-source-graphql/gatsby-node.js:82:13

and otherwise if I use your code as is, it complains that MediaItem is not defined (and I should use createTypes)

I have created a sample project: https://github.com/rburgst/gatsby-rebuilding-problem/tree/experiments/sourceNodes, it would be super kind if you could help me fix it.

[sanderploegsma]

if I clone your example project everything seems to work correctly. Judging by the error you posted it looks like you have multiple versions of the graphql dependency, maybe you're using an older version of Yarn? If you remove the node_modules folder and do a clean install, it should work.

[rburgst]

You are correct, i had 2 graphql versions in my dependencies. Sorry for not reading the error well enough. Thanks so much! It works now. I will update my test repo so people can find a working solution if they encounter the same issue.