Soft 404s in Google Search Console Despite Valid Pages – Middleware, Error Boundary, and Valid 200 Responses in Place

[ayazwx]

Summary

Hello, Everyone,

I am posting on this forum seeking guidance regarding the ongoing soft 404 issues in Google Search Console for my site, https://www.clicktoguide.com. In production, all pages are valid and successfully returning 200 status.

🛠️ What We Have Done So Far:

1. Implemented Custom 404.tsx and not-found.tsx

• Proper 404 handling is in place in our /app (Next.js 14.2.5, React 18.3.1, App Router).

• No content is missing on valid URLs that can be interacted with by users.

2. Global Error Boundary Added

• To catch any unexpected runtime exceptions (Something went wrong.) render a fallback UI for all app routes.

3. Monitoring Edge Middleware Enabled

• With OpenTelemetry logs, it is validated that requests from googlebot are being served and status 200 is returned.

• Example user agent:

• Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P)…(compatible; Google-InspectionTool/1.0)

• Logs show valid execution:

• POST / status=200 src/middleware cache MISS

4. Verified the deployment behavior Vercel

• Vercel returns 200 with content. There is no such condition encountered on these paths, notFound() will not be triggered, nor will throw occur.

• Expected behavior for static and dynamic routes for real users, as well as crawlers, has been observed.

5. Still Reports Soft 404 Errors

• Using Google’s URL Inspection tool:

• ❌ "Soft 404: Page cannot be indexed."

• ➕ "Application error: a client-side exception has occurred (see browser console).”

But:

• Some of these URL Inspection runs do not include logs.

• There appears to be no execution of the middleware, which indicates a client-side failure.

• The fallback error boundary also does not render.

🤔 Our Hypothesis:

• Googlebot is causing a client-side hydration error, possibly a race condition, script timing, or incompatible SSR/CSR interaction.

• Next.js, even on valid pages, injecting not-found.js chunks may confuse Googlebot.

• Googlebot’s inspection tool is in an edge case where middleware/ error boundary is bypassed silently causing a blank page or uncaught error.

• Google might be shown blank DOM with broken hydration that does not happen in real-user scenarios.

🔧 What We’re Looking For:

• Has anyone experienced this problem and found a solution?

• Has anyone encountered the issue where Google Search Console reports Soft 404 despite having a 200 page full of content?

• Any way to disable not-found.js chunk injection on valid routes known?

• Can we modify next.config.js to include custom headers for a bot-specific fallback?

• Is there a suggested approach to emulate Googlebot’s crawl exactly how GSC does?

📉 Current Status:

• Soft 404s decreased ~10% since implementing fixes and revalidating in GSC.

• Issue still persists intermittently, especially for URLs that should be fully valid.

• Vercel support is investigating; we’re looking for community insight in the meantime.

Additional information

No response

Example

No response

[errorproofdev]

Thanks for bringing this up here - I think it's the first time I've seen a thread about this despite it being extremely prevalent on Google's Search Console Community, Reddit, and other articles online.

Our experience has been almost identical to you. The major difference is we are self-hosting our instance on a single AWS EC2 vs Vercel.

We are using [email protected] with App Router, and had been fine for almost 2 years with zero soft 404s. Suddenly on December 3rd we were hit with 3 soft 404s that quickly spiked to ~200 pages.

This is nothing like we've ever seen before (or I've ever experienced in 10 years of development), and we've had very similar steps to diagnose and troubleshoot without luck.

Below I'll link some references of other users reporting similar issues (some using similar frameworks like Nuxt). Google's response has been next to nothing on the matter; however, I would say things point towards:

1. Crawl Budget changes from algorithim changes
2. Googlebot encountering CSR errors (this one is odd, as we have tested disabling javascript entirely and our website's HTML and content serves fine
3. Prefetching (RSC and _rsc parameters) playing some role in either crawl budget or googlebot inability to correctly understand them?
4. Severely de-prioritized website leading to Google not ever coming back after the 'Fetch HTML' stage and 'rendering' the page for indexing.
5. Multiple quick deployments leading to version skew issues and Google caching old stale HTML (which loads broken or 404'd css/js chunks)

We've attempted dozens of changes over 4 months, and have not had any luck. This has led to our home page being de-indexed and strangely appears to only be affecting pages that are "SSG" on build. We've held off on upgrading to Next@v15 as we have not seen indications this will fix anything.

Let us know if you find out anything else, or anybody can provide insight into how we can further troubleshoot or debug what Googlebot is actually seeing when fetching the HTML.

References:

• https://support.google.com/webmasters/thread/322653126?authuser=1&hl=en
• https://www.reddit.com/r/nextjs/s/dkbBDmKMHt
• https://www.reddit.com/r/SEO/s/LHRjmSgzqx
• https://support.google.com/webmasters/thread/327530752/sudden-drop-on-google-bots-crawling-since-feb-4th-2025?hl=en&authuser=1&sjid=393150578379882359-NC
• https://www.reddit.com/r/webdev/s/bbIU4wodjy
• https://www.intrepidonline.com/blog/seo/how-to-solve-soft-404s-without-losing-your-mind-or-traffic/

Thanks!

[emilpriver]

Hey, We're having the same issues and do only seem to happen with our Next.JSs sites. I have talked to the CTO of Vercel on x (https://x.com/cramforce/status/1909703205698912486) and they talked about Skew Protection and https://github.com/uncurated-tests/graceful-degradation-hydration-error.

We have fixed any hydration error we have (sentry don't report anything).

We tried prerender.io as well with the same result

---

Hey @cramforce
I know we have been talking about this on X before but none of the stuff seems to be working. This soft-404 is random meaning that some day we get soft-404 on 1 page and some day do we get soft-404 on 400 pages. Is this something which is being worked on @ Vercel?

[emilpriver]

Hey @errorproofdev
Many thanks for the detailed text.
Some of the stuff we have done is that we're using skew protection already, but we also cache this files in Cloudflare for 365 days to prevent that they still exists. I don't know how good this work tho.

The only thing I can think of is that a external scripts is loading and failing to load for googlebot and causing white pages

[errorproofdev]

@aha-kid @emilpriver, no problem I hope it helps in some way. We spent many months in this state and were beginning to give up hope.

Something we are going to build internally and I'd recommend pursuing is an E2E testing tool via headless browser like puppeteer or playwright to fetch and then render any soft 404 pages in question.

Since Googlebot uses chromium puppeteer can very closely mimic its behavior in this way (ie. You can specify exact user agent and even the same chromium binary).

The latest confirmed version of chromium was 74 but this was as of 2019 in this blog article. They mention specifically they'll be keeping Googlebot updated which makes me wonder if this could be playing a factor in why seemingly out of the blue so many soft 404 issues occurred in JavaScript heavy frameworks. They also link to this separate checklist of common JavaScript rendering issues where they mention:

Ensure that your application uses feature detection for all critical APIs that it needs and provide a fallback behavior or polyfill where applicable.

and

Make sure your web components render as expected. Use the Rich Results Test or the URL Inspection Tool to check if the rendered HTML has all content you expect.
WRS flattens the light DOM and shadow DOM.

The fact Puppeteer is a chromium based headless browser means it follows a similar web rendering strategy as Googlebot and proven to be better at reproducing crawl/render issues better then a normal browser like Chrome.

TechnicalSeo has a suite of tools that can be extremely helpful in debugging and one is exactly what I am referencing: Fetch / Render. We were able to reproduce the client side error via this tool and are working on debugging things further as I am curious on what exactly is throwing and want to make sure this doesn't happen again.

Please let me know if you're able to reproduce the issue -- if yes you should be able to then whitelist the IP TechnicalSEO's puppeteer script is coming from via Cloudflare or NGINX on any review or staging environments and begin testing changes to address them.

Anything you guys or anyone else is able to confirm please chime in as all of this is extremely interesting and seems fairly widespread.

Thanks!

[emilpriver]

Many thanks! I will try with Puppeteer to see if we can re-produce it. This problem is like for a nail in a haystack. 0 clue where too look haha :D

[emilpriver]

what seems to be working is to block static/chunks in robots.txt. Somehow after we added this files to robots.txt does it seem to be working

this is now our robots.txt

User-Agent: *
Disallow: *?reviews=open
Disallow: *?unavailable=
Disallow: /_next/static/chunks/app/

[ayazwx]

Hello, there is not much activity. Did you solve the "Soft 404" problem?

[aha-kid]

Can't say anything yet, but

"Disallow: /_next/static/chunks/app/" used to work with my code. After I added it, only redirect link were threat as soft404. (Finger-cross, I didn't give it more time to prove)

Then I went back to here and looking if there are another method. Then someone also mention to use disallow but Barry Hunter which seems to be google dev said that it is not really appropriate method.

So I was looking for the answer until Paw-leen (vercel staff) comment to use graceful boundary here so I give it a try by adding wrap in the layout.tsx after section. I know that you already added Global Error Boundary but "GracefullyDegradingErrorBoundary" seems to be different. It return the whole html instead of "Application error: a client-side exception has occurred (see browser console)".

↓
https://nextjs.org/docs/app/api-reference/file-conventions/error#graceful-error-recovery-with-a-custom-error-boundary

So I give myself a try by adding both "Disallow: /_next/static/chunks/app/", and "GracefullyDegradingErrorBoundary" and waiting for search console report and it doesn't look good. Adding both make google bot cannot see some part of my page that run in js like swiper component, So I decided to remove disallow and keep Graceful last Saturday and waiting for a new search console report.

Really hope this will fix and Google not to change to a new strategy again.
(My web average clicked went down from 400 to 80 per day 🥲)

[errorproofdev]

Can't say anything yet, but

"Disallow: /_next/static/chunks/app/" used to work with my code. After I added it, only redirect link were threat as soft404. (Finger-cross, I didn't give it more time to prove)

Then I went back to here and looking if there are another method. Then someone also mention to use disallow but Barry Hunter which seems to be google dev said that it is not really appropriate method.

So I was looking for the answer until Paw-leen (vercel staff) comment to use graceful boundary here so I give it a try by adding wrap in the layout.tsx after section. I know that you already added Global Error Boundary but "GracefullyDegradingErrorBoundary" seems to be different. It return the whole html instead of "Application error: a client-side exception has occurred (see browser console)".

↓ https://nextjs.org/docs/app/api-reference/file-conventions/error#graceful-error-recovery-with-a-custom-error-boundary

So I give myself a try by adding both "Disallow: /_next/static/chunks/app/", and "GracefullyDegradingErrorBoundary" and waiting for search console report and it doesn't look good. Adding both make google bot cannot see some part of my page that run in js like swiper component, So I decided to remove disallow and keep Graceful last Saturday and waiting for a new search console report.

Really hope this will fix and Google not to change to a new strategy again. (My web average clicked went down from 400 to 80 per day 🥲)

What you did is what seemed to fix it for us. Leave the graceful error boundary and never disallow chunks as it results in unexpected errors when Google tries to render.

Leave it for some time.. we’ve noticed Google seems slower at processing requests etc.

You can always request revalidation ( just don’t spam them).

good luck!

[aha-kid]

Thank you! I learned many things from you though.

[emilpriver]

We're doing the exact same. No new soft 404 since 2 may and more pages are indexed everyday. Vercel CTO told us that Disallowing /_next/static/chunks/app/ is not good and I belive him, but this seems to be the only think which worked. Disallowing chunks and graceful shutdown seems to be working for us

[assa1605]

Hi, our team is also having trouble with soft404.

We are aware that GracefullyDegradingErrorBoundary is supposed to render correctly when a rendering error occurs. Therefore, even if we place error.tsx, GracefullyDegradingErrorBoundary catches the client error and displays the screen before the error, so the user will not know what is happening. Our team is hesitant to deploy GracefullyDegradingErrorBoundary because it changes the error behavior.

I know it has been some time since then, but has sof404 been resolved?
I would like to know if there has been any progress.😖

[Oummas]

Hey everyone, I ran into the same issue starting with Next.js 15.2. After some investigation, I came across a configuration option that disables metadata streaming in generateMetadata. Since applying it, we’ve observed a noticeable reduction in soft 404s, likely due to more consistent SSR-rendered metadata being available at crawl time.