🐛 [Cloud_Firestore] Multiple crashes on 3.1.7: Fatal Exception: java.lang.RuntimeException Internal error in Cloud Firestore (24.0.1).

[KubaStachecki]

Bug report

Describe the bug

Since firebase 3.1.7 app crashes in production a lot. I can't replicate this on my side, but I got multiple reports and stability % is falling like a knife. Reports from crashlytics are not pointing in anything specific in my code, hence I'm not sure what to sample.

Sample report:

DelayedClientCall.java line 447
io.grpc.internal.DelayedClientCall$DelayedListener.onMessage

Fatal Exception: java.lang.RuntimeException
Internal error in Cloud Firestore (24.0.1).

com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3 (AsyncQueue.java:539)
com.google.firebase.firestore.util.-$$Lambda$AsyncQueue$jx84dqgUsF4ojecSMurRqFLFD1Y.run (Unknown Source:2)
android.os.Handler.handleCallback (Handler.java:938)
android.os.Handler.dispatchMessage (Handler.java:99)
android.os.Looper.loop (Looper.java:233)
android.app.ActivityThread.main (ActivityThread.java:8030)
java.lang.reflect.Method.invoke (Method.java)
com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:631)
com.android.internal.os.ZygoteInit.main (ZygoteInit.java:978)

Caused by java.lang.NullPointerException
Attempt to invoke virtual method 'com.google.firebase.firestore.model.DocumentKey com.google.firebase.firestore.model.MutableDocument.getKey()' on a null object reference

com.google.firebase.firestore.model.mutation.MutationBatch.applyToLocalView (MutationBatch.java:125)
com.google.firebase.firestore.local.LocalDocumentsView.recalculateAndSaveOverlays (LocalDocumentsView.java:164)
com.google.firebase.firestore.local.LocalDocumentsView.recalculateAndSaveOverlays (LocalDocumentsView.java:195)
com.google.firebase.firestore.local.LocalStore.lambda$acknowledgeBatch$3$LocalStore (LocalStore.java:310)
com.google.firebase.firestore.local.-$$Lambda$LocalStore$lVdtE1Q-smZ5G8FkMVocZAwo0B4.get (Unknown Source:4)
com.google.firebase.firestore.local.SQLitePersistence.runTransaction (SQLitePersistence.java:228)
com.google.firebase.firestore.local.LocalStore.acknowledgeBatch (LocalStore.java:301)
com.google.firebase.firestore.core.SyncEngine.handleSuccessfulWrite (SyncEngine.java:440)
com.google.firebase.firestore.core.MemoryComponentProvider$RemoteStoreCallback.handleSuccessfulWrite (MemoryComponentProvider.java:109)
com.google.firebase.firestore.remote.RemoteStore.handleWriteStreamMutationResults (RemoteStore.java:666)
com.google.firebase.firestore.remote.RemoteStore.access$500 (RemoteStore.java:53)
com.google.firebase.firestore.remote.RemoteStore$2.onWriteResponse (RemoteStore.java:201)
com.google.firebase.firestore.remote.WriteStream.onNext (WriteStream.java:185)
com.google.firebase.firestore.remote.WriteStream.onNext (WriteStream.java:49)
com.google.firebase.firestore.remote.AbstractStream$StreamObserver.lambda$onNext$1$AbstractStream$StreamObserver (AbstractStream.java:119)
com.google.firebase.firestore.remote.-$$Lambda$AbstractStream$StreamObserver$IZBnAkL3D_hQ5ofbDpSmX2lhuPo.run (Unknown Source:4)
com.google.firebase.firestore.remote.AbstractStream$CloseGuardedRunner.run (AbstractStream.java:67)
com.google.firebase.firestore.remote.AbstractStream$StreamObserver.onNext (AbstractStream.java:110)
com.google.firebase.firestore.remote.FirestoreChannel$1.onMessage (FirestoreChannel.java:131)
io.grpc.internal.DelayedClientCall$DelayedListener.onMessage (DelayedClientCall.java:447)
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal (ClientCallImpl.java:656)
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext (ClientCallImpl.java:641)
io.grpc.internal.ContextRunnable.run (ContextRunnable.java:37)
io.grpc.internal.SerializingExecutor.run (SerializingExecutor.java:133)
java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:462)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run (ScheduledThreadPoolExecutor.java:301)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run (AsyncQueue.java:234)
java.lang.Thread.run (Thread.java:923)

Steps to reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. '...'
3. See error or incorrect behavior

Expected behavior

A clear and concise description of what you expected to happen.

Sample project

Providing a minimal example project which demonstrates the bug in isolation from your main App greatly enhances the chance of a timely fix.
Please link to the public repository URL.

---

Additional context

Add any other context about the problem here.

---

Flutter doctor

Run flutter doctor and paste the output below:

/Users/kuba/flutter/bin/flutter doctor --verbose
[✓] Flutter (Channel stable, 2.8.1, on macOS 11.5.2 20G95 darwin-x64, locale en-GB)
• Flutter version 2.8.1 at /Users/kuba/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 77d935af4d (7 weeks ago), 2021-12-16 08:37:33 -0800
• Engine revision 890a5fca2e
• Dart version 2.15.1

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
• Android SDK at /Users/kuba/Library/Android/sdk
• Platform android-31, build-tools 29.0.3
• ANDROID_HOME = /Users/kuba/Library/Android/sdk
• Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
• All Android licenses accepted.

Click To Expand

PASTE OUTPUT INSIDE HERE

---

Flutter dependencies

Run flutter pub deps -- --style=compact and paste the output below:

firebase_core: ^1.12.0 firebase_auth: ^3.3.6 firebase_analytics: ^9.1.0 firebase_dynamic_links: ^4.0.5 cloud_firestore: ^3.1.7 firebase_database: ^9.0.6 firebase_storage: ^10.2.6 firebase_messaging: ^11.2.6 firebase_crashlytics: ^2.5.0 firebase_app_check: ^0.0.6+5 firebase_remote_config: ^2.0.0

Click To Expand

PASTE OUTPUT INSIDE HERE

---

[darshankawar]

@KubaStachecki
Can you try below and see if it helps ?

buildTypes {
       release {
           minifyEnabled false
           shrinkResources false

[darshankawar]

Similar / related: #7985

[KubaStachecki]

Hi! Yes, both settings are already in the code:

`buildTypes {
        release {
            signingConfig signingConfigs.release
            shrinkResources false
            minifyEnabled false
            ndk {
                debugSymbolLevel 'SYMBOL_TABLE'
            }
        }
        debug {
            signingConfig signingConfigs.release
            shrinkResources false
            minifyEnabled false
        }
    }`

[bswhite1]

We also witnessed extreme crash repetition, frequency and growth with cloud_firestore release 3.1.7. We immediately rolled back to cloud_firestore 3.1.6 and firebase_core 1.11.0. We witnessed 6 new crash types (including the one mentioned above by @KubaStachecki). We can provide trace stacks of these, if desired. We could not produce these errors in debug / during manual QA prior to production roll-out (Unfortunately we aren't using Firebase Test Labs at this time), and as a result, discovered these crashes when customers starting calling, and alarms ringing. Reverting to prior release of firestore and firebase_core removed all 6 crash types from our crashalytics, and we have been running crash free (Aside from the TransactionStreamHandler crash fixed in firestore 3.1.7) for 3 days now on the reverted release. The only thing changed between the trouble crash production release and the reverted release were library increments on cloud_firestore and firebase_core. Every one of the new 6 crashes have to do with a blame frame matching "io.grpc.internal.***". We witnessed these crashes in production primarily on Android 11 and the Motorola G Fast, though other phones experienced the crashes as well. We can provide additional details if desired.

[darshankawar]

Looking at the crash log, it seems to be related to native Android sdk. See below issues from that repo for your reference:

firebase/firebase-android-sdk#382
firebase/firebase-android-sdk#875

But I'll keep it open for further insights from the team.

/cc @Salakar @russellwheatley

[darshankawar]

Similar crash report from cloud_firestore 3.1.7: #7985

[russellwheatley]

Hey @KubaStachecki, the cloud_firestore 3.1.7 version upgrades the Firebase BOM to 29.0.4 which also updates the Firestore package from 24.0.0 to 24.0.1.

The other change made in cloud_firestore plugin was to the Transaction handler which involved using the Looper class (see diff).

The stack trace leads me to believe the firebase android sdk might have a bug (or requires further gradle config). Not sure if you're able to override the Firebase BOM to 29.0.4 in a non-prod environment whilst using cloud_firestore version 3.1.6 and recreate the bug?

[russellwheatley]

Hmmm, actually, I do see the Firestore transaction handler invoked in this line:

com.google.firebase.firestore.local.SQLitePersistence.runTransaction (SQLitePersistence.java:228)

So it probably is the change to the transaction handler 🤔.

[KubaStachecki]

Unfortunately I can't reproduce this crash locally so far, so I can't test this 29.0.4 - 3.1.6 scenario, but I can confirm that crashes disappeared after returning dependencies to firebase_core: 1.11.0 and cloud_firestore: 3.1.6 - thank you @bswhite1 !

[russellwheatley]

Yes, I think it is probably the transaction update causing the problem so I will get this remedied in the next release 👍

[pranavo72bex]

i am facing the same issue in release mode. I know we can resolve this problem by 'shrinkResources false minifyEnabled false'
but I want to minify my app. how can I resolve this issue?

[JT501]

I got the same problem in development environment using cloud_firestore 3.1.7 and firebase_core 1.12.0, it causes my app crash every time I open it.

Workaround:
Rollback to cloud_firestore 3.1.6 and firebase_core 1.11.0

Hope it will be fixed soon.

[pranavo72bex]

I got the same problem in development environment using cloud_firestore 3.1.7 and firebase_core 1.12.0, it causes my app crash every time I open it.

Workaround: Rollback to cloud_firestore 3.1.6 and firebase_core 1.11.0

Hope it will be fixed soon.

In my case it is not working even with same version of cloud_firestore 3.1.6 and firebase_core 1.11.0 with minifyEnable true ...hope it will fix soon ..minifyEnable false is not even proper solution

[bswhite1]

Could you rollback the TransactionStreamHandler fix from 3.1.7 until this is resolved? That way we can get the other updates available.

[russellwheatley]

After internal conversations and reviewing the stack trace, we're not convinced this problem originates from FlutterFire. @mk-dev-1 & @pranavo72bex have confirmed the rollback did not make a difference for them. We also note that the stack trace is made almost entirely from the native android SDK and its dependencies. The current implementation also solves a previous transaction bug.

for these reasons, we're holding back on any implementation reversal. If someone could provide a reproducible sample, I'd be very grateful. That way we'd be able to diagnose the problem and get it solved.

[pranavo72bex]

After internal conversations and reviewing the stack trace, we're not convinced this problem originates from FlutterFire. @mk-dev-1 & @pranavo72bex have confirmed the rollback did not make a difference for them. We also note that the stack trace is made almost entirely from the native android SDK and its dependencies. The current implementation also solves a previous transaction bug.

for these reasons, we're holding back on any implementation reversal. If someone could provide a reproducible sample, I'd be very grateful. That way we'd be able to diagnose the problem and get it solved.

DID you try to test the firebase package with minifyEnabled true??

`buildTypes {
        release {
            signingConfig signingConfigs.release
            **shrinkResources true
            minifyEnabled true**
        }
        debug {
            signingConfig signingConfigs.release
            **shrinkResources true**
            **minifyEnabled true**
        }
    }`

[mk-dev-1]

After internal conversations and reviewing the stack trace, we're not convinced this problem originates from FlutterFire. @mk-dev-1 & @pranavo72bex have confirmed the rollback did not make a difference for them. We also note that the stack trace is made almost entirely from the native android SDK and its dependencies. The current implementation also solves a previous transaction bug.
for these reasons, we're holding back on any implementation reversal. If someone could provide a reproducible sample, I'd be very grateful. That way we'd be able to diagnose the problem and get it solved.

DID you try to test the firebase package with minifyEnabled true??

`buildTypes {
        release {
            signingConfig signingConfigs.release
            **shrinkResources true
            minifyEnabled true**
        }
        debug {
            signingConfig signingConfigs.release
            **shrinkResources true**
            **minifyEnabled true**
        }
    }`

Yes, tried both: shrinkResources & minifyEnabled on and off. It if is off, everything's working perfectly and it is a proposed solution for this error in other threads. However, I would not consider this a real solution....

There is a related thread here: firebase/firebase-android-sdk#3490

[russellwheatley]

Great work @mk-dev-1, so it appears this is a bug at the level of the firebase-android-sdk and it has been resolved with this merge: firebase/firebase-android-sdk#3495.

I'll update this channel once this has been released on the android sdk and subsequently on FlutterFire.

[mk-dev-1]

Thanks @russellwheatley - any ideas on when that update will be available?

[russellwheatley]

Not sure. Last firebase android update was Feb 20th. So it is due a release soon. Then FlutterFire needs to be released with firebase android sdk bump.

[google-oss-bot]

Hey @KubaStachecki. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[mk-dev-1]

Version 24.0.2 was released a few hours ago - I have overwritten the depencency like this

    implementation 'com.google.firebase:firebase-firestore:24.0.2'

in my app level build.gradle.

It is showing in the dependency tree like this:

|    |    |    +--- com.google.firebase:firebase-firestore:24.0.1 -> 24.0.2 (c)
[...]
|    +--- com.google.firebase:firebase-firestore -> 24.0.2
[...]
+--- com.google.firebase:firebase-firestore:24.0.2 (*)

Unfortunately, the crash remains:

E/AndroidRuntime(25933): FATAL EXCEPTION: main
E/AndroidRuntime(25933): java.lang.RuntimeException: Internal error in Cloud Firestore (24.0.2).
E/AndroidRuntime(25933):        at com.google.firebase.firestore.e0.h.run(:1)
E/AndroidRuntime(25933):        at android.os.Handler.handleCallback(Handler.java:938)
E/AndroidRuntime(25933):        at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime(25933):        at android.os.Looper.loopOnce(Looper.java:226)
E/AndroidRuntime(25933):        at android.os.Looper.loop(Looper.java:313)
E/AndroidRuntime(25933):        at android.app.ActivityThread.main(ActivityThread.java:8633)
E/AndroidRuntime(25933):        at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime(25933):        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567)
E/AndroidRuntime(25933):        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1133)
E/AndroidRuntime(25933): Caused by: e.e.b.c.i.j: java.lang.RuntimeException: java.lang.ExceptionInInitializerError
E/AndroidRuntime(25933):        at e.e.b.c.i.O.n(:3)
E/AndroidRuntime(25933):        at com.google.firebase.firestore.d0.K.c(:1)
E/AndroidRuntime(25933):        at com.google.firebase.firestore.d0.n.a(Unknown Source:6)
E/AndroidRuntime(25933):        at e.e.b.c.i.B.run(Unknown Source:21)
E/AndroidRuntime(25933):        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
E/AndroidRuntime(25933):        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E/AndroidRuntime(25933):        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
E/AndroidRuntime(25933):        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
E/AndroidRuntime(25933):        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
E/AndroidRuntime(25933):        at com.google.firebase.firestore.e0.s$c$b.run(Unknown Source:15)
E/AndroidRuntime(25933):        at java.lang.Thread.run(Thread.java:920)
E/AndroidRuntime(25933): Caused by: java.lang.RuntimeException: java.lang.ExceptionInInitializerError
E/AndroidRuntime(25933):        at e.e.b.c.i.P.run(Unknown Source:17)
E/AndroidRuntime(25933):        at com.google.firebase.firestore.e0.C.a(:1)
E/AndroidRuntime(25933):        at com.google.firebase.firestore.e0.m.run(Unknown Source:4)
E/AndroidRuntime(25933):        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
E/AndroidRuntime(25933):        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
E/AndroidRuntime(25933):        ... 1 more
E/AndroidRuntime(25933): Caused by: java.lang.ExceptionInInitializerError
E/AndroidRuntime(25933):        at h.b.n0.d.forTarget(Unknown Source:0)
E/AndroidRuntime(25933):        at h.b.n0.e.a(:1)
E/AndroidRuntime(25933):        at h.b.N.b(:2)
E/AndroidRuntime(25933):        at com.google.firebase.firestore.d0.N.e(:2)
E/AndroidRuntime(25933):        at com.google.firebase.firestore.d0.r.call(Unknown Source:2)
E/AndroidRuntime(25933):        at e.e.b.c.i.P.run(Unknown Source:4)
E/AndroidRuntime(25933):        ... 5 more
E/AndroidRuntime(25933): Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: h.b.k0.values []
E/AndroidRuntime(25933):        at java.lang.Enum.enumValues(Enum.java:270)
E/AndroidRuntime(25933):        at java.lang.Enum.access$000(Enum.java:61)
E/AndroidRuntime(25933):        at java.lang.Enum$1.create(Enum.java:277)
E/AndroidRuntime(25933):        at java.lang.Enum$1.create(Enum.java:275)
E/AndroidRuntime(25933):        at libcore.util.BasicLruCache.get(BasicLruCache.java:63)
E/AndroidRuntime(25933):        at java.lang.Enum.getSharedConstants(Enum.java:289)
E/AndroidRuntime(25933):        at java.lang.Class.getEnumConstantsShared(Class.java:2459)
E/AndroidRuntime(25933):        at java.util.EnumSet.getUniverse(EnumSet.java:407)
E/AndroidRuntime(25933):        at java.util.EnumSet.noneOf(EnumSet.java:109)
E/AndroidRuntime(25933):        at java.util.EnumSet.of(EnumSet.java:235)
E/AndroidRuntime(25933):        at h.b.n0.d.<clinit>(Unknown Source:89)
E/AndroidRuntime(25933):        ... 11 more
E/AndroidRuntime(25933): Caused by: java.lang.NoSuchMethodException: h.b.k0.values []
E/AndroidRuntime(25933):        at java.lang.Class.getMethod(Class.java:2103)
E/AndroidRuntime(25933):        at java.lang.Class.getDeclaredMethod(Class.java:2081)
E/AndroidRuntime(25933):        at java.lang.Enum.enumValues(Enum.java:267)
E/AndroidRuntime(25933):        ... 21 more

[bswhite1]

@mk-dev-1, it appears you need to create a separate issue. This issue, as defined in first post is to fix: "Attempt to invoke virtual method 'com.google.firebase.firestore.model.DocumentKey com.google.firebase.firestore.model.MutableDocument.getKey()' on a null object reference
"
That is what was suppose to be fixed in 24.0.2

Your issue appears to be around initialization. Also sounds like you have a locally reproducible example?

[mk-dev-1]

@bswhite1 Based on your feedback I have added a separate issue for this. Unfortunately, I yet have to find a way to reproduce the issue. Setting up a blank project with just firestore, a stream listener and release configuration does not crash the app. Let's continue this in the new issue then....

[pranavo72bex]

Did you enable minify in your blank project?

[luckyhandler]

Overriding the dependencies in my android's build.gradle solves the issue. So it's a native Android issue.
implementation 'com.google.firebase:firebase-firestore:24.0.2'
@mk-dev-1

Is there a new FLutterFire version in the pipeline for this fix?

[mk-dev-1]

@pranavo72bex Yes, I did ;-)

@luckyhandler: Does it fix the original issue raised by KubaStachecki, or does it fix the issue I raised a couple of posts earlier or the one in #8265?

[luckyhandler]

It fixes the original issue by @KubaStachecki with the

Attempt to invoke virtual method 'com.google.firebase.firestore.model.DocumentKey com.google.firebase.firestore.model.MutableDocument.getKey()' on a null object reference

stacktrace

[russellwheatley]

@luckyhandler, thank you for confirming the upgrade as a fix. I'll raise a PR and look to get a release out ASAP. Thank you.

[pranavo72bex]

@

Overriding the dependencies in my android's build.gradle solves the issue. So it's a native Android issue. implementation 'com.google.firebase:firebase-firestore:24.0.2' @mk-dev-1

Is there a new FLutterFire version in the pipeline for this fix?

This solve the issues in my case ..thanks man you are life saver 🤗

[mk-dev-1]

As for the issue I raised a couple of posts earlier - after finally identifying a working commit in my app's repository history, I was able to identify and resolve that issue, too. Replacing classpath 'com.android.tools.build:gradle:4.2.2' with classpath 'com.android.tools.build:gradle:4.1.0' seems to have done the trick....

Happy to close this issue & Thanks for everyone's support