🐛 [cloud_firestore] After some time, Cloud Firestore sync stops working #6749
Labels
impact: crowd
Affects many people, though not necessarily a specific customer with an assigned label. (P2)
plugin: cloud_firestore
resolution: fixed
A fix has been merged or is pending merge from a PR.
type: bug
Something isn't working
Comments
wujek-srujek
added
Needs Attention
|
Hi @wujek-srujek |
markusaksli-nc
added
blocked: customer-response
|
Hi @markusaksli-nc No, I can't because there isn't anything special what I'm doing. Here is the code for listening: FirebaseFirestore.instance.collection('items').snapshots().map(
(querySnapshot) => querySnapshot.docs.map(_mapDocumentSnapshot),
);
Future<void> main() async {
await runZonedGuarded<Future<void>>(() async {
WidgetsFlutterBinding.ensureInitialized();
await SystemChrome.setPreferredOrientations([
DeviceOrientation.portraitUp,
DeviceOrientation.portraitDown,
]);
await Firebase.initializeApp();
if (kDebugMode) {
// Report crashes in release mode only.
await FirebaseCrashlytics.instance.setCrashlyticsCollectionEnabled(false);
}
FlutterError.onError = FirebaseCrashlytics.instance.recordFlutterError;
runApp(MyApp());
}, (error, stack) => FirebaseCrashlytics.instance.recordError(error, stack));
}Pretty standard code, I would say. As I mentioned before, it all works fine for some time after starting the app, and before my issues started, it had worked fine for months no matter how much time the app was open. |
google-oss-bot
added
Needs Attention
|
Roughly how long does it take for it to stop working? Are there no errors/warnings in the logs? You're saying this reproduces consistently with multiple devices? |
markusaksli-nc
added
blocked: customer-response
|
I have just tested this using the iOS simulator (debug mode) and simultaneously on an Android device (release mode). Here are the observations:
The good thing is I can reproduce the issue in debug mode using the simulator, which should make troubleshooting easier. The bad thing is: I'm at a loss how I can even debug this, there are no logs at all and I see nothing wrong - no connection errors, no timeouts, nothing. Is there some kind of 'debug' or 'verbose' mode I could turn on? You will have noticed from my snippets above that I'm also using Crashlytics - which doesn't show any issues either. I will try to narrow the time window down by decreasing the time after which I perform the sync test, maybe 1.5 h is not necessary. |
google-oss-bot
added
Needs Attention
|
I wasn't able to reproduce this but I'll label it based on the report. The android side sounds familiar to #4305. What android device did you test this with? |
markusaksli-nc
added
plugin: cloud_firestore
and removed
Needs Attention
|
Pixel 3XL. I also have an iPhone XS device whose behavior is 100% consistent, i.e. also stops syncing after some time. |
|
The issue seems to be very similar, it might be the same one - the next time I can reproduce the issue I will wait for a few minutes to see if the data is synced with a delay, or never. So far, I never waited so long and simply assumed the data will never arrive. Will comment when I know more. However, I have the same issue on iOS... |
|
Seems to be a different issue for me - once sync stopped working (changing in web console didn't get propagated to the app on Android) I waited for 5 minutes with a timer, but the update never came. Turning airplane mode off and on doesn't help in my case either. |
|
Just to chime in: We're getting similar reports from our users. Seems to be handheld devices primarily, which often goes into standby. |
|
Hi, it's been a while, are there any news on this issue? The problem still occurs, 100% reproducible when the phones are left for about an hour without doing anything (get locked etc.). This basically prevents my app from being released. |
|
I can't really offer an update. Since I wasn't able to reproduce it and we're not seeing many reports of this (well #4305 sounded similar I suppose) it's hard to immediately fix anything here without some digging. I can try bringing this up to see if anyone has some ideas. |
|
I'll help creating a repro when I have time. I don't know if it happens on iOS, but it definitely happens on Android. The easiest way is to run it in debug mode via Wifi:
If you have the firestore console open, you should see nothing happen there yet. After a while it will show up, which could be a few minutes, or never (the "never" is harder to reproduce, but it does also happens for our clients). A bit of info: Could also be related to #6736 which outputs similar errors. #6736 is being investigated in the Android repo here firebase/firebase-android-sdk#2877 but there hasn't been any real traction yet. |
|
I redid the steps and I'm seeing a lot of log entries like this one: This happens to me even before I wake up the device, i.e. it happens as soon as it Wifi is shut down - my device is still in standby mode but my console already shows a lot of them. I'll update this comment once I test what happens when I wake up the device (~1.5 hrs from now). Update: upon waking the device I immediately got many exceptions like the one above. When I tried to add a document I got even more of the same, and the document didn't show up in Firestore. After waiting a few minutes I started getting: I'm testing with my real app which does require authentication, but I was logged in and adding the document worked initially. After some time I killed the app and started it again, and added a few documents and they showed up in Firestore Web Console immediately, without any auth errors - which means the auth part works fine, right? So yeah, I can confirm the steps by @larssn work to reproduce the issue, but in my case I got even more errors (as auth is involved). |
Yeah sounds like it. Auth should be completely transparent once logged in. I have a feeling the error might be really far downstream; maybe the problem is all the way down in the GRPC layer, since Think we need more eyes on this one. @markusaksli-nc Thoughts? Ps. I should stress that this issue could be considered critical, since you could easily (on some devices) end up in situations where data is never sent to firestore, which could cause havoc if that data is mission critical. |
|
@larssn Are you on the latest version of the plugin? The behavior you described sounds a lot like something that should have been fixed in a previous firebase release which should be picked up in the latest |
|
Out of curiosity, does anybody facing the issue also face it on iOS, like I do? There has been a lot of talk about Android-specific bugs in the native SDK, but I have the feeling the issue might be somewhere else: infrastructure or in the Dart code, since it is 100% consistent and reproducible on both platforms for me. |
|
@wujek-srujek Most of our users are using Android and therefore the reports we have are Android related. |
|
It's been 2 months since the original filing of the issue, any updates? |
|
I also have been facing this issue for the very long time(#4305). I really wished this could be solved, but I am pretty sure that the problem still exists and it is critical. At least, I hope there is an API as an alternative for checking firestore daemon health status and restarting it without restarting app. So the company can handle it from the perspective of user experience design. |
|
The issue still persists. Any news? |
|
Here's a reproduction of the issue in Flutter: link I wrote to firebase support, but they didn't wanna help without a reproduction in native Android/iOS, so I'm gonna make one in Android at the least. |
|
Here's a reproduction in pure Android: link I CANNOT reproduce the issue in pure Android, but I'd like some more eyes on it. Basically the instructions from #6749 (comment) still applies, with the only difference being that you run it from Android Studio instead. You can check the logcat for what is outputted. I output all info at the error-level, so it's easier to spot. The thing to look for is the difference in the local timestamp, and the server timestamp, which is what I log. |
|
@larssn I will try to do it this week, but it's not high on my priority list I'm afraid, I have long given up on Firestore. |
|
With the projects that @larssn have made it looks like the issue is related to FlutterFire only and not Firestore. Do we all need to give up or can you @markusaksli-nc please pull some strings and give this critical issue some attention? |
|
Not sure if I should make a new issue since the topic has diverged from "stops working", to "coming out of standby, firestore is very slow to start sync'ing data". And again to stress: This only seems to happen with the Flutter version for yet undiscernible reasons. @markusaksli-nc Thoughts? |
|
I haven't been on FlutterFire issue triage for a while so I'm not sure what the status here is, from what I remember the team should be aware of this issue. @larssn what do you mean by "only seems to happen with the Flutter version"? |
|
@markusaksli-nc Thanks for replying! Just to bring you up to speed:
This is in stark contrast to the Flutter version where it takes one whole minute for data to arrive. The two POCs are very simple and pretty much identical in implementation. But in case I'm missing something here, I'd like some more eyes on it. |
|
Found this: https://groups.google.com/g/firebase-talk/c/9D9rXvhBmUM/m/wFBVapeGAwAJ Mike from 4 years ago is describing the issue exactly... 😞 Wonder why my Android POC didn't show the problem then. |
darshankawar
added
the
impact: crowd
|
I'll make the team aware about this issue. |
Follow-up researchI've been doing some research on this issue, and here's what I've found: It seems this issue is related to Android's "Doze mode", which allows apps to go into various levels of sleep. Here's my observations/log from using the flutter reproduction app and playing around with Doze mode. The device is unplugged from charging, and the screen is turned off. Things start to go wrong when GRPC reports a I turn on the screen on this line: Notice that after this line, the SDK attempts to write to the backend. This fails, and it starts doing exponential backoff which makes things worse in this case. Also notice all the DNS resolution errors that happens in spite of having internet access again. When the line All in all, it can take minutes to get to a working state again. Full logTurning off doze modeYou can use the command To turn doze mode back on, use the command There's a section on exemptions from doze mode, which might be a valid workaround for some people, which can be found here: https://developer.android.com/training/monitoring-device-state/doze-standby SummarySo this seems to be an unhealthy cocktail of Flutter+gRPC+Doze that causes this issue. I'm not convinced there isn't an issue with gRPC, since we're seeing those DNS resolution errors when coming out of sleep, which might compound the problem. I'm not sure where to begin fixing this, but I hope some of these observations are helpful and can give a clue to what is happening. Other issuesThis is one of those issues that has been reported many times, fixed many times, but apparently not good enough yet. Here's some other issues that all point to the same thing: SO: Other: |
|
This PR upstream is of interest IMO: grpc/grpc-java#8855 |
|
@larssn Sorry for taking so much time with it. I can't reproduce the issue on native Android either, still have issues on Flutter, though, no change here. |
|
@darshankawar This is fixed in Firebase BoM 29.2.0 which contains GRPC 1.44.1 |
|
You can try the version if you're impatient by adding the following to android/app/build.gradle: |
10 tasks
|
I updated to everything latest (Flutter 2.10.4, all the dependencies including FlutterFire etc., forced FirebaseSDKVersion to 29.3.0 on Android and to 8.14.0 (as described here: https://firebase.flutter.dev/docs/manual-installation/#overriding-firebase-sdk-versions) and verified that these are indeed the versions used) and the problem still occurs for my app, both on Android and on iOS. Behavior:
I have no idea what is going on, but the summary is:
This issue should not be closed. |
darshankawar
added
the
resolution: fixed
I'd try to make a simple reproduction of it: Thats what I did above, which convinced me that the problem was not a bug in my end. 🙂 |
|
What I did instead is I created a completely new Firebase project (I wanted to change the packageName and bundleId anyway), regenerated the whole project (i.e. the android, ios etc. folders) with Flutter 2.10.4, updated to everything latest etc. Basically I redid everything around the code and assets, and now sync works fine. When I test with the old Firebase project and older Flutter (but latest package versions), it doesn't work on either platform. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Bug report
Both on Android and iOS, the sync works fine when the application starts and keeps working for some time. However, after some time (not sure how much) sync stops working - sending doesn't work, and receiving in another instance of the app doesn't receive changes (I can see this by changing data directly in the console). When the application is force-stopped and restarted, it starts working again immediately. This behavior is 100% consistent between Android and iOS.
I started having the issue only recently, maybe 2-3 weeks ago. Before this, this application and Firestore integration had been working flawlessly for months and there have been no code changes at all since May 19, and no code changes to anything related to Firebase or Firestore for half a year (except for version updates).
Has anything changed in your backend, and is some change in code required for it to start working without issues again?
I upgraded everything about a week ago but it didn't help.
Flutter doctor
Run
flutter doctorand paste the output below:Click To Expand
Flutter dependencies
Run
flutter pub deps -- --style=compactand paste the output below:Click To Expand
The text was updated successfully, but these errors were encountered: