Upload often fails or gets stuck, especially when uploading 3 or more pictures

[nicolas-raoul]

Steps to reproduce:

1. Take 3 (or more) interesting pictures of at least 2 MB (for instance pictures of cars or houses. If you need I can also provide many pictures, just send an email to the address found here).
2. Get a slow Internet connection by installing the Throttly app and setting it to Good 2G 450kbps↓ 150kbps↑.
3. In your gallery app, select the 3 (or more) pictures, tap Share, then select the Commons app.
4. As soon as the upload starts, turn off your screen and do not touch your phone for 5 minutes.
5. Some of the pictures will probably fail to upload.

Many other apps always upload reliably (for instance Gmail with attachments) even with slow Internet connections, even the Commons browser-based uploader seems to perform better, so hopefully we can fix this.

Possible strategy:

1. Reproduce the issue with the Android Studio debugger (or similar), find out what the exact error message is.
2. Based on the error message, change the app's network parameters, such as timeout or buffer sizes.
3. If still not reliable, implement a retry loop (for instance try again 3 times).

Measuring improvements:

• Percentage of failure with the reproduction steps above, as of 2023 January 12's master branch:
  • 1 failure out of a 3-pictures set upload on fast WiFi: 2 occurrences
  • 2 failures out of a 10-pictures set upload on fast WiFi: 1 occurrence
  • 6 failures out of a 9-pictures set upload on fast 5G: 1 occurrence
  • 0 failures out of a 2-pictures set upload on slow WiFi: 1 occurrence

Some uploads get stuck (no progress despite the progress bar showing an animation). Among those stuck, some file have actually been uploaded already, some not.

Aggravating factors:

• Brief network disconnections, for instance when entering an elevator for 1 minute, seem to make uploads less reliable. I have not measured this theory but this could be investigated by putting your phone in a metal box (use a ping chart app to verify)
• I had many successes uploading from the custom picker, and many failures uploading by sharing from the gallery. There might be something about the app going to the background, or something other about activities, that might have an effect on reliability.

Recent Google Play comments:

• "Not a single photo is uploading even though i tried a dozen times"
• "50% of uploads are failed"
• "fails to upload pictures without saying the reason"
• "I can't upload photos. They all say failed. Before it worked fine"
• "Photos cannot be uploaded"

[ayushanand308]

@nicolas-raoul This seems to be an interesting issue to solve. Willing to work on it.

[nicolas-raoul]

@ayushanand308 Are you planning to apply for Google Summer of Code, by any chance?

[ayushanand308]

@nicolas-raoul Yes I am!

[nicolas-raoul]

@ayushanand308 Then please the instructions at https://github.com/commons-app/commons-app-documentation/blob/master/android/Students.md, at the "Fix 1 bug" step please choose a different bug, because this one is among the bugs that make the GSoC's task itself :-) How about #5057 for instance? Thank you for your interest!

[Shantanub05]

@nicolas-raoul This seems to be an interesting issue to solve. Willing to work on it.

Hey Ayush, I would like to contribute to this issue along with you for GSOC 2023.....Can we connect?

[sivasubramaniamv]

This error occurs when the app sends the "Failed to Upload" notification.

The error is thrown at line 227 in UploadClient.java in the upload package

This happens when the condition uploadResult.getUpload() == null evaluates to true at line 223

Here uploadResult is of type UploadResponse

So uploadResult.getUpload() returns the nullable property upload of type UploadResult which is a data class

At line 221 in UploadClient.java the value of a gson.fromJson() call is stored in the uploadResult field

So uploadResult.getUpload() == null indicates that the upload property was returned as null by gson after parsing a JSON response from the server, which indicates that the server likely returned an empty or null response,

As uploadResult.getUpload() == null is true, an exception of type MwException is thrown.

The getErrorCode() method in MwException class returns error.getCode() where error is of type MwServiceError

A top-level comment in MwServiceError.java says "Gson POJO for a MediaWiki API error"

The class MwServiceError extends ServiceError which likely represents an error reported by the server via the API. It says "The API reported an error in the payload."

All this happens when the uploadClient.uploadFileFromStash() method is called at line 296 in UploadWorker.kt

As the uploadClient.uploadFileFromStash() call causes an exception the onErrorReturn { } block is executed. A null value is passed here which causes the NullPointerException as shown in the Logcat screenshot attached first.

The exact error message is as below

The message Could not acquire locks on server rdb1 indicates that this is likely a problem with the server, rather than the Android client.

Two key methods are executed during the process in UploadWorker.kt:

1. uploadClient.uploadFileToStash() and
2. uploadClient.uploadFileFromStash()

while uploadFileToStash() turns out to be executed without a problem, uploadFileFromStash() causes the exception.

So what is Stash?

I'm still unsure what it is. But it is clear that it's something the server handles. Refer https://www.mediawiki.org/wiki/Manual:UploadStash

I tried uploading multiple times and ended up with the exact same error.

@nicolas-raoul here are my findings about this issue. Discussing with the maintainers of Wikimedia Android Data Client and the API developers would be very helpful in solving this issue.

[nicolas-raoul]

Stashing is a way to first send the picture's JPEG bytes to the server, then finalize the "upload" (file registration) a bit later.

[sivasubramaniamv]

Steps to reproduce the error:

1. Select a few images and start upload
2. A notification alert with the message "Uploading file: file-name" is received
3. After the upload progress bar reaches the end, a notification alert with the message "Failed to upload file-name" is received
4. The following error is received

@nicolas-raoul Can you verify if the behavior is the same for you?

[nicolas-raoul]

@siva-subramaniam-v Does step 3 happen for you every time after steps 1 and 2? For me, after steps 1 and 2, step 3 sometimes happens.

[sivasubramaniamv]

When I tested yesterday after step 2 the notification message changed from "Uploading file: file-name" to "Finishing uploading file-name" and then to step 3 ("Failed to upload file-name"). But this sequence happens consistently for me and leads to the same error.

What else happens for you after step 2 other than step 3?

[nicolas-raoul]

@siva-subramaniam-v For me after step 2 upload is usually successful. If I understand correctly, multi-upload never works for you? If yes, would you mind creating a new GitHub issue? Thank you! 🙂

[nicolas-raoul]

On latest master, upload often gets stuck when uploading a set of around 10 pictures, this happened today on a very fast WiFi at home, not moving, but switching display off, on Pixel 6:

I can send the logcat if necessary.

[nicolas-raoul]

I just got upload stuck but uploading several medium-size sets (10 sets of 2 to 10 pictures).

[RitikaPahwa4444]

Hi @nicolas-raoul, could you please share the logs too (with the device unplugged)?

[nicolas-raoul]

Solved by Ritika. :-)