Prototype Uploads are hanging

[jocelynk]

I am having an issue of being unable to use the loopback-component-storage to do uploads. It does not work with IBM COS or the file system. I did not create any custom remote method but am using the prototype upload method provided by the connector.

Error I am getting:

[1] Error: Request aborted
[1]     at IncomingMessage.<anonymous> (/Users/jocelynkong/Watson_Practice/Business-Solutions-Dev/document-management/node_modules/formidable/lib/incoming_form.js:120:19)
[1]     at emitNone (events.js:105:13)
[1]     at IncomingMessage.emit (events.js:207:7)
[1]     at abortIncoming (_http_server.js:410:9)
[1]     at socketOnClose (_http_server.js:404:3)
[1]     at emitOne (events.js:120:20)
[1]     at Socket.emit (events.js:210:7)
[1]     at TCP._handle.close [as _onclose] (net.js:547:12)
[1] Unhandled error for request POST /filestore/wca-jocelyn/upload: Error: Request aborted
[1]     at IncomingMessage.<anonymous> (/Users/jocelynkong/Watson_Practice/Business-Solutions-Dev/document-management/node_modules/formidable/lib/incoming_form.js:120:19)
[1]     at emitNone (events.js:105:13)
[1]     at IncomingMessage.emit (events.js:207:7)
[1]     at abortIncoming (_http_server.js:410:9)
[1]     at socketOnClose (_http_server.js:404:3)
[1]     at emitOne (events.js:120:20)
[1]     at Socket.emit (events.js:210:7)
[1]     at TCP._handle.close [as _onclose] (net.js:547:12)

Sample CURL response I am using:

curl -X POST \
  http://localhost:3000/api/filestore/wca-jocelyn/upload \
  -H 'Accept: multipart/form-data' \
  -H 'Cache-Control: no-cache' \
  -H 'Postman-Token: fb26baca-9d56-7d8c-5916-f66f403a2ea8' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F file=@/Users/jocelynkong/Watson_Practice/Business-Solutions-Dev/CNC/experimental.html

Sample project: https://github.ibm.com/Jocelyn-Kong/lb-debug

[dhmlau]

@b-admike , is it the same issue you've been helping?

[b-admike]

Yeah, I was debugging @jocelynk's app, but didn't find the cause of the issue. However, #196, among others, is very similar to this issue, and the sandbox app from there, https://github.com/PArns/loopback-stuck-test provides a working example.

[hacksparrow]

@jocelynk when you use the -F file=@ option with curl, it will add its own multipart/form-data boundary value. The code generated by Postman will add a duplicate boundary, which causes formidable to end the stream.

It is beyond our scope to decide and implement which boundary value to use while parsing forms.

To fix your CURL code just remove the duplicate -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \.

This is a case of improper setting of headers, so closing this.

[jocelynk]

@hacksparrow I removed the header and it still does not work. The request still hangs and I still get the same error:

 Error: Request aborted
[1]     at IncomingMessage.<anonymous> (/Users/jocelynkong/Watson_Practice/Business-Solutions-Dev/document-management/node_modules/formidable/lib/incoming_form.js:120:19)
[1]     at IncomingMessage.emit (events.js:127:13)
[1]     at IncomingMessage.emit (domain.js:421:20)
[1]     at abortIncoming (_http_server.js:441:9)
[1]     at socketOnClose (_http_server.js:435:3)
[1]     at Socket.emit (events.js:132:15)
[1]     at Socket.emit (domain.js:421:20)
[1]     at TCP._handle.close [as _onclose] (net.js:550:12)

I am using this curl command:

curl -X POST \
  http://localhost:3000/api/filestore/5a8c5028698e79249dfa7a78/upload \
  -H 'Accept: multipart/form-data' \
  -F file=@/Users/jocelynkong/Watson_Practice/Business-Solutions-Dev/CNC/template.html

I have given up using the loopback-component-storage connector, so you can leave this Github issue closed, but I think there is still a bug with uploading files using this connector.

[b-admike]

Re-opening because @jocelynk is still seeing the issue.

[hacksparrow]

@jocelynk remove that -H 'Accept: multipart/form-data' header. An average server does not return multipart/form-data, and an average client cannot process them. You probably wanted to set the Content-Type header instead of Accept.

If you want to set at all, use -H 'Accept: */*', which most HTTP clients do.

@b-admike this is a case of improper headers.

[jocelynk]

@hacksparrow This is not a case of improper headers. I have tried both of these curl commands and both return the same Request Aborted error.

curl -X POST \
  http://localhost:3000/api/filestore/5a8c5028698e79249dfa7a78/upload \
  -H 'Accept: */*' \
  -F file=@/Users/jocelynkong/Watson_Practice/Business-Solutions-Dev/CNC/template.html

and

curl -X POST \
  http://localhost:3000/api/filestore/5a8c5028698e79249dfa7a78/upload \
  -F file=@/Users/jocelynkong/Watson_Practice/Business-Solutions-Dev/CNC/template.html

The prototype upload method does not work. See this issue here that was never resolved and was marked as stale instead: #205

[hacksparrow]

@jocelynk thanks for the feedback, I have re-opened #205, and will take a look at it.

What loopback version are you using? Please mention the OS too. I am on a Mac with loopback 3.18.3 (latest). The equivalent curl commands posted by you works for me.

[jocelynk]

I am on a Mac using loopback 3.11.1.

[hacksparrow]

Can you update to 3.18.3 and see if you still face the issue, please?

[jocelynk]

I am still facing the same issue after the update. These are packages and versions I am using:

    "loopback": "^3.18.3",
    "loopback-boot": "^2.26.1",
    "loopback-cascade-delete-mixin": "^2.0.0",
    "loopback-component-explorer": "^5.3.0",
    "loopback-component-storage": "^3.3.1",
    "loopback-connector-mongodb": "^3.4.3",
    "loopback-connector-rest": "^3.1.1",
    "loopback-datasource-juggler": "^3.16.0",

Do you have a sample app somewhere that I can clone and try to run? I get the same errors on the samples available on this repo.

[jocelynk]

@hacksparrow I have discovered the issue. It is because I am using multer as middleware to filter my files. Does loopback-component-storage use formidable under the covers? I'm guessing multer is trying to read the same stream and interfering with the reading done by formidable. You can close this issue, but is there somewhere you can update in the documentation on this repo that prototype uploads will not work if you try to use multer in conjunction with loopback-component-storage?

[hacksparrow]

@jocelynk yes, loopback-component-storage uses formidable for multi part parsing. I will add a note regarding this behavior in the docs. Thanks.

[pbalan]

@jocelynk how did you handle this? I am getting an error when using multer as well, and if I remove multer, I get a validation error.

@hacksparrow @b-admike could you help check why I get the validation error if I remove multer?

Sample repo https://github.com/pbalan/component-blog

Steps to reproduce

• git clone https://github.com/pbalan/component-blog
• npm install
• npm start
• Need to login with user
• endpoint POST http://localhost:3000/api/users/login

{
"email":  "[email protected]",
"password":  "password"
}

• Remove multer from test/fixtures/simple-app/server/server.js
• endpoint POST http://localhost:3000/api/post?access_token=SET_ACCESS_TOKEN

{
    "postTitle": "Blog post - one",
    "postType": "text",
    "postText": "dummy post data",
    "createdAt": "2018-09-30T16:46:21.737Z",
    "modifiedAt": "2018-09-30T16:46:21.739Z",
    "isEnabled": true,
    "deletedAt": null,
    "isReported": false
}

[pbalan]

UPDATE: I figured out why validation fails. Its because I am using form-data in POSTMAN and headers were not application/x-www-form-urlencoded when I remove multer.

I am still looking help to process multipart/form-data to pass the file to /api/containers/blog-media/upload with or without creating a remote method.

Current behaviour: socket hangs

Expected behaviour: file uploads and returns the response from files API.

[arjamizo]

@pbalan I think you might have problems with loopback-component-storage

    Model.afterRemote('upload', function(ctx, unused, next) {
      let storageConfig = settings['component-storage'];
      let userId = null;
      if (undefined !== ctx.req.accessToken) {
        userId = ctx.req.accessToken && ctx.req.accessToken.userId;
      }

so if you would go to
node_modules/component-storage/common/models/Model.Files.json
and set

    "createdBy": {
      "type": "String",
      "required": false
    }

then you should not be getting.
Basically you might want to:

1. enable loopback#auth middleware to fill req.accessToken,
2. amend Model.Files.json required to false (can be done even with a /bin/touch server/boot/99-enable-anon-uploads.js new middleware or even beforeRemote on Containers.shared.upload)
3. any other way

Generally it would be much easier if you would provide us a failing test case with your scenario :)

[pbalan]

@arjamizo Thanks for your time. I have provided the repo to reproduce the error above. And I think this is not the main issue. If you pass the token in URL string, you can overcome the above issue you mentioned.

In my case, I am not passing just the file. I am passing a file with form fields. In that case it doesn't work quite well. The only solution to this I found was to create a remote method and disable the default remote "create" method for Post.

I'll post my solution using formidable soon.