Pinning to Pinata disabled due to "Rate limit exceeded"

[RubenKelevra]

• OS: Windows
• Version of IPFS Desktop 0.18.1

Describe the bug
Pinata integration stopped working. So it worked before and after some update in the recent past it stopped working. I thought it's a temporary outage as I saw it first. But that's quite a while ago, maybe a month or two, and it's still not working.

To Reproduce
Steps to reproduce the behavior:

1. -> Settings
2. Click "+ Add Service"
3. Select Pinata
4. Go to Pinata's API page (which is linked)
5. Create a new key
6. Add it to the dialog opened in step 3
7. Click on Save
8. Pinata's status is "Pins: Failed to fetch"

Expected behavior
The integrated service connection should work.

Screenshots

[hacdias]

@RubenKelevra I think it's related to your API token: it's either expired or doesn't have the required permissions (Pinning API permissions).

[RubenKelevra]

@RubenKelevra I think it's related to your API token: it's either expired or doesn't have the required permissions (Pinning API permissions).

Well, the token is new and has all permissions active.

[hacdias]

Very odd, it works well here. Did you use the correct token? It's the JWT token (secret access token) and not the API secret.

[hacdias]

@RubenKelevra any updates on this?

[RubenKelevra]

Hey @hacdias, sorry for the delay. I think the issue was related to my ISP - it's working fine now without any change 🤷

Thanks for the efforts to help me, highly appreciated!

[lidel]

FWIW I noticed similar problem while testing webui in ipfs-desktop and clicking between settings and files.

Poked at things and this.. is not false-negative.

Pinata is rate-limiting, and HTTP checks sent by ipfs pin remote service ls --stat (used by ipfs-webui for determining if service is in a good shape) are not successful when executed in short window of time:

$ ipfs pin remote service ls --stat                                                                                                         ~
Pinata       https://api.pinata.cloud/psa     0/0/11/0
estuary      https://api.estuary.tech/pinning 14/14/1/0
pinata-debug http://127.0.0.1:5000/psa        invalid
pinata-test  https://testapi.pinata.cloud/psa 0/0/16/5

$ ipfs pin remote service ls --stat                                                                                                         ~
Pinata       https://api.pinata.cloud/psa     invalid
estuary      https://api.estuary.tech/pinning 14/14/1/0
pinata-debug http://127.0.0.1:5000/psa        invalid
pinata-test  https://testapi.pinata.cloud/psa invalid

I suspect that if you switch from Settings to Files, and try to pin something you hit the rate limit quite often.

Low level error can be reproduced with curl

$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"error":{"reason":"Rate limit exceeded","details":"You have exceeded the API rate limit. Wait a minute and slow down your requests."}}

@obo20 any suggestions how we avoid triggering this rate limiter?

ipfs pin remote service ls --stat sends 4 requests (one for each status), and you seem to block around 6 requests in some short window – pretty low – perhaps we could just bump this to more reasonable throttling (eg. 30-60 requests / minute)?

[RubenKelevra]

@lidel yeah, I can confirm that I still have issues. It jumped back to "Failed to fetch".

So I created a new key, and when I add it shows that it was working and but it jumps to "Failed to fetch" when I turn on auto-upload.

So not sure if pinata doesn't want users to use their pin-service or if that's an IPFS issue but it's kind of annoying since I have to manually upload everything to their service.

[lidel]

I wonder if we need to add support for Retry-After: 60 to https://github.com/ipfs/go-pinning-service-http-client/ and ipfs pin remote CLI.

Pinata returns:

< HTTP/1.1 429 Too Many Requests
< Server: nginx/1.16.1
< Date: Mon, 07 Feb 2022 21:08:07 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 135
< Connection: keep-alive
< X-Powered-By: Express
< Vary: Origin
< Access-Control-Allow-Credentials: true
< X-RateLimit-Limit: 5
< X-RateLimit-Remaining: 0
< X-RateLimit-Reset: 1644268110
< Retry-After: 60

Open question if ipfs pin remote command hanging for a minute is a better UX than:

$ ipfs pin remote ls --service=Pinata
Error: reason: "Rate limit exceeded", details: "You have exceeded the API rate limit. Wait a minute and slow down your requests.": 429 Too Many Requests