RequestsHTTPTransport: retries option with POST method does not effect

[khiker]

• RequestsHTTPTransport's retries option uses requests library.
• RequestsHTTPTransport's HTTP(S) request uses POST method by default.
• requests's max_retries option does not effects with POST method by default.
  • requests.adapters.Retry == urllib3.util.retry.Retry
  • https://stackoverflow.com/a/35707701
  • https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry
  • https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry.DEFAULT_ALLOWED_METHODS

---

1. run tshark to catpture packets.

    $ sudo tshark -i eth1 'dst host httpstat.us'
   

2. open another terminal and run retry.py

    $ mkdir xxx && cd xxx
    $ pipenv install gql
    $ pipenv run python -V
   3.9.2
    $ vi retry.py
    $ pipenv run python ./retry.py
   

3. confirm output of tshark and i can see 1 post request only.

   Running as user "root" and group "root". This could be dangerous.
   Capturing on 'eth1'
       1 0.000000000 xxx → xxx TCP 94 57164 → 80 [SYN] Seq=0 Win=64800 Len=0 MSS=1440 SACK_PERM=1 TSval=4064012293 TSecr=0 WS=128
       2 0.003444162 xxx → xxx TCP 74 57164 → 80 [ACK] Seq=1 Ack=1 Win=64896 Len=0
       3 0.003465623 xxx → xxx TCP 274 POST /500 HTTP/1.1  [TCP segment of a reassembled PDU]
       4 0.003475453 xxx → xxx HTTP/JSON 1491 POST /500 HTTP/1.1 , JavaScript Object Notation (application/json)
       5 0.233443595 xxx → xxx TCP 74 57164 → 80 [ACK] Seq=1618 Ack=1174 Win=64128 Len=0
       6 0.239596636 xxx → xxx TCP 74 57164 → 80 [FIN, ACK] Seq=1618 Ack=1174 Win=64128 Len=0
       7 0.243453262 xxx → xxx TCP 74 57164 → 80 [ACK] Seq=1619 Ack=1175 Win=64128 Len=0
   

retry.py: (to recieve 500 response, use httpstat.us)

from gql import Client, gql
from gql.transport.requests import RequestsHTTPTransport

resp = Client(
    transport=RequestsHTTPTransport(
        url="http://httpstat.us/500",
        retries=3
    ),
    fetch_schema_from_transport=True,
).execute(gql("query { xxx { value }}"))
print(resp)

[claudinoac]

I'm having the same problem you described. All the times a request fails in gql, it doesn't retry it, and returns the error.

[leszekhanusz]

@khiker @claudinoac
Could you please check that the PR #249 fixes the problem? Thanks

Note: to have debugging info, you can use:

import logging
logging.basicConfig(level=logging.DEBUG)

[khiker]

Confirmed fix. Thank you.

---

 $ mkdir /tmp/xxx
 $ cd /tmp/xxx
 $ git clone https://github.com/leszekhanusz/gql.git
 $ cd gql
 $ git checkout fix_requests_post_retries
 $ cd -
 $ pipenv install --pre gql\[all\]
 $ grep '_method' .venv/lib/python3.9/site-packages/gql/transport/requests.py
                        allowed_methods=None,
 $ cp /path/to/retry.py .
 $ pipenv run python ./retry.py
...
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): httpstat.us:80
DEBUG:urllib3.connectionpool:http://httpstat.us:80 "POST /500 HTTP/1.1" 500 25
DEBUG:urllib3.util.retry:Incremented Retry for (url='/500'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Retry: /500
DEBUG:urllib3.connectionpool:http://httpstat.us:80 "POST /500 HTTP/1.1" 500 25
DEBUG:urllib3.util.retry:Incremented Retry for (url='/500'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Retry: /500
DEBUG:urllib3.connectionpool:http://httpstat.us:80 "POST /500 HTTP/1.1" 500 25
DEBUG:urllib3.util.retry:Incremented Retry for (url='/500'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Retry: /500
DEBUG:urllib3.connectionpool:http://httpstat.us:80 "POST /500 HTTP/1.1" 500 25
Traceback (most recent call last):
  File "/tmp/xxx/.venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/tmp/xxx/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  File "/tmp/xxx/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  File "/tmp/xxx/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  File "/tmp/xxx/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 836, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/tmp/xxx/.venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses'))
...

retry.py:

from gql import Client, gql
from gql.transport.requests import RequestsHTTPTransport

import logging
logging.basicConfig(level=logging.DEBUG)

resp = Client(
    transport=RequestsHTTPTransport(
        url="http://httpstat.us/500",
        retries=3
    ),
    fetch_schema_from_transport=True,
).execute(gql("query { xxx { value }}"))
print(resp)

[leszekhanusz]

Fixed in version v3.0.0b0