[Doc] clarify retry loop

[kaisecheng]

The retry policy confuses users mainly due to the unclear boundary of retry_failed and automatic_retries.

This PR explains retry_failed set to false does not affect the number of retries in lib level

[kaisecheng]

non-retriable exceptions are from https://github.com/apache/httpcomponents-client/blob/4.5.x/httpclient/src/main/java/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java#L104-L108

[andsel]

From a user perspective maybe is better not to report the Java exceptions names, but the concept they carries.
So it could be something like:

Non retriable errors could regard SSL problems or host not resolved but also connection problem or OS/JVM level interruptions happening during a request.

[kaisecheng]

awesome suggestion!

[andsel]

The splitting seems good, I've left a note on how to improve the description of what not retriable errors are, without listing any specific Java exception

[andsel]

From a user perspective maybe is better not to report the Java exceptions names, but the concept they carries.
So it could be something like:

Non retriable errors could regard SSL problems or host not resolved but also connection problem or OS/JVM level interruptions happening during a request.

[karenzone]

Fantastic explanations, @kaise. Thank you for taking this on.

I left some comments about wording and formatting for your consideration. I can't test the documentation until changes are applied, but I'll be happy to retest.

[karenzone]

I left some comments wrt formatting.

Now

Here's how it renders now:

Proposed

We can make it easier for users to scan and parse with some formatting tweaks (as suggested inline):

Let me know what you think and if you would like to discuss. Thanks for your work on this.

[karenzone]

Also, what do you think about noting in option descriptions which retry type they apply to?

[kaisecheng]

Also, what do you think about noting in option descriptions which retry type they apply to?

If we have some markers to make it clear which options belong to which types, it would be very good. But now I only see verbose messages, so my new commits give a pointer (see Retry Policy...) to all four options. I hope these hints are clear enough.

[kaisecheng]

My change was misleading. We don't have plugin-level requests, instead, we have plugin-level retries

[kaisecheng]

@karenzone Wow! The new format is so amazing! You raise it to the next level 🚀

Committed all your suggestion.
I copy some important information from "Retry Policy" session to those four options, make a pointer from the options to the "Retry Policy", and update some wording. Please have a look to the last commit. Thank you.

[karenzone]

Left some suggestions for wording tweaks for your consideration. Otherwise, LGTM!
Thank you for clarifying how retries work.

[karenzone]

Suggested change

	When set to `true` and `automatic_retries` is enabled, this will cause non-idempotent HTTP verbs (such as POST) to be retried.
	When this option is set to `true` and `automatic_retries` is enabled, non-idempotent HTTP verbs (such as POST) to be retried.