Handle audit exceptions

[albertzaharovits]

This PR ensures that exceptions in the AuditTrail methods are properly handled.
Currently the log4j appender is configured to swallow exceptions, but after this is merged we actually document the advice to turn that on (or turn it on ourselves).

Related #29905
#62916

[albertzaharovits]

@tvernum @ywangd I've raised this in relation to #62916 .
Log4j currently swallows exceptions, so the audit methods have so far been sprinkled liberally disregarding the behaviour when throw. But #62916 complicates it because the audit methods might throw (due to bugs or due to a mischievous transport clients); so we need to consider (and test) the throw behaviour. Then if the throw behaviour is robust, we can advise users to toggle the ignoreExceptions in the audit appender (or toggle it ourselves) and close #29905 (if auditing fails, you'd be getting failures for every subsequent requests - the known state).

[albertzaharovits]

WIP only handles exceptions in the AuthorizationService and no tests.

[ywangd]

A general comment: maybe we should add a setting for this behaviour, i.e. fail when an exception is caught during auditing time? Because I am not sure whether all customers would be more concerned about these exception than letting cluster continue to handle requests. Also #29905 says "add option" if you read it literally.

[tvernum]

This approach seems right to me, although I agree with @ywangd's earlier comments - not everyone will want to fail all cluster activity just because auditing failed.

[albertzaharovits]

I'm going to close this as I don't have the time to continue working on it.
Anyone please feel free to reuse anything you can scavange out of it.