Add otel trace instrumentation on gRPC calls

[Fricounet]

I'm proposing to add some basic tracing instrumentation of the gRPC calls made by the CSI clients using the code in connection.go to connect to the CSI driver.
The feature simply consists in adding the otelgrpc.UnaryClientInterceptor() to the existing ChainUnaryInterceptor. This is enough to create traces for all gRPC calls and they can be easily exported by the client.

I think it can be done by creating a new ConnectWithGrpcInterceptor function that will call the connect function with the added interceptor as a DialOption. This way the feature will be opt-in for the users.

I can contribute it (and the implementation in the CSI components) if it is something the community wants to see. I already implemented something similar on the azuredisk-csi-driver and plan to do the same for aws and gcp drivers.

[jsafrane]

/cc

[Fricounet]

@jsafrane here is an example of the information gathered on a trace by the gRPC interceptor.
I collapsed the os and process fields because they are just some metadata that are not linked to the gRPC interceptor itself. Here, the relevant field for the interceptor are event which is the gRPC message itself and rpc which indicates which method was called.
I never noticed any confidential information leaking this way but if you have an example of a CSI components where you think it might happen, I'd be glad to test it on my side to check if there is any leakage.

[jsafrane]

Thanks for the example, it looks good - it has just the method name and not secrets that are in method arguments.

[mauriciopoppe]

/cc