InstantiatingGrpcChannelProviderTest is constantly failing on macos-12 with Java 8 and Java 11

[blakeli0]

InstantiatingGrpcChannelProviderTest failed last night in nightly-java8 (macOS-12) and nightly (macOS-12, 11) due to NoClassDefFound errors.

[burkedavison]

java.lang.AssertionError: java.lang.ClassNotFoundException: sun/nio/fs/LinuxFileSystemProvider
	at sun.nio.fs.DefaultFileSystemProvider.createProvider(DefaultFileSystemProvider.java:45)
	at sun.nio.fs.DefaultFileSystemProvider.create(DefaultFileSystemProvider.java:63)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(FileSystems.java:108)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.access$000(FileSystems.java:89)
	at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:98)
	at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:96)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:96)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.<clinit>(FileSystems.java:90)
	at java.nio.file.FileSystems.getDefault(FileSystems.java:176)
	at java.nio.file.Paths.get(Paths.java:138)
	at sun.misc.Launcher$ExtClassLoader.findLibrary(Launcher.java:238)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1823)
	at java.lang.Runtime.loadLibrary0(Runtime.java:843)
	at java.lang.System.loadLibrary(System.java:1136)
	at sun.security.ec.SunEC$1.run(SunEC.java:60)
	at sun.security.ec.SunEC$1.run(SunEC.java:58)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.ec.SunEC.<clinit>(SunEC.java:58)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:221)
	at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:206)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
	at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
	at sun.security.jca.ProviderList.getProvider(ProviderList.java:233)
	at sun.security.jca.ProviderList.getService(ProviderList.java:331)
	at sun.security.jca.GetInstance.getInstance(GetInstance.java:157)
	at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:137)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createMtlsChannelCredentials(InstantiatingGrpcChannelProvider.java:367)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProviderTest.getMtlsObjectFromTransportChannel(InstantiatingGrpcChannelProviderTest.java:[568](https://github.com/googleapis/sdk-platform-java/actions/runs/9359761293/job/25763994478#step:9:569))
	at com.google.api.gax.rpc.mtls.AbstractMtlsTransportChannelTest.testUseClientCertificate(AbstractMtlsTransportChannelTest.java:70)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: java.lang.ClassNotFoundException: sun/nio/fs/LinuxFileSystemProvider
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at sun.nio.fs.DefaultFileSystemProvider.createProvider(DefaultFileSystemProvider.java:43)
	... 38 more

[burkedavison]

java.lang.AssertionError: java.lang.ClassNotFoundException: jdk/net/LinuxSocketOptions
	at jdk.net/jdk.net.ExtendedSocketOptions$PlatformSocketOptions.newInstance(ExtendedSocketOptions.java:331)
	at jdk.net/jdk.net.ExtendedSocketOptions$PlatformSocketOptions.create(ExtendedSocketOptions.java:345)
	at jdk.net/jdk.net.ExtendedSocketOptions$PlatformSocketOptions.<clinit>(ExtendedSocketOptions.java:353)
	at jdk.net/jdk.net.ExtendedSocketOptions.<clinit>(ExtendedSocketOptions.java:165)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at java.base/sun.net.ext.ExtendedSocketOptions.<clinit>(ExtendedSocketOptions.java:109)
	at java.base/java.net.PlainSocketImpl.<clinit>(PlainSocketImpl.java:59)
	at java.base/java.net.Socket.setImpl(Socket.java:523)
	at java.base/java.net.Socket.<init>(Socket.java:88)
	at java.base/javax.net.ssl.SSLSocket.<init>(SSLSocket.java:162)
	at java.base/sun.security.ssl.BaseSSLSocketImpl.<init>(BaseSSLSocketImpl.java:61)
	at java.base/sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:123)
	at java.base/sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:72)
	at java.base/javax.net.ssl.SSLContextSpi.getDefaultSocket(SSLContextSpi.java:163)
	at java.base/javax.net.ssl.SSLContextSpi.engineGetDefaultSSLParameters(SSLContextSpi.java:188)
	at java.base/javax.net.ssl.SSLContext.getDefaultSSLParameters(SSLContext.java:454)
	at io.grpc.netty.shaded.io.netty.handler.ssl.JdkSslContext.defaultProtocols(JdkSslContext.java:125)
	at io.grpc.netty.shaded.io.netty.handler.ssl.JdkSslContext.access$100(JdkSslContext.java:59)
	at io.grpc.netty.shaded.io.netty.handler.ssl.JdkSslContext$Defaults.init(JdkSslContext.java:108)
	at io.grpc.netty.shaded.io.netty.handler.ssl.JdkSslContext.<clinit>(JdkSslContext.java:73)
	at io.grpc.netty.shaded.io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:829)
	at io.grpc.netty.shaded.io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:615)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$DefaultProtocolNegotiator.newNegotiator(NettyChannelBuilder.java:676)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:573)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyChannelTransportFactoryBuilder.buildClientTransportFactory(NettyChannelBuilder.java:187)
	at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:668)
	at io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:260)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:462)
	at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:107)
	at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:85)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:256)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:250)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProviderTest.testWithNonGCECredentials(InstantiatingGrpcChannelProviderTest.java:384)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
Caused by: java.lang.ClassNotFoundException: jdk/net/LinuxSocketOptions
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at jdk.net/jdk.net.ExtendedSocketOptions$PlatformSocketOptions.newInstance(ExtendedSocketOptions.java:328)
	... 36 more

[burkedavison]

java.lang.NoClassDefFoundError: Could not initialize class io.grpc.netty.shaded.io.netty.handler.ssl.JdkSslClientContext
	at io.grpc.netty.shaded.io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:829)
	at io.grpc.netty.shaded.io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:615)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$DefaultProtocolNegotiator.newNegotiator(NettyChannelBuilder.java:676)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:573)
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyChannelTransportFactoryBuilder.buildClientTransportFactory(NettyChannelBuilder.java:187)
	at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:668)
	at io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:260)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:462)
	at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:107)
	at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:85)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:256)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:250)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProviderTest.testWithDirectPathDisabled(InstantiatingGrpcChannelProviderTest.java:412)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)

[blakeli0]

Investigating in #2891

[alicejli]

Failed again last night: #2929 and #2928

[alicejli]

Same as 2 nights ago: #2926 and #2927

[lqiu96]

We have a suspicion that this line in the test is causing the failures:

sdk-platform-java/gax-java/gax-grpc/src/test/java/com/google/api/gax/grpc/InstantiatingGrpcChannelProviderTest.java

Line 92 in aad24f6

System.setProperty("os.name", "Linux");

It is setting the os.name property before each test, even for tests that end up creating a gRPC channel. It is possible that the underlying grpc logic does some stuff based on the OS it determined. Assumption is that it can't find Linux specific files or can't load linux specific class files on a Mac machine because it doesn't exist.