Skip to content

NullPointerException #1769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
aqingsao opened this issue Jan 27, 2021 · 8 comments
Closed

NullPointerException #1769

aqingsao opened this issue Jan 27, 2021 · 8 comments
Assignees
@TomGranot

Comments

@aqingsao
Copy link

I've been using async-http-client for a long time, it works fine in many cases even at a high QPS of 100K.
But recently some errors happen when calls to a specific api(nothing special, just a POST http with Content-Type: multipart/form-data), lots of exceptions are thrown at a low QPS of 800, and below is the stack trace:

`java.lang.NullPointerException: null
at org.asynchttpclient.netty.channel.Channels.getAttribute(Channels.java:31) ~[async-http-client-2.12.2.jar!/:na]
at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:467) ~[async-http-client-2.12.2.jar!/:na]
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.exceptionCaught(AsyncHttpClientHandler.java:201) ~[async-http-client-2.12.2.jar!/:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:264) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:138) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.http.HttpContentDecoder.channelInactive(HttpContentDecoder.java:235) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.55.Final.jar!/:4.1.55.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]

Any idea of what can I do?
@TomGranot
Copy link
Contributor

Hey @aqingsao - new maintainer here, still learning the ropes.

Can you show the exact code you used + mention some specs? Might help the investigation.

@TomGranot TomGranot self-assigned this Feb 1, 2021
@aqingsao
Copy link
Author

aqingsao commented Feb 1, 2021

Hi TomGranot,
Glad to see u and welcome.
I use AsyncHttpClient to build our load test platform, it works fine in most cases.
But in a few cases, some exceptions(NullPointerException) are thrown at about 0.1% rate or even 30% in a very special case.
Actually these case are just normal http POST requests of type 'form-data'.

Some codes are pasted in below but I don't think it provides any clues.

this.builder.setUserAgent("xxx"); this.builder.setThreadFactory(new CustomThreadFactory("netty-client")); this.builder.setIoThreadsCount(Math.max(Runtime.getRuntime().availableProcessors() / 2, 2)); this.builder.setMaxConnections(MAX_CONNECTIONS); this.builder.setMaxConnectionsPerHost(MAX_CONNECTIONS_PER_HOST); this.builder.setMaxRequestRetry(0); try { this.builder.setSslContext(SslContextBuilder.forClient() .sslProvider(SslProvider.OPENSSL) .trustManager(InsecureTrustManagerFactory.INSTANCE).build() ); } catch (SSLException e) { log.error("Failed to set SSL config", e); } this.builder.setCookieStore(null); new DefaultAsyncHttpClient(this.builder.build());

httpClient.executeRequest(request, new ProgressHandler(request, callback)).toCompletableFuture();

@rachid-o
Copy link
Contributor

rachid-o commented Feb 1, 2021

I think I have the same issue. We have a unit test where we simulate a "connection reset by peer" which worked fine on async-http-client version 2.12.1
But when updating to version 2.12.2 we got this NPE:

 java.lang.NullPointerException
	at org.asynchttpclient.netty.channel.Channels.getAttribute(Channels.java:31)
	at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:467)
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.exceptionCaught(AsyncHttpClientHandler.java:201)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:264)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:138)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.codec.http.HttpContentDecoder.channelInactive(HttpContentDecoder.java:235)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
	at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311)
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

For what I've seen in NettyRequestSender.abort() there is this line:

Object attribute = Channels.getAttribute(future.channel());

and future.channel() == null

@rachid-o
Copy link
Contributor

rachid-o commented Feb 5, 2021 •
edited
Loading

Some extra information.
Before suspecting async-http-client we downgraded Netty to 4.1.53.final, but then we still got the same problem.

We could reproduce it at our side with the following unit test which uses WireMock (Kotlin code):

private val httpClient: AsyncHttpClient = DefaultAsyncHttpClient(
    DefaultAsyncHttpClientConfig.Builder()
        .setRequestTimeout(2000) // Overwrite the default of 60 secs so that unit test fail faster on timeouts
        .build()
)
private val wireMockServer: WireMockServer = WireMockServer(wireMockConfig().dynamicPort())

@Test
fun `Should throw IOException when connection is reset by peer`() {
    wireMockServer.stubFor(
        get(urlEqualTo("/test"))
            .willReturn(
                aResponse()
                    .withFault(Fault.CONNECTION_RESET_BY_PEER)
            )
    )
    val testSubscriber = TestSubscriber<Response>()
    response().subscribe(testSubscriber)
    testSubscriber.awaitTerminalEvent()
    testSubscriber.assertError(IOException::class.java)
}

I took a quick look in the source code, but couldn't directly find a way to create a test case simulating a "connection reset by peer" in this repo.
@TomGranot let me know if I can assist. Maybe you or someone else can push me in the right direction so that I can create a failing unit test.

@dzharikhin
Copy link

dzharikhin commented Feb 9, 2021 •
edited
Loading

Hi, I extracted resetting server logic from the tests I have. Here's simple test

public class AsyncHttpClientTest {

    private String resettingServerAddress;

    @Before
    public void setUp() {
        resettingServerAddress = createResettingServer();
    }

    @Test
    public void testAsyncHttpClient() throws ExecutionException, InterruptedException {
        try {
            new DefaultAsyncHttpClient().executeRequest(new org.asynchttpclient.RequestBuilder("GET").setUrl(resettingServerAddress)).get();
        } catch (ExecutionException e) {
            var ex = e.getCause();
            assertTrue(ex instanceof SocketException);
            assertTrue(ex.getMessage().equalsIgnoreCase("connection reset"));
        }
    }

    private static String createResettingServer() {
        return createServer(sock -> {
            try (Socket socket = sock) {
                socket.setSoLinger(true, 0);
                var inputStream = socket.getInputStream();
                //to not eliminate read
                OutputStream.nullOutputStream().write(startRead(inputStream));
            }
        });
    }

    private static String createServer(MoreFunctionalInterfaces.FailableConsumer<Socket, Exception> handler) {
        Exchanger<Integer> portHolder = new Exchanger<>();
        var t = new Thread(() -> {
            try (ServerSocket ss = new ServerSocket(0)) {
                portHolder.exchange(ss.getLocalPort());
                while (true) {
                    handler.accept(ss.accept());
                }
            } catch (Exception e) {
                if (e instanceof InterruptedException) {
                    Thread.currentThread().interrupt();
                }
                throw new RuntimeException(e);
            }
        });
        t.setDaemon(true);
        t.start();
        return tryGetAddress(portHolder);
    }

    private static String tryGetAddress(Exchanger<Integer> portHolder) {
        try {
            return "http://localhost:" + portHolder.exchange(0);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    }

    private static byte[] startRead(InputStream inputStream) throws IOException {
        byte[] buffer = new byte[4];
        int length = inputStream.read(buffer);
        return Arrays.copyOf(buffer, length);
    }
}

@rachid-o
Copy link
Contributor

rachid-o commented Feb 9, 2021 •
edited
Loading

That certainly helps @dzharikhin!
I molded your code into a fork in this branch: master...rachidbm:2.12.2
There I see the NPE, which I don't see when running the same test on master...rachidbm:2.12.1

@rachid-o rachid-o mentioned this issue Feb 9, 2021
Merged
This was referenced Mar 15, 2021
Closed
Closed
@rachid-o
Copy link
Contributor

rachid-o commented Mar 31, 2021 •
edited
Loading

Thnx for merging @TomGranot, this issue can be closed now.

FWIW, when using the new version of async-http-client (2.12.3) the issue is solved in our project.

@TomGranot
Copy link
Contributor

@rachidbm Excellent, closing this as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TomGranot TomGranot

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@aqingsao @rachid-o @dzharikhin @TomGranot