Epoll based Netty Transport error in Function Development Tool on Linux

[anuchandy]

Function Development Tool on Linux (I guess this is true for production Linux Function as well) fails when using "Epoll based Netty Transport" in the application. The issue can be reproduced by creating a function app with netty modules as it dependency along with "netty-Epoll-for-Linux".

Steps to reproduce:

Create an Azure Ubuntu 18.04 Linux VM.

SSH and set up

Set Package feed:

wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update

Install Function tool:

sudo apt-get install azure-functions-core-tools-3

Intstall Java8:

sudo add-apt-repository --remove ppa:webupd8team/java
sudo apt install apt-transport-https && sudo apt update && sudo apt install -y openjdk-8-jdk

Set JAVA_HOME:

Find path to Java 8:

sudo update-alternatives --config java
Example: > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

Open environment file:

sudo vi /etc/environment

Set Java home, save and exit:

JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"

Reload environment:

source /etc/environment

Install Maven:

sudo apt install maven

Download repro-function project to this Linux box, package and Run:

azure-func-netty.zip

mvn clean package
mvn azure-functions:run &

Hit function endpoint:

curl http://localhost:7071/api/HttpExample?name=hello

You should get the below error:

Caused by: java.lang.NoSuchMethodError: 
io.netty.channel.SingleThreadEventLoop.<init>(Lio/netty/channel/EventLoopGroup;Ljava/util/concurrent/Executor;ZLjava/util/Queue;Ljava/util/Queue;Lio/netty/util/concurrent/RejectedExecutionHandler;)V
	at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:87)
	at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:143)
	at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:52)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:45)
	at org.example.Function.doHttpGet(Function.java:71)
	at org.example.Function.run(Function.java:55)
	... 16 more

Here is a normal console app, having the same netty dependencies but runs fine

linux-use-netty1.zip

Download this project to the above Linux box and run:

mvn clean compile exec:java

The Epoll based code work as expected.

[anuchandy]

Linking similar issue in function library: Azure/azure-functions-java-library#110

[amamounelsayed]

#365

Thank you @anuchandy We are working on removing the conflicts between java worker dependencies and user code. Specially with netty.

[amamounelsayed]

@anuchandy Just curious why did you need to build your image? you can use https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-function-linux-custom-image?tabs=bash%2Cportal&pivots=programming-language-java

Do you mind give your feedback using customer container? FYI the above code will be broken regardless it is windows or linux as it is related to netty dependency which we are working on.

[anuchandy]

@amamounelsayed thank you for following up, I used a raw Linux VM because the customer setup was Linux, it was not explicit in the customer's original issue that it's a Linux container. Thanks for sharing the instruction link. I will request the customer to follow this thread.

Yes, you are right code is broken, because once EPoll is excluded, we'll run into another conflicting issue. but I felt this EPoll specific issue interesting given base type it uses is already in the classpath. Not sure it has anything to do with ServiceLoading or any similar approach that netty uses to load this optional EPoll Dependency.

Btw Is it always recommended to use containers for testing and debugging? I was so far debugging directly from IntelliJ, instead of linking IntelliJ to the container.

[amamounelsayed]

@anuchandy sorry for my delay, there is no recommended way for testing and debugging. But I prefer that testing at least will be on the container as this will be 1-1 with the production.