[BUG] unable to create spark context from jupyterlab

[sdabhi23]

Bug

Expected behaviour

Should be able to create spark context using code from the given examples

Current behaviour

The spark-master logs show the following error when trying to connect to the cluster from the notebook:

ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.

java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 6543101073799644159, local class serialVersionUID = 1574364215946805297

The root cause is the mismatch between the scala versions. Spark 3.0.0 is pre-built using Scala 2.12.10 while the containers are bundled with Scala 2.12.11. This is a known issue.

Reference:

1. https://stackoverflow.com/questions/45755881/unable-to-connect-to-remote-spark-master-ror-transportrequesthandler-error-wh
2. https://almond.sh/docs/usage-spark

   ammonite-spark handles loading Spark in a clever way, and does not rely on a specific Spark distribution. Because of that, you can use it with any Spark 2.x version. The only limitation is that the Scala version of Spark and the running Almond kernel must match, so make sure your kernel uses the same Scala version as your Spark cluster. Spark 2.0.x - 2.3.x requires Scala 2.11. Spark 2.4.x supports both Scala 2.11 and 2.12.

Steps to reproduce

1. Use docker-compose up -d to bring up the cluster
2. Create a new Scala notebook and create Spark context using the following code:

import $ivy.`org.apache.spark::spark-sql:2.4.4`;

import org.apache.log4j.{Level, Logger};
Logger.getLogger("org").setLevel(Level.OFF);

import org.apache.spark.sql._

val spark = {
  NotebookSparkSession.builder()
    .master("spark://spark-master:7077")
    .config("spark.executor.instances", "4")
    .config("spark.executor.memory", "2g")
    .getOrCreate()
}

Steps to check Scala version

1. Jupyterlab Launcher shows the Scala version installed
2. Launch a terminal on spark-master and fire up the spark shell using the command: bin/spark-shell

From my spark-master node:

# bin/spark-shell
20/09/05 10:07:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://d8b69de92ccc:4040
Spark context available as 'sc' (master = local[*], app id = local-1599300457142).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0
      /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_265)
Type in expressions to have them evaluated.
Type :help for more information.

scala> ^C
# scala
Welcome to Scala 2.12.11 (OpenJDK 64-Bit Server VM, Java 1.8.0_265).
Type in expressions for evaluation. Or try :help.

scala> ^C
#

Possible solutions (optional)

Bundle the Spark 3.0.0 containers with Scala 2.12.10 and check for similar issues for other versions of Spark as well

Comments (optional)

I can help resolve this issue with a PR

Checklist

Please provide the following:

• Docker Engine version: 19.03.8
• Docker Compose version: 1.25.5

[sdabhi23]

Also looks like almond does not support spark 3.0.0 yet

Known issues with ammonite (link):

• Local clusters, Mesos, and Kubernetes, aren't supported yet.
• No scala 2.10 or 2.11 support (support for those was dropped by Ammonite).

[amhambra]

Dear Shrey,
I did the same on the current docker-compose.yml and got below. So we need to adapt Scala versions. Any hints about upgrade the JupterLab Scala kernal?

Will dig myself deeper as the Python sample is super!

Thanks
Frank

# bin/spark-shell
20/09/18 09:52:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://c78f26f5eeb7:4040
Spark context available as 'sc' (master = local[*], app id = local-1600422729476).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_265)
Type in expressions to have them evaluated.
Type :help for more information.

scala> # scala
Welcome to Scala 2.12.11 (OpenJDK 64-Bit Server VM, Java 1.8.0_265).
Type in expressions for evaluation. Or try :help.

scala> #
#

[amhambra]

Nope, did not help. Adapted to almond 0.10.6 and scala 2.11.12 but same error as in this topic.

# bin/spark-shell
20/09/18 10:56:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://c2deed8d2214:4040
Spark context available as 'sc' (master = local[*], app id = local-1600426624180).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_265)
Type in expressions to have them evaluated.
Type :help for more information.

scala> #
# scala
Welcome to Scala 2.12.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_265).
Type in expressions for evaluation. Or try :help.

scala> #
#

diff

diff --git a/build/build.yml b/build/build.yml
index 008de29..d7b2f65 100644
--- a/build/build.yml
+++ b/build/build.yml
@@ -1,5 +1,5 @@
 applications:
-  scala: "2.12.11"
+  scala: "2.12.12"
   spark: "2.4.4"
   hadoop: "2.7"
   jupyterlab: "2.1.4"


diff --git a/build/docker/jupyterlab/Dockerfile b/build/docker/jupyterlab/Dockerfile
index eece7d3..d9830d4 100644
--- a/build/docker/jupyterlab/Dockerfile
+++ b/build/docker/jupyterlab/Dockerfile
@@ -28,7 +28,7 @@ ARG scala_version
 RUN apt-get install -y ca-certificates-java --no-install-recommends && \
     curl -Lo coursier https://git.io/coursier-cli && \
     chmod +x coursier && \
-    ./coursier launch --fork almond:0.10.0 --scala ${scala_version} -- --display-name "Scala ${scala_version}" --install && \
+    ./coursier launch --fork almond:0.10.6 --scala ${scala_version} -- --display-name "Scala ${scala_version}" --install && \
     rm -f coursier
 
 # -- Layer: R kernel for SparkR

More thinking ...

Cheers
Frank

[amhambra]

OK, giving up. Also a standalone scala app accessing Spark Server has problems. Only the pyspac.ipynb is working fine.

[andre-marcos-perez]

Hey guys, I am little busy right now at work and will get at you as soon as I can. Anyway, thanks for contributing so far.

[amhambra]

Hey again,
got it working and added a Pull for review. For me it is working after upgrade to Spark 3.0.1.
Cheers
Frank

[andre-marcos-perez]

Hi there @amhambra and @sdabhi23, thanks for your guys contribution. Indeed the mismatch version between the compiled Spark app and the install Scala version is an issue! I will be hard coding the Hadoop and Scala version for each Spark version on the build scripts, removing a bit of developer freedom but guarantying the correctness of the cluster.

@amhambra, I will be closing your PR to work on a more robust one, thanks! If you like to be added on the contributors list, please submit a pull request to the readme file.

[andre-marcos-perez]

The truth is that I would love to remove the Spark 2.x dependency but SparkR on CRAN repository only goes up to Spark 2.x version, so I am kind stuck on it until they release a stable version.