Merge pull request #27 from andre-marcos-perez/develop

Develop

Author: andre-marcos-perez

Diff for: .github/ISSUE_TEMPLATE/bug_report.md

@@ -9,27 +9,38 @@ assignees: 'andre-marcos-perez'
 
 ## Introduction
 
-Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice parallel computing in distributed environments through our projects. :sparkles:
+Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice
+parallel computing in distributed environments through our projects. :sparkles:
 
 ## Bug
 
+Please fill the template below.
+
 ### Expected behaviour
 
+*Describe the expected behaviour*
+
 ### Current behaviour
 
+*Describe the current behaviour*
+
 ### Steps to reproduce
 
-1. Step 1
-2. Step 2
-3. Step 3
+1. *Step 1*
+2. *Step 2*
+3. *Step 3*
 
 ### Possible solutions (optional)
 
+*Add some solutions, if any*
+
 ### Comments (optional)
 
+*Add some comments, if any*
+
 ### Checklist
 
 Please provide the following:
 
-- [] Docker Engine version:
-- [] Docker Compose version:
+- [] Docker Engine version: *Can be found using `docker version`, e.g.: 19.03.6*
+- [] Docker Compose version: *Can be found using `docker-compose version`, e.g.: 1.21.0*

Diff for: .github/ISSUE_TEMPLATE/feature_request.md

@@ -9,10 +9,17 @@ assignees: 'andre-marcos-perez'
 
 ## Introduction
 
-Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice parallel computing in distributed environments through our projects. :sparkles:
+Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice
+parallel computing in distributed environments through our projects. :sparkles:
 
 ## Feature
 
+Please fill the template below.
+
 ### Description
 
+*Describe your feature request*
+
 ### Comments (optional)
+
+*Add some comments, if any*

Diff for: .github/pull_request_template.md

@@ -1,20 +1,27 @@
 ## Introduction
 
-Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice parallel computing in distributed environments through our projects. :sparkles:
+Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice
+parallel computing in distributed environments through our projects. :sparkles:
 
 ## Pull Request
 
-### Description
+### Issue
+
+- *Issue number with link, e.g.: [#22](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/issues/22)*
 
 ### Changes
 
-- Change 1
-- Change 2
+- *High level description of change 1*
+- *High level description of change 2*
+- *...*
 
 ### Comments (optional)
 
+*Add some comments, if any*
+
 ### Checklist
 
 Please make sure to check the following:
 
-- [] I have followed the steps in the [CONTRIBUTING.md](../CONTRIBUTING.md) file.
+- [] I have followed the steps in the [CONTRIBUTING.md](../CONTRIBUTING.md) file.
+- [] I am aware that pull requests that do not follow the rules will be automatically rejected.

Diff for: .github/workflows/ci.yml

@@ -3,7 +3,7 @@ name: build
 on:
 
   schedule:
-    - cron:  '0 0/12 * * *'
+    - cron:  '0 0 * * *'
 
   push:
     branches: [ master ]
@@ -195,6 +195,7 @@ jobs:
           cd ${GITHUB_WORKSPACE}/build
           docker build \
             --build-arg build_date="$(date -u +'%Y-%m-%d')" \
+            --build-arg scala_version="${SCALA_VERSION}" \
             --build-arg spark_version="${SPARK_VERSION}" \
             --build-arg jupyterlab_version="${JUPYTERLAB_VERSION}" \
             -f docker/jupyterlab/Dockerfile \
@@ -212,6 +213,7 @@ jobs:
           cd ${GITHUB_WORKSPACE}/build
           docker build \
             --build-arg build_date="$(date -u +'%Y-%m-%d')" \
+            --build-arg scala_version="${SCALA_VERSION}" \
             --build-arg spark_version="${SPARK_VERSION}" \
             --build-arg jupyterlab_version="${JUPYTERLAB_VERSION}" \
             -f docker/jupyterlab/Dockerfile \
@@ -227,6 +229,7 @@ jobs:
           cd ${GITHUB_WORKSPACE}/build
           docker build \
             --build-arg build_date="$(date -u +'%Y-%m-%d')" \
+            --build-arg scala_version="${SCALA_VERSION}" \
             --build-arg spark_version="${SPARK_VERSION}" \
             --build-arg jupyterlab_version="${JUPYTERLAB_VERSION}" \
             -f docker/jupyterlab/Dockerfile \

Diff for: CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [v1.1.0](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/releases/tag/v1.1.0) (2020-08-09)
+
+### Features
+
+ - Scala kernel for JupyterLab;
+ - Jupyter notebook with Spark Scala API example.
+
+### Repository
+
+ - Docs general improvements;
+ - Pull request template refactored.
+
+## [v1.0.0](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/releases/tag/v1.0.0) (2020-07-30)
+
+### Tech Stack
+
+ - **Infra**
+   - Python 3.7
+   - Scala 2.12
+   - Docker Engine 1.13.0+
+   - Docker Compose 1.10.0+
+
+ - **Apps**
+   - JupyterLab 2.1.4
+   - Apache Spark 2.4.0, 2.4.4 and 3.0.0
+
+### Features
+
+ - Docker compose file to build the cluster from your own machine;
+ - Docker compose file to build the cluster from Docker Hub;
+ - GitHub Workflow CI with Docker Hub to build the cluster daily.
+
+### Repository
+
+- Contributing rules;
+- GitHub templates for Bug Issue, Feature Request and Pull Request.
+
+### Community
+
+ - Article on [Medium](https://towardsdatascience.com/apache-spark-cluster-on-docker-ft-a-juyterlab-interface-418383c95445).

Diff for: CONTRIBUTING.md

@@ -1,24 +1,25 @@
 # Contributing
 
-Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice distributed
-and parallel computing through our projects. Please follow the template bellow to contribute.
+Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice
+parallel computing in distributed environments through our projects. :sparkles:
 
 ### Steps to contribute
 
-1. Fork the project;
-2. Create your feature branch, we use [gitflow](https://github.com/nvie/gitflow);
-3. Do your magic :rainbow:;
-4. Commit your changes;
-5. Push to your feature branch;
-6. Create a new pull request on the **develop** branch.
+1. Create an [issue](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/issues) to discuss features and bugs;
+2. Fork the project;
+3. Create your feature branch, we use [gitflow](https://github.com/nvie/gitflow);
+4. Do your magic :rainbow:;
+5. Commit your changes;
+6. Push to your feature branch;
+7. Create a new pull request from your  the **develop** branch.
 
 ### Contributions ideas
 
 - [] Microsoft Windows build script;
-- [x] DockerHub CI/CD integration;
+- [x] Docker Hub CI/CD integration;
 - [] Spark submit support;
-- [] JupyterLab Scala kernel;
-- [] Jupyter notebook with Apache Spark Scala API examples;
+- [x] JupyterLab Scala kernel;
+- [x] Jupyter notebook with Apache Spark Scala API examples;
 - [] JupyterLab R kernel;
 - [] Jupyter notebook with Apache Spark R API examples;
 - [] Test coverage.

Diff for: README.md

@@ -1,15 +1,18 @@
 # Apache Spark Standalone Cluster on Docker
 > The project just got its [own article](https://towardsdatascience.com/apache-spark-cluster-on-docker-ft-a-juyterlab-interface-418383c95445) at Towards Data Science Medium blog! :sparkles:
 
-This project gives you an out-of-the-box **Apache Spark** cluster in standalone mode with a **JupyterLab** interface and a simulated **Apache Hadoop Distributed File System**, all built on top of **Docker**. Learn Apache Spark through its Python API, **PySpark**, by running the [Jupyter notebooks](build/workspace/pyspark.ipynb) with examples on how to read, process and write data.
+This project gives you an **Apache Spark** cluster in standalone mode with a **JupyterLab** interface built on top of **Docker**.
+Learn Apache Spark through its Scala and Python API (PySpark) by running the Jupyter [notebooks](build/workspace/) with examples on how to read, process and write data.
 
 <p align="center"><img src="docs/image/cluster-architecture.png"></p>
 
 ![build](https://github.com/andre-marcos-perez/spark-standalone-cluster-on-docker/workflows/build/badge.svg?branch=master)
 ![jupyterlab-latest-version](https://img.shields.io/docker/v/andreper/jupyterlab/2.1.4-spark-3.0.0?color=yellow&label=jupyterlab-latest)
 ![spark-latest-version](https://img.shields.io/docker/v/andreper/spark-master/3.0.0-hadoop-2.7?color=yellow&label=spark-latest)
 ![docker-version](https://img.shields.io/badge/docker-v1.13.0%2B-blue)
-![docker-compose-version](https://img.shields.io/badge/docker--compose-v3.0%2B-blue)
+![docker-compose-file-version](https://img.shields.io/badge/docker--compose-v1.10.0%2B-blue)
+![spark-scala-api](https://img.shields.io/badge/spark%20api-scala-red)
+![spark-pyspark-api](https://img.shields.io/badge/spark%20api-pyspark-red)
 
 ## TL;DR
 
@@ -21,105 +24,104 @@ docker-compose up
 ## Contents
 
 - [Quick Start](#quick-start)
-- [Tech Stack Version](#tech-stack-version)
-- [Contributing](#contributing)
+- [Tech Stack](#tech-stack)
 - [Docker Hub Metrics](#docker-hub-metrics)
+- [Contributing](#contributing)
 - [Contributors](#contributors)
 
 ## <a name="quick-start"></a>Quick Start
 
 ### Cluster overview
 
-| Application                | URL                                      | Description                                                |
-| -------------------------- | ---------------------------------------- | ---------------------------------------------------------- |
-| **JupyterLab**             | [localhost:8888](http://localhost:8888/) | Cluster interface with PySpark built-in notebook           |
-| **Apache Spark Master**    | [localhost:8080](http://localhost:8080/) | Spark Master node                                          |
-| **Apache Spark Worker I**  | [localhost:8081](http://localhost:8081/) | Spark Worker node with 1 core and 512m of memory (default) |
-| **Apache Spark Worker II** | [localhost:8082](http://localhost:8082/) | Spark Worker node with 1 core and 512m of memory (default) |
+| Application            | URL                                      | Description                                                 |
+| ---------------------- | ---------------------------------------- | ----------------------------------------------------------- |
+| JupyterLab             | [localhost:8888](http://localhost:8888/) | Cluster interface with Scala and PySpark built-in notebooks |
+| Apache Spark Master    | [localhost:8080](http://localhost:8080/) | Spark Master node                                           |
+| Apache Spark Worker I  | [localhost:8081](http://localhost:8081/) | Spark Worker node with 1 core and 512m of memory (default)  |
+| Apache Spark Worker II | [localhost:8082](http://localhost:8082/) | Spark Worker node with 1 core and 512m of memory (default)  |
+
+### Prerequisites
+
+ - Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/), check **infra** [supported versions](#tech-stack)
 
-### Build from DockerHub
+### Build from Docker Hub
 
-1. Install [Docker and Docker Compose](https://docs.docker.com/get-docker/), check **infra** [supported versions](#tech-stack-version);
-2. Download the source code or clone the repository;
-3. Edit the [docker compose](docker-compose.yml) file with your favorite tech stack version, check **apps** [supported versions](#tech-stack-version);
-4. Build the cluster;
+1. Download the source code or clone the repository;
+2. Edit the [docker compose](docker-compose.yml) file with your favorite tech stack version, check **apps** [supported versions](#tech-stack);
+3. Build the cluster;
 
 ```bash
 docker-compose up
 ```
 
-5. Run Apache Spark code using the provided [Jupyter notebook](build/workspace/pyspark.ipynb) with PySpark examples.
+4. Run Apache Spark code using the provided Jupyter [notebooks](build/workspace/) with Scala and PySpark examples;
+5. Stop the cluster by typing `ctrl+c`.
 
 ### Build from your local machine
 
 > **Note**: Local build is currently only supported on Linux OS distributions.
 
-1. Install [Docker and Docker Compose](https://docs.docker.com/get-docker/), check **infra** [supported versions](#tech-stack-version);
-2. Download the source code or clone the repository;
-3. Move to the build directory;
+1. Download the source code or clone the repository;
+2. Move to the build directory;
 
 ```bash
 cd build
 ```
 
-4. Edit the [build.yml](build/build.yml) file with your favorite tech stack version;
-5. Match those version on the [docker compose](build/docker-compose.yml) file;
-6. Make the build script executable;
-
-```bash
-chmod +x build.sh
-```
-
-7. Build the images;
+3. Edit the [build.yml](build/build.yml) file with your favorite tech stack version;
+4. Match those version on the [docker compose](build/docker-compose.yml) file;
+5. Build the images;
 
 ```bash
-./build.sh
+chmod +x build.sh ; ./build.sh
 ```
 
-8. Build the cluster;
+6. Build the cluster;
 
 ```bash
 docker-compose up
 ```
 
-9. Run Apache Spark code using the provided [Jupyter notebook](build/workspace/pyspark.ipynb) with PySpark examples.
+7. Run Apache Spark code using the provided Jupyter [notebooks](build/workspace/) with Scala and PySpark examples;
+8. Stop the cluster by typing `ctrl+c`.
 
-## <a name="tech-stack-version"></a>Tech Stack Version
+## <a name="tech-stack"></a>Tech Stack
 
 - Infrastructure
 
-| App                | Version            |
-| ------------------ | ------------------ |
-| **Docker**         | 1.13.0+            |
-| **Docker Compose** | 3.0+               |
+| Component      | Version |
+| -------------- | ------- |
+| Docker Engine  | 1.13.0+ |
+| Docker Compose | 1.10.0+ |
+| Python         | 3.7     |
+| Scala          | 2.12    |
+
+- Jupyter Kernels
+
+| Component      | Version | Provider                        |
+| -------------- | ------- | ------------------------------- |
+| Python         | 2.1.4   | [Jupyter](https://jupyter.org/) |
+| Scala          | 0.10.0  | [Almond](https://almond.sh/)    |
 
 - Applications
 
-| App                | Version                 | Latest             |
-| ------------------ | ----------------------  | ------------------ |
-| **Apache Spark**   | 2.4.0 \| 2.4.4 \| 3.0.0 | 3.0.0              |
-| **Apache Hadoop**  | 2.7                     | 2.7                |
-| **JupyterLab**     | 2.1.4                   | 2.1.4              |
+| Component      | Version                 | Docker Tag                                           |
+| -------------- | ----------------------  | ---------------------------------------------------- |
+| Apache Spark   | 2.4.0 \| 2.4.4 \| 3.0.0 | **\<spark-version>**-hadoop-2.7                      |
+| JupyterLab     | 2.1.4                   | **\<jupyterlab-version>**-spark-**\<spark-version>** |
 
-- Tech
+## <a name="docker-hub-metrics"></a>Docker Hub Metrics
 
-| App                | Version            |
-| ------------------ | ------------------ |
-| **Python**         | 3.7                |
-| **Scala**          | 2.12               |
+| Image                                                          | Latest Version Size                                                                                 | Downloads                                                                 |
+| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
+| [JupyterLab](https://hub.docker.com/r/andreper/jupyterlab)     | ![docker-size](https://img.shields.io/docker/image-size/andreper/jupyterlab/latest)                 | ![docker-pull](https://img.shields.io/docker/pulls/andreper/jupyterlab)   |
+| [Spark Master](https://hub.docker.com/r/andreper/spark-master) | ![docker-size](https://img.shields.io/docker/image-size/andreper/spark-master/latest)               | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-master) |
+| [Spark Worker](https://hub.docker.com/r/andreper/spark-worker) | ![docker-size](https://img.shields.io/docker/image-size/andreper/spark-worker/latest)               | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-worker) |
 
 ## <a name="contributing"></a>Contributing
 
 We'd love some help. To contribute, please read [this file](CONTRIBUTING.md).
 
-## <a name="docker-hub-metrics"></a>Docker Hub Metrics
-
-| Image                                                              | Latest Version Size                                                                                 | Pulls                                                                     |
-| ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
-| **[JupyterLab](https://hub.docker.com/r/andreper/jupyterlab)**     | ![docker-size](https://img.shields.io/docker/image-size/andreper/jupyterlab/latest)    | ![docker-pull](https://img.shields.io/docker/pulls/andreper/jupyterlab)   |
-| **[Spark Master](https://hub.docker.com/r/andreper/spark-master)** | ![docker-size](https://img.shields.io/docker/image-size/andreper/spark-master/latest) | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-master) |
-| **[Spark Worker](https://hub.docker.com/r/andreper/spark-worker)** | ![docker-size](https://img.shields.io/docker/image-size/andreper/spark-worker/latest) | ![docker-pull](https://img.shields.io/docker/pulls/andreper/spark-worker) |
-
 ## <a name="contributors"></a>Contributors
 
  - **André Perez** - [dekoperez](https://twitter.com/dekoperez) - [email protected]