Support C++ (#130)

* Implemented c++ pipeline execution. Build is still missing.

* Implemented build of c++ pipelines

* Fine tuning and some small bug fixes

* Fixed two failing tests

* Added test for scheduler

* Tiny UI improvement

* Added c++ dockerfile and updated goreleaser. Updated readme (partially). Removed mapping of variables in docker-entrypoint file

* Updated readme file again

* Fixed incorrect dockerfils

Author: michelvocks

.goreleaser.yml

@@ -32,17 +32,32 @@ dockers:
     skip_push: false
     binary: gaia-linux-amd64
     dockerfile: docker/Dockerfile.golang
+    extra_files:
+      - docker/docker-entrypoint.sh
     tag_templates:
     - "{{ .Tag }}-go"
   - image: gaiapipeline/gaia
     skip_push: false
     binary: gaia-linux-amd64
     dockerfile: docker/Dockerfile.java
+    extra_files:
+      - docker/docker-entrypoint.sh
+      - docker/settings-docker.xml
     tag_templates:
     - "{{ .Tag }}-java"
   - image: gaiapipeline/gaia
     skip_push: false
     binary: gaia-linux-amd64
     dockerfile: docker/Dockerfile.python
+    extra_files:
+      - docker/docker-entrypoint.sh
+    tag_templates:
+    - "{{ .Tag }}-python"
+  - image: gaiapipeline/gaia
+    skip_push: false
+    binary: gaia-linux-amd64
+    dockerfile: docker/Dockerfile.cpp
+    extra_files:
+      - docker/docker-entrypoint.sh
     tag_templates:
-    - "{{ .Tag }}-python"
+    - "{{ .Tag  }}-cpp"

README.rst

@@ -61,6 +61,25 @@ Installation
 
 The installation of gaia is simple and often takes a few minutes.
 
+Using docker
+~~~~~~~~~~~~
+
+The following command starts gaia as a daemon process and mounts all data to the current folder. Afterwards, gaia will be available on the host system on port 8080. Use the standard user **admin** and password **admin** as initial login. It is recommended to change the password afterwards.
+
+.. code:: sh
+
+    docker run -d -p 8080:8080 -v $PWD:/data gaiapipeline/gaia:latest
+
+This uses the image with the *latest* tag which includes all required libraries and compilers for all supported languages. If you prefer a smaller image suited for your prefered language, have a look at the `available docker image tags`_.
+
+Manually
+~~~~~~~~
+
+It is possible to install Gaia directly on the host system.
+This can be achieved by downloading the binary from the `releases page`_.
+
+Gaia will automatically detect the folder of the binary and will place all data next to it. You can change the data directory with the startup parameter *--homepath* if you want.
+
 Using helm
 ~~~~~~~~~~
 
@@ -82,25 +101,6 @@ To deploy gaia:
 
     make deploy-kube
 
-Using docker
-~~~~~~~~~~~~
-
-The following command starts gaia as a daemon process and mounts all data to the current folder. Afterwards, gaia will be available on the host system on port 8080. Use the standard user **admin** and password **admin** as initial login. It is recommended to change the password afterwards.
-
-.. code:: sh
-
-    docker run -d -p 8080:8080 -v $PWD:/data gaiapipeline/gaia:latest
-
-This uses the image with the *latest* tag which includes all required libraries and compilers for all supported languages. If you prefer a smaller image suited for your prefered language, have a look at the `available docker image tags`_.
-
-Manually
-~~~~~~~~
-
-It is possible to install Gaia directly on the host system.
-This can be achieved by downloading the binary from the `releases page`_.
-
-Gaia will automatically detect the folder of the binary and will place all data next to it. You can change the data directory with the startup parameter *--homepath* if you want.
-
 Usage
 -----
 
@@ -196,6 +196,37 @@ Java
         }
     }
 
+C++
+~~~~
+
+.. code:: cpp
+
+   #include "cppsdk/sdk.h"
+   #include <list>
+   #include <iostream>
+
+   void DoSomethingAwesome(std::list<gaia::argument> args) throw(std::string) {
+      std::cerr << "This output will be streamed back to gaia and will be displayed in the pipeline logs." << std::endl;
+
+      // An error occured? Return it back so gaia knows that this job failed.
+      // throw "Uhh something badly happened!"
+   }
+
+   int main() {
+      std::list<gaia::job> jobs;
+      gaia::job awesomejob;
+      awesomejob.handler = &DoSomethingAwesome;
+      awesomejob.title = "DoSomethingAwesome";
+      awesomejob.description = "This job does something awesome.";
+      jobs.push_back(awesomejob);
+
+      try {
+         gaia::Serve(jobs);
+      } catch (string e) {
+         std::cerr << "Error: " << e << std::endl;
+      }
+   }
+
 Pipelines are defined by jobs and a function usually represents a job. You can define as many jobs in your pipeline as you want.
 
 Every function accepts arguments. Those arguments can be requested from the pipeline itself and the values passed back in from the UI.
@@ -237,7 +268,7 @@ The SDK implements the Gaia plugin gRPC interface and offers helper functions li
 
 Which programming languages are supported?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-We currently fully support Golang, Java and Python.
+We currently fully support Golang, Java, Python and C++.
 
 When do you support programming language **XYZ**?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docker/Dockerfile

@@ -80,7 +80,7 @@ RUN ln -svT "/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" /docker-j
 ENV JAVA_HOME /docker-java-home
 
 ENV JAVA_VERSION 8u181
-ENV JAVA_DEBIAN_VERSION 8u181-b13-1~deb9u1
+ENV JAVA_DEBIAN_VERSION 8u181-b13-2~deb9u1
 
 # see https://bugs.debian.org/775775
 # and https://github.com/docker-library/java/issues/19#issuecomment-70546872
@@ -131,6 +131,26 @@ ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
 COPY settings-docker.xml /usr/share/maven/ref/
 # --------------- End Maven Part ---------------
 
+# --------------- Start C++ Part ---------------
+RUN apt-get update && apt-get install -y \
+            build-essential autoconf git pkg-config \
+            automake libtool curl make g++ unzip \
+            && apt-get clean
+
+# install protobuf first, then grpc
+ENV GRPC_RELEASE_TAG v1.16.x
+RUN git clone -b ${GRPC_RELEASE_TAG} https://github.com/grpc/grpc /var/local/git/grpc && \
+                  cd /var/local/git/grpc && \
+    git submodule update --init && \
+    echo "--- installing protobuf ---" && \
+    cd third_party/protobuf && \
+    ./autogen.sh && ./configure --enable-shared && \
+    make -j$(nproc) && make install && make clean && ldconfig && \
+    echo "--- installing grpc ---" && \
+    cd /var/local/git/grpc && \
+    make -j$(nproc) && make install && make clean && ldconfig
+# --------------- Start C++ Part ---------------
+
 # install additional deps
 RUN apt-get update && apt-get install -y --no-install-recommends \
 		python2.7-dev \

docker/Dockerfile.cpp

@@ -0,0 +1,44 @@
+FROM debian:stretch
+
+RUN apt-get update && apt-get install -y \
+    build-essential autoconf git pkg-config \
+    automake libtool curl make g++ unzip \
+    && apt-get clean
+
+# install protobuf first, then grpc 
+ENV GRPC_RELEASE_TAG v1.16.x 
+RUN git clone -b ${GRPC_RELEASE_TAG} https://github.com/grpc/grpc /var/local/git/grpc && \
+	            cd /var/local/git/grpc && \
+    git submodule update --init && \
+    echo "--- installing protobuf ---" && \
+    cd third_party/protobuf && \
+    ./autogen.sh && ./configure --enable-shared && \
+    make -j$(nproc) && make install && make clean && ldconfig && \
+    echo "--- installing grpc ---" && \
+    cd /var/local/git/grpc && \
+    make -j$(nproc) && make install && make clean && ldconfig
+
+# Gaia internal port and data path.
+ENV GAIA_PORT=8080 \
+    GAIA_HOMEPATH=/data
+
+# Directory for the binary
+WORKDIR /app
+
+# Copy gaia binary into docker image
+COPY gaia-linux-amd64 /app
+
+# Fix permissions 
+RUN chmod +x ./gaia-linux-amd64
+
+# Set homepath as volume
+VOLUME [ "${GAIA_HOMEPATH}" ]
+
+# Expose port
+EXPOSE ${GAIA_PORT}
+
+# Copy entry point script
+COPY docker-entrypoint.sh /usr/local/bin/
+
+# Start gaia
+ENTRYPOINT [ "docker-entrypoint.sh" ]

docker/docker-entrypoint.sh

@@ -1,4 +1,4 @@
 #!/usr/bin/env sh
 
 # Start gaia
-exec /app/gaia-linux-amd64 --port=${GAIA_PORT} --homepath=${GAIA_HOMEPATH} --worker=${GAIA_WORKER}
+exec /app/gaia-linux-amd64

frontend/client/assets/cplusplus.png frontend/client/assets/cpp.png

@@ -9,7 +9,7 @@
           <div class="outer-box">
             <router-link :to="{ path: '/pipeline/detail', query: { pipelineid: pipeline.p.id }}" class="hoveraction">
               <div class="outer-box-icon-image">
-                <img :src="getImagePath(pipeline.p.type)" class="outer-box-image" v-bind:class="{ 'outer-box-image-python': pipeline.p.type === 'python', 'outer-box-image': pipeline.p.type !== 'python' }">
+                <img :src="getImagePath(pipeline.p.type)" v-bind:class="pipelineImageClass(pipeline.p.type)">
               </div>
               <div>
                 <span class="subtitle">{{ pipeline.p.name }}</span>
@@ -103,6 +103,14 @@ export default {
         })
     },
 
+    pipelineImageClass (type) {
+      return {
+        'outer-box-image-python': type === 'python',
+        'outer-box-image-cpp': type === 'cpp',
+        'outer-box-image': type !== 'python' && type !== 'cpp'
+      }
+    },
+
     checkPipelineArgs (pipeline) {
       this.pipeline = pipeline
 
@@ -259,6 +267,15 @@ export default {
   transform: translate(-50%, -50%);
 }
 
+.outer-box-image-cpp {
+  position: absolute;
+  width: 27px;
+  height: 30px;
+  top: 50%;
+  left: 50%;
+  transform: translate(-50%, -50%);
+}
+
 .hoveraction:hover .outer-box-icon-image {
   border-color: #4da2fc !important;
 }

frontend/client/views/overview/index.vue

@@ -77,8 +77,8 @@
               </div>
             </div>
             <div class="content" style="display: flex;">
-              <div class="pipelinetype" title="C++ (not yet supported)" v-tippy="{ arrow : true,  animation : 'shift-away'}" v-bind:class="{ pipelinetypeactive: createPipeline.pipeline.type === 'cplusplus' }" data-tippy-hideOnClick="false">
-                <img src="~assets/cplusplus.png" class="typeimage typeimagenotyetsupported">
+              <div class="pipelinetype" title="C++" v-tippy="{ arrow : true,  animation : 'shift-away'}" v-on:click="createPipeline.pipeline.type = 'cpp'" v-bind:class="{ pipelinetypeactive: createPipeline.pipeline.type === 'cpp' }" data-tippy-hideOnClick="false">
+                <img src="~assets/cpp.png" class="typeimage">
               </div>
               <div class="pipelinetype" title="Node.js (not yet supported)" v-tippy="{ arrow : true,  animation : 'shift-away'}" v-bind:class="{ pipelinetypeactive: createPipeline.pipeline.type === 'nodejs' }" data-tippy-hideOnClick="false">
                 <img src="~assets/nodejs.png" class="typeimage typeimagenotyetsupported">
@@ -108,7 +108,7 @@
                   <div v-else-if="props.row.statustype === 'success'" style="color: green;">{{ props.row.statustype }}</div>
                   <div v-else style="color: red;">{{ props.row.statustype }}</div>
                 </td>
-                <td>{{ props.row.pipeline.type }}</td>
+                <td>{{ prettifyPipelineType(props.row.pipeline.type) }}</td>
                 <td :title="props.row.created" v-tippy="{ arrow : true,  animation : 'shift-away'}">{{ convertTime(props.row.created) }}</td>
                 <td>
                   <a class="button is-green-button is-small" @click="showStatusOutputModal(props.row.output)">
@@ -529,6 +529,13 @@ export default {
       this.statusOutputModal = false
       this.statusOutputMsg = ''
       this.$emit('close')
+    },
+
+    prettifyPipelineType (type) {
+      if (type === 'cpp') {
+        return 'C++'
+      }
+      return type
     }
   }
 }

frontend/client/views/pipeline/create.vue

@@ -34,6 +34,9 @@ const (
 	// PTypePython python plugin type
 	PTypePython PipelineType = "python"
 
+	// PTypeCpp C++ plugin type
+	PTypeCpp PipelineType = "cpp"
+
 	// CreatePipelineFailed status
 	CreatePipelineFailed CreatePipelineType = "failed"
 
@@ -90,6 +93,9 @@ const (
 
 	// TmpGoFolder is the name of the golang temporary folder
 	TmpGoFolder = "golang"
+
+	// TmpCppFolder is the name of the c++ temporary folder
+	TmpCppFolder = "cpp"
 )
 
 // User is the user object

gaia.go

@@ -33,7 +33,7 @@ func UserLogin(c echo.Context) error {
 	// Authenticate user
 	user, err := storeService.UserAuth(u, true)
 	if err != nil || user == nil {
-		gaia.Cfg.Logger.Error("invalid credentials provided", "message", err.Error())
+		gaia.Cfg.Logger.Info("invalid credentials provided", "username", u.Username)
 		return c.String(http.StatusForbidden, "invalid username and/or password")
 	}