Merge branch 'master' of https://github.com/gaia-pipeline/gaia into issue-30

Author: bidhan-a

README.rst

@@ -4,13 +4,9 @@
 
 |build-status| |go-report| |go-doc| |apache2| |chat| |codecov|
 
-gaia is an open source automation platform which makes it easy and fun
-to build powerful pipelines in any programming language. Based on 
-`HashiCorp's go-plugin`_ and `gRPC`_, gaia is efficient, fast, lightweight
-and developer friendly. Gaia is currently alpha! `Do not use it for mission 
-critical jobs yet!`_
+Gaia is an open source automation platform which makes it easy and fun to build powerful pipelines in any programming language. Based on `HashiCorp's go-plugin`_ and `gRPC`_, gaia is efficient, fast, lightweight and developer friendly. Gaia is currently alpha! `Do not use it for mission critical jobs yet!`_
 
-Develop pipelines with the help of SDKs (currently only Go) and simply check-in your code into a git repository. Gaia automatically clones your code repository, compiles your code to a binary and executes it on-demand. All results are streamed back and formatted to a user-friendly graphical output.
+Develop powerful `pipelines <What is a pipeline?_>`_ with the help of `SDKs <Why do I need an SDK?_>`_ (currently only Go) and simply check-in your code into a git repository. Gaia automatically clones your code repository, compiles your code to a binary and executes it on-demand. All results are streamed back and formatted to a user-friendly graphical output.
 
 Motivation
 ==========
@@ -23,18 +19,20 @@ The majority of tech people are not motivated to take up this work and they are
 
 One of the main reasons for this is the abstraction and poor execution of many automation tools. They come with their own configuration (`YAML`_ syntax) specification or limit the user to one specific programming language. Testing is nearly impossible because most automation tools lack the ability to mock services and subsystems. Even tiny things, for example parsing a JSON file, are sometimes really painful because external, outdated libraries were used and not included in the standard framework.
 
-We believe it's time to remove all these abstractions and come back to our roots. Are you tired of writing endless lines of YAML-code? Are you sick of spending days forced to write in a language that does not suit you and is not fun at all? Do you enjoy programming in a language you like? Then gaia is for you.
+We believe it's time to remove all these abstractions and come back to our roots. Are you tired of writing endless lines of YAML-code? Are you sick of spending days forced to write in a language that does not suit you and is not fun at all? Do you enjoy programming in a language you like? Then Gaia is for you.
 
 How does it work?
 =================
 
 .. begin-architecture
 
-Gaia is based on `HashiCorp's go-plugin`_. It's a plugin system that uses `gRPC`_ to communicate over HTTP2. HashiCorp developed this tool initially for `Packer`_ but it's now heavily used by `Terraform`_, `Nomad`_, and `Vault`_ too.
+Gaia is based on `HashiCorp's go-plugin`_. It's a plugin system that uses `gRPC`_ to communicate over `HTTP/2`_. HashiCorp developed this tool initially for `Packer`_ but it's now heavily used by `Terraform`_, `Nomad`_, and `Vault`_ too. 
 
-Pipelines can be written in any programming language (gRPC support is a prerequisite) and can be compiled locally or simply over the build system. Gaia clones the git repository and automatically builds the included pipeline. 
+Plugins, which we named pipelines, are applications which can be written in any programming language as long as `gRPC`_ is supported. All functions, which we call Jobs, are exposed to Gaia and can form up a dependency graph which describes the order of execution.
 
-After a pipeline has been started, all log output from the included jobs are returned back to gaia and displayed in a detailed overview with their final result status.
+Pipelines can be compiled locally or simply over the build system. Gaia clones the git repository and automatically builds the included pipeline. If a change (`git push`_) happened, Gaia will automatically rebuild the pipeline for you.  
+
+After a pipeline has been started, all log output are returned back to Gaia and displayed in a detailed overview with their final result status.
 
 Gaia uses `boltDB` for storage. This makes the installation step super easy. No external database is currently required.
 
@@ -134,10 +132,35 @@ Gaia will compile it and add it to it's store for later execution.
 
 Please find a bit more sophisticated example in our `go-example repo`_.
 
-Documentation
-=============
+Documentation and more
+======================
+
+Please find the docs at https://docs.gaia-pipeline.io. We also have a tutorials section over there with examples and real use-case scenarios. For example, `Kubernetes deployment with vault integration`_.
+
+Questions and Answers (Q&A)
+---------------------------
+
+What problem solves **Gaia**?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Literally every tool which were designed for automation, continuous integration (CI), and continuous deployment (CD) like Spinnaker, Jenkins, Gitlab CI/CD, TravisCI, CircleCI, Codeship, Bamboo and many more, introduced their own configuration format. Some of them don't even support *configuration/automation as code*. This works well for simple tasks like running a ``go install`` or ``mvn clean install`` but in the real world there is more to do.
+
+Gaia is the first platform which does not limit the user and provides full support for almost all common programming languages without losing the features offered by todays CI/CD tools.    
+
+What is a **pipeline**?
+~~~~~~~~~~~~~~~~~~~~~~~
+A pipeline is a real application with at least one function (we call it Job). Every programming language can be used as long as gRPC is supported. We offer SDKs (currently only Go but others are already in development) to support the development. 
+
+What is a **job**?
+~~~~~~~~~~~~~~~~~~
+A job is a function, usually globally exposed to Gaia. Dependent on the dependency graph, Gaia will execute this function in a specific order.
+
+Why do I need an **SDK**?
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+The SDK implements the Gaia plugin gRPC interface and offers helper functions like serving the gRPC-Server. This helps you to focus on the real problem instead of doing the boring stuff.
 
-Please find the docs at https://docs.gaia-pipeline.io. We also have a interesting tutorials section over there. For example, `Kubernetes deployment with vault integration`_.   
+When do you support programming language **XYZ**?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+We are working hard to support as much programming languages as possible but our resources are limited and we are also mostly no experts in all programming languages. If you are willing to contribute, feel free to open an issue and start working.
 
 Roadmap
 =======
@@ -182,6 +205,8 @@ If you have any questions feel free to contact us on `gitter`_.
 .. _`go-example repo`: https://github.com/gaia-pipeline/go-example
 .. _`gitter`: https://gitter.im/gaia-pipeline
 .. _`Kubernetes deployment with vault integration`: https://docs.gaia-pipeline.io/tutorials/kube-vault-deploy/
+.. _`git push`: https://git-scm.com/docs/git-push
+.. _`HTTP/2`: https://http2.github.io/
 
 .. |build-status| image:: https://circleci.com/gh/gaia-pipeline/gaia/tree/master.svg?style=shield&circle-token=c0e15edfb08f8076076cbbb55558af6cfecb89b8
     :alt: Build Status

pipeline/build_golang_test.go

@@ -1,6 +1,7 @@
 package pipeline
 
 import (
+	"bytes"
 	"context"
 	"fmt"
 	"io/ioutil"
@@ -112,10 +113,10 @@ func TestExecuteBuildFailPipelineBuild(t *testing.T) {
 	}()
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = tmp
-	var logOutput strings.Builder
+	buf := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &logOutput,
+		Output: buf,
 		Name:   "Gaia",
 	})
 	b := new(BuildPipelineGolang)
@@ -142,10 +143,10 @@ func TestExecuteBuildContextTimeout(t *testing.T) {
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = tmp
 	// Initialize shared logger
-	var logOutput strings.Builder
+	buf := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &logOutput,
+		Output: buf,
 		Name:   "Gaia",
 	})
 	b := new(BuildPipelineGolang)
@@ -164,10 +165,10 @@ func TestExecuteBuildBinaryNotFoundError(t *testing.T) {
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = tmp
 	// Initialize shared logger
-	var logOutput strings.Builder
+	buf := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &logOutput,
+		Output: buf,
 		Name:   "Gaia",
 	})
 	currentPath := os.Getenv("PATH")
@@ -189,10 +190,10 @@ func TestCopyBinary(t *testing.T) {
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = tmp
 	// Initialize shared logger
-	var logOutput strings.Builder
+	buf := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &logOutput,
+		Output: buf,
 		Name:   "Gaia",
 	})
 	b := new(BuildPipelineGolang)
@@ -224,10 +225,10 @@ func TestCopyBinarySrcDoesNotExist(t *testing.T) {
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = tmp
 	// Initialize shared logger
-	var logOutput strings.Builder
+	buf := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &logOutput,
+		Output: buf,
 		Name:   "Gaia",
 	})
 	b := new(BuildPipelineGolang)

pipeline/git_test.go

@@ -1,6 +1,7 @@
 package pipeline
 
 import (
+	"bytes"
 	"os"
 	"strconv"
 	"strings"
@@ -28,10 +29,10 @@ func TestUpdateAllPipelinesRepositoryNotFound(t *testing.T) {
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = tmp
 	// Initialize shared logger
-	var b strings.Builder
+	b := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &b,
+		Output: b,
 		Name:   "Gaia",
 	})
 
@@ -49,10 +50,10 @@ func TestUpdateAllPipelinesAlreadyUpToDate(t *testing.T) {
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = "tmp"
 	// Initialize shared logger
-	var b strings.Builder
+	b := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &b,
+		Output: b,
 		Name:   "Gaia",
 	})
 	repo := &gaia.GitRepo{
@@ -82,10 +83,10 @@ func TestUpdateAllPipelinesAlreadyUpToDateWithMoreThanOnePipeline(t *testing.T)
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = "tmp"
 	// Initialize shared logger
-	var b strings.Builder
+	b := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &b,
+		Output: b,
 		Name:   "Gaia",
 	})
 	repo := &gaia.GitRepo{
@@ -124,10 +125,10 @@ func TestUpdateAllPipelinesHundredPipelines(t *testing.T) {
 	gaia.Cfg = new(gaia.Config)
 	gaia.Cfg.HomePath = "tmp"
 	// Initialize shared logger
-	var b strings.Builder
+	b := new(bytes.Buffer)
 	gaia.Cfg.Logger = hclog.New(&hclog.LoggerOptions{
 		Level:  hclog.Trace,
-		Output: &b,
+		Output: b,
 		Name:   "Gaia",
 	})
 	repo := &gaia.GitRepo{

security/tls.go

@@ -0,0 +1,164 @@
+package security
+
+import (
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"io/ioutil"
+	"math/big"
+	"os"
+	"path/filepath"
+	"time"
+
+	"github.com/gaia-pipeline/gaia"
+)
+
+const (
+	rsaBits      = 2048
+	maxValidCA   = 17520 // 2 years
+	maxValidCERT = 48    // 48 hours
+	orgName      = "gaia-pipeline"
+	orgDNS       = "gaia-pipeline.io"
+
+	// CA key name
+	certName = "ca.crt"
+	keyName  = "ca.key"
+)
+
+// GenerateCA generates the CA and puts it into the data folder.
+// The CA will be always overwritten on startup.
+func GenerateCA() error {
+	// Cleanup old certs if existing.
+	// We ignore the error here cause files might be non existend.
+	caCertPath := filepath.Join(gaia.Cfg.DataPath, certName)
+	caKeyPath := filepath.Join(gaia.Cfg.DataPath, keyName)
+	cleanupCerts(caCertPath, caKeyPath)
+
+	// Set time range for cert validation
+	notBefore := time.Now()
+	notAfter := notBefore.Add(time.Hour * maxValidCA)
+
+	// Generate serial number
+	serialNumberLimit := new(big.Int).Lsh(big.NewInt(1), 128)
+	serialNumber, err := rand.Int(rand.Reader, serialNumberLimit)
+	if err != nil {
+		return err
+	}
+
+	// Generate CA template
+	template := &x509.Certificate{
+		SerialNumber: serialNumber,
+		Subject: pkix.Name{
+			Organization: []string{orgName},
+		},
+		NotBefore: notBefore,
+		NotAfter:  notAfter,
+
+		IsCA:                  true,
+		KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
+		BasicConstraintsValid: true,
+		DNSNames:              []string{orgDNS},
+	}
+
+	// Generate the key
+	key, err := rsa.GenerateKey(rand.Reader, rsaBits)
+	if err != nil {
+		return err
+	}
+
+	// Create certificate authority
+	derBytes, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key)
+	if err != nil {
+		return err
+	}
+
+	// Write out the ca.crt file
+	certOut, err := os.Create(caCertPath)
+	if err != nil {
+		return err
+	}
+	pem.Encode(certOut, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes})
+	certOut.Close()
+
+	// Write out the ca.key file
+	keyOut, err := os.OpenFile(caKeyPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
+	if err != nil {
+		return err
+	}
+	pem.Encode(keyOut, &pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key)})
+	keyOut.Close()
+
+	return nil
+}
+
+// createSignedCert creates a new key pair which is signed by the CA.
+func createSignedCert() (string, string, error) {
+	caCertPath := filepath.Join(gaia.Cfg.DataPath, "ca.crt")
+	caKeyPath := filepath.Join(gaia.Cfg.DataPath, "ca.key")
+
+	// Load CA plain
+	caPlain, err := tls.LoadX509KeyPair(caCertPath, caKeyPath)
+	if err != nil {
+		return "", "", err
+	}
+
+	// Parse certificate
+	ca, err := x509.ParseCertificate(caPlain.Certificate[0])
+	if err != nil {
+		return "", "", err
+	}
+
+	// Generate serial number
+	serialNumberLimit := new(big.Int).Lsh(big.NewInt(1), 128)
+	serialNumber, err := rand.Int(rand.Reader, serialNumberLimit)
+	if err != nil {
+		return "", "", err
+	}
+
+	// Set time range for cert validation
+	notBefore := time.Now()
+	notAfter := notBefore.Add(time.Hour * maxValidCERT)
+
+	// Prepare certificate
+	cert := &x509.Certificate{
+		SerialNumber: serialNumber,
+		Subject: pkix.Name{
+			Organization: []string{orgName},
+		},
+		NotBefore:    notBefore,
+		NotAfter:     notAfter,
+		SubjectKeyId: []byte{1, 2, 3, 4, 6},
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		DNSNames:     []string{orgDNS},
+	}
+	priv, _ := rsa.GenerateKey(rand.Reader, rsaBits)
+	pub := &priv.PublicKey
+
+	// Sign the certificate
+	certSigned, err := x509.CreateCertificate(rand.Reader, cert, ca, pub, caPlain.PrivateKey)
+
+	// Public key
+	certOut, err := ioutil.TempFile("", "crt")
+	pem.Encode(certOut, &pem.Block{Type: "CERTIFICATE", Bytes: certSigned})
+	certOut.Close()
+
+	// Private key
+	keyOut, err := ioutil.TempFile("", "key")
+	pem.Encode(keyOut, &pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(priv)})
+	keyOut.Close()
+
+	return certOut.Name(), keyOut.Name(), nil
+}
+
+// cleanupCerts removes certificates at the given path.
+func cleanupCerts(crt, key string) error {
+	if err := os.Remove(crt); err != nil {
+		return err
+	}
+	return os.Remove(key)
+}