Skip to content

Improve CI/CD for simcore and add staging/production stages #738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Apr 30, 2019

What do these changes do?

  • Improve build speed on branches:
    if necessary travis env variables are set, the docker images will be pushed to dockerhub:
    • travis first build the docker images
    • travis pushes the images to dockerhub
    • travis pulls the image for integration-testing/system-testing which improves speed a lot on branch push
    • on pull-request builds, the build will still occur for integration/system-testing but will still use the cache from itisfoundation official images to speed-up
  • implements git release workflow:
    • travis pushes the images to dockerhub with appropriate tags if on master branch
    • travis pushes the images to dockerhub with appropriate tags if on staging branch
    • travis pulls staging image and pushes to production if on a git tag is present (no build, no test, it takes what was tested in staging)
  • documentation for release workflow added in README.md file

Bonus:

  • refactoring of Makefile
  • moved simcore-sdk tests to integration test suite as they use the swarm (would need to be further divided in unit/integration suite)
  • fixed some pytest warnings
  • fixes upstream branches to fix master builds #551

Related issue number

#713

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • Runs in the swarm
  • If you design a new module, add your user to .github/CODEOWNERS

… renamed to more generic pullstaging-latest tag not used at build time but tagged after testing only
@sanderegg sanderegg changed the title WIP-Improve CI/CD for simcore and add staging/production stages Improve CI/CD for simcore and add staging/production stages May 15, 2019
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI is getting serious stuff ! Well done 👍

@ghost
Copy link

ghost commented May 15, 2019

DeepCode analyzed this pull request.
There is 1 new info report.

Click to see more details.

Copy link
Member

@mguidon mguidon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I am considering to use the travis encryption stuff to test the datcore API.

@sanderegg sanderegg requested a review from pcrespov May 16, 2019 07:15
@sanderegg sanderegg merged commit 023a767 into ITISFoundation:master May 16, 2019
@sanderegg sanderegg deleted the is713/rename_staging_to_master branch May 16, 2019 11:47
sanderegg added a commit that referenced this pull request May 22, 2019
* Delete settings.json

* Fixes for alpha tester (#748)

* fake-template-projects.json created. 2 templates moved here
* Kember use case added
* MattWard use case added
* 4x CC 0D added
* Provide a way to refetch data from DataManager
* connect the kember-model to the kember-viewer
* moved null handler to node_ports package
* set node ports log level to same as sidecar
* removed Unused import logging
* New issue: add UX_improvemnts label and asign it to oSPARC Product Backlog project

* Permission policies (#731)

* LinkButton and LinkLabel moved
* Project -> Study
* default hardcoded permissions added
* Added some restrictions
* Link -> Edge
* Moved permission checks to data level
* Set Default Role
* more permission checks
* token permission checks
* datcore read permission check
* study.node.data.* permission check
* study.node.rename permission check
* services.filtered.read permission check
* fixed plotly minor bug
* Test permissions
* Moved some removeNode code to Workbench
* Moved removeEdge code to Workbench
* test edge create/delete
* test node rename
* deserializeNodes and deserializeEdges added
* deserializeEdges fixed for 'outputNode's
* got rid of canStart
* Typo: presgined -> presigned
* do not allow anonymous users to add/delete tokens
* do not allow anonymous users to update user profile

* Frontend privileges. 2nd iteration (#753)

* help points to our itisfoundation.github.io
* typo
* Do not show Debug level in logger to User and Anon
* Do not show nodeIds in NodesTree to User and Anon
* Tester and higher level roles can switch their role to a lower one

* Is730/serve studies (#747)

Implements a first prototype for issue #730 

Adds new subsystem ``services/web/server/src/simcore_service_webserver/studies_access.py``
 - Open services/web/server/tests/sandbox/*.md and click any link
 - If user is logged in, it copies the project and displays it
 - if anonymous, it creates a temporary user and does the previous step
 - Merges changes from PR #734 that contains the UI part.
   - queries id in url and opens page with specified study
   - fixes log out
   - preview of permissions (client-side only)

Next step will incorporate privileges and user roles on the server side.

* Quick fixes (#742)

A couple of iframe bugs fixed:

Maximizing an iframe now works.
We can now resize a SidePanel over an iframe
Other bugs:

File picker bug fixed. #741

* Update study_access_demo.md

Added link to actual emulator [skip travis-ci]

* Logger context dependant (#754)

* replace label by nodeId in logger
* Bring back the update page alert window
* added link to version/commit to login page
* startInteractiveNode out from populateNodeData
* Log message changes when label is changed
* Logger node context dependant

* Is234/compile requirements (#733)

Implements #234 : normalizes dependencies for python packages in osparc-simcore repository

- documented (see ``docs/python-dependencies.md``) new workflow with python requirements
- updates ``packages`` and ``services`` 
- updates ``ops\travis`` CI
   - moves out a part of unittest ``simcore-sdk`` into a new unit-test for ``service-library``
   - installation is always performed as ``pip install -r requirements/ci.txt ``
   -  @mguidon,  sidecar as NO tests (created ``ops\tracis`` but disabled)!

* Fixed makefile for all OS
* Fixing storage tests:
 - downgraded aiopg until bug is fixed
* Fixes overlapping port
* Enhaces doc
* Downgraded aio-pika
 - avoid test-errors in webserver integration!
 - cleanup

* Fix input node's title (#771)

* fix #770

* Improve CI/CD for simcore and add staging/production stages (#738)

build docker images once and push to dockerhub
pull from dockerhub for testing
implement release workflow

* Is788/allow different stack name (#789)

* director/sidecar independant of stack name using SWARM_STACK_NAME
* allow choosing port where simcore platform is available

* fixed link to image

* Maintenance/update requirements (#792)

* Upstead api/tests requirements
* Updates travis system-testing requirements
* Updated s3wrapper requirements
* update servicelib requirements
* Updates simcore-sdk requirements and with this all packages requirements
* Updates apihub/tests requirements
* Fixes missing ops/travis requirements
* Updates director
* Updates sidecar
* updated storage

* Fix: 2 file management bugs (#782)

* getLowerRoles fixed
* CopyFile returns true/false if request sent + test
* DeleteFile returns true/false if request sent + test
* getChildrenRoles by checking inheritance
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

upstream branches to fix master builds
4 participants