CID/CI-and-CD-Plan

New Plan for Continuous Integration and Delivery

In each p≡p repo we shall have a CI manifest file (eg. .gitlab-ci.yml, .build.yml, etc.) which defines a build pipeline containing the following jobs, which run on each push to the remote:

  1. Build from current HEAD of repo for defined targets (eg. Debian, RHEL, MacOS Catalina)
  2. Test the builds in respective target environments

IF tests succeed continue to:

  1. Publish binaries to a remote bin depot
  2. Generate packages for defined distros/OSes
  3. Verify and test package installation in respective distros/OSes
  4. Run end-to-end tests in defined distros/OSes eg. test adapter build against HEAD of Engine, and latest release of Engine

IF all succeeds then finally:

  1. Push binaries and packages to a depot, and if applicable push docker images to our image registry

Published Artifacts

Binaries, packages, and docker images should all be made publicly accessible.

This will allow for 3rd parties (included p≡p security) to use these artifacts for development, as well as in their own CI pipelines.

Infrastructure we need for this

  • [x] Self-hosted Gitlab instance
  • [ ] Build Server
  • [ ] hg => git
  • [ ] Build artifact depot
  • [ ] Docker image registry (eg. Docker Registry 2.0)
  • [ ] Monitoring

Build Server

Utilizing the Custom Executor in a Gitlab Runner we are able to run builds inside of QEMU/KVM.

This allows us to target multiple OSes, distros, and CPU architectures.

See this repo for an example setup from dvn, defined in an Ansible Playbook.

hg 2 git

For conversion from Mercurial to Git, hg2git has been successfully used to convert pEpJSONServerAdapter to Git: https://pep-security.lu/gitlab/cid/pEpJSONServerAdapter/

Artifact Depot

We can use a simple http server with SSL to host our binaries and packages. This should be a (virtual) host dedicated to solely this purpose. It must have ssh and rsync, so that the build environments can upload build artifacts.

We will populate build environments with an authorized ssh key via Gitlab CI/CD Environment Variables

TODO

  • Discuss reproducible builds
  • Address package and binary signing
  • Security & threat-model assessment