CID/Internal Deployment

About this Document

This is a moderated document.
Please do not just edit!

If you like to contribute:

  • Preferrably, use the suggestions section at the very bottom
  • If not handy, Mark your edits with (TODO-yourname:)
  • Ask me if you want to become a co-maintainer, so you get full edit rights

Please feel free to get in touch with us:

  • IRC pEp Intern: dvn
  • Mail: dvn@pep-project.org

  • IRC pEp Intern: heck
  • Mail: heck@pep.foundation

Internal Deployment - Binary Packages

We need to build binary packages of adapter libraries for internal use. This document is about binary distributions only, and the platforms in scope are Linux, MacOS, and Windows only.

Code Repository

There is a git repository for this project. All source code relevant to this project needs to be maintained there.
If there are source code files referenced in this document, thats where you find them.

https://pep-security.lu/gitlab/cid/Internal-Deployment/

JIRA

Also, there is a separate JIRA project “Internal Deployment”.

https://pep.foundation/jira/projects/INDE/

Requirements For All Distributions

  • arch:
    • x86_64
    • x86_32 (i686) on linux only. Win ??? TBD
  • package design:
    1. in case of package manager (deb, rpm), there must be a package design
    2. in case of pypi installation (python wheel) it must be self-contained
    3. in case of MacOS .pkg it must contain everything, but must be able to update

Distribution Specific Requirements

In order to be able to define a distribution, the following aspects have to be clarified:

  • Adapter
  • Platform identifier (incl. edition, version) (e.g RHEL7, Debian10 stable)
  • Target language version (e.g cpython3.7)
  • Package format (e.g rpm, wheel, deb)

Developing The Automated Package Build Process

When a distribution has been defined, the aim is to have a fully automated build process generating a package for that distribution that can be executed over and over again and against different versions of the software repositories, with little effort. The following steps describe the procedure, which will lead to that result.

Prototype The Package Build

This is the procedure to develop the process that needs to be carried out to build a binary package for a certain distribution.
Ideally, the package build process will be automated to the extent possible.
All the binaries need to be built on the exact target plattform.
The binary package build process must be:

  • Create/get base image (VM or container) for the build system with the exact specs of the target system (arch/OS(version)/python(version)/syslibs etc..).
  • Install common system dependencies in the most native way on this platform. (e.g apt-get/yum/.pkg/macports/.msi/.exe)
  • Build the whole pEp-base on this machine.

    • Sequoia
    • yml2
    • libetpan-fdik
    • asn1c
    • pEpEngine
  • Build the Adapter to be packaged
  • Build the package using the resulting binaries.

The result of this process should be:

  • A base image (VM or docker)
  • build instructions to manually build the package.

Automate The Package Build

Once the build instructions are verified to work manually for a certain distribution, the steps can be automated. To automate the package build process:

  • Make the baseimage (VM/docker) available online.
  • Transform the manual build instruction into scripts/dockerfile (whatever is applicable platform-dependent)
  • Create documentation on how to run this automated build.
  • Make everything available in the Git repo: https://pep-security.lu/gitlab/cid/internal-deployment

Running the automated build should look something like this

  • Get VM/docker base image (derived from process above)
  • Execute package build scripts (shell/dockerfile (platform-dependent))

pEpPythonAdapter

Ticket: https://pep.foundation/jira/browse/PYADPT-62

Requirements

The requirement for the pEpPythonAdapter binary pkg are as follows:

  • Target Platforms:
    • RHEL 6,7,8
    • Debian 10
    • MacOS >= 10.14
    • Win Versions TBD.
  • Python Versions: 3.6, 3.7

  • PKG formats:
    • Linux
      • Python Wheel
      • rpm
      • deb
    • MacOS
      • Python Wheel
      • MacPorts
      • .pkg
    • Win
      • Python Wheel
      • TBD

Current state

Lets track the current state of the distributions here. Distributions are characterized by adapter/dist-format/platform/runtime-dep-versions. If there is too much documentation for a particular distribution, please create a new wiki page an link from here.
If a distribution is not applicable (e.g python 3.7 is not easily possible on CentOS 7 ), please not down why it is not applicable so we can justify whether to support it or not.

Debian 10 / Wheel / Python3.7

Supported.

Package Build Instructions

Create a base image:

  • Debian 10 amd64
  • Standard System Utilities
  • ssh server

Run these scripts:

  • install-sys-deps-debian.sh
  • build-pep-debian.sh
Pacakge Installation Instructions

The installation instructions are in the package .zip file.

Debian 10 / Wheel / Python3.6

CentOS 8 / Wheel / Python3.7

CentOS 8 / Wheel / Python3.6

CentOS 7 / Wheel / Python3.7

CentOS 7 / Wheel / Python3.6

CentOS 6 / Wheel / Python3.7

CentOS 6 / Wheel / Python3.6

MacOS 10.15 / Wheel / Python3.6

MacOS 10.15 / Wheel / Python3.7

MacOS 10.14 / Wheel / Python3.6

MacOS 10.14 / Wheel / Python3.7

pEpJNIAdapter

Ticket: https://pep.foundation/jira/browse/JNI-90

Requirements

The requirements nee to be clarified, please see ticket.

  • Target Platforms:
    • RHEL 6,7,8
    • Debian 10
    • MacOS >= 10.14
    • Win Versions TBD.
  • Java Versions: Java 8 TDB???

  • PKG formats:
    • Linux
      • zipfile ??? TDB
      • rpm
      • deb
    • MacOS
      • MacPorts
      • .pkg
    • Win
      • TBD

Appendix

Glossary

Distribution: A distribution is the package for a certain adapter/dist-format/platform/runtime-dep-versions.

IRCLogs

Please read these irc logs, they contain important information.

fdik - heck:

14:12 <        heck> | but again, case a)
14:12 <        heck> | from above package manager case
14:13 <        heck> | should i split the packages up into sequoia/engine/adapter ?
14:13 <        fdik> | One Package for SequoiaPGP, one for pEpEngine, one for pEpPythonAdapter
14:13 <        fdik> | yes
14:13 <        heck> | because we are not in the repo, i dont know wheter its a good idea because the package manager cant handle the deps
14:13 <        fdik> | pEpEngine including libetpan-fdik
14:14 <        fdik> | we're creating this to approach the repos
14:14 <        heck> | yes i asssumed that, enigne ships litetpan.so
14:14 <        heck> | ok, so then the enduser as a next step, will download 3 .rpm files (in a zip file)
14:14 <        fdik> | better rename it to libetpan-fdik to avoid conflicts
14:15 <        heck> | install sequoia.rpm
14:15 <        heck> | install engine.rpm
14:15 <        heck> | intall pythonadapter.rpm?
14:15 <        heck> | ?
14:16 <        heck> | sequoia.rpm will have 1 file in there sequoia_openpgp_ffi.so
14:17 <        heck> | engine will contaion pepengine.so/etpan.so
14:17 <        heck> | pythonadapter.rpm will contain pep.libboostpythonblabla.so
14:17 <        fdik> | for Sequoia you have to coordinate this with the Sequoia team in case you want to deploy it anywhere else but internally (or for PoCs with our customers)
 17:16 <        heck> | question is, do we need/want to install seq/engine system wide when installing the pEpPythonAdapter?
 17:16 <        fdik> | there is no concept for “installing system libs” on macOS
 17:16 <        fdik> | no, we usually don't
 17:17 <        fdik> | when using pip we don't for sure
 17:17 <        heck> | ok, then the user need to set LD_LIBRARY_PATH
 17:18 <        heck> | or we have to change/add the  RPATH of the pEp.cpython-37m-x86_64-linux-gnu.so
 17:18 <        fdik> | in macOS there is no LD_LIBRARY_PATH
 17:18 <        fdik> | this is Linux only
 17:19 <        fdik> | the shared library concept is different from OS to OS
 17:19 <        heck> | there is. DYNLD_LIBRARY_PATH (which only works with sip disabled)
 17:19 <        heck> | i know
 17:20 <        fdik> | you probably mean dyld, the dynamic linker
 17:20 <        fdik> | man 1 dyld
 17:20 <        heck> | exactly
 17:20 <        fdik> | in macOS we have a concept of Application Bundles and Frameworks
 17:21 <        heck> | in other words, without SIP disabled you cannot load a shared lib that is not installed system wide.
 17:21 <        fdik> | wrong
 17:21 <        heck> | ok, good to know
 17:21 <        fdik> | Please read the documentation about Applications Bundles and Frameworks, respectively
 17:21 <        heck> | ok
 17:22 <        fdik> | On Windows we have local DLLs, which are even preferred
 17:22 <        heck> | the solution will be, as you say, very platform specific, and at the moment my main focus is linux
 17:22 <        heck> | RHEL and deb
 17:22 <        fdik> | The only Linux system, which is actively used in a p≡p project, is Redhat
 17:22 <        fdik> | for p≡p for SWIFT
 17:22 <        heck> | let me see how to deal with mac later
 17:23 <        fdik> | round about 80% of the p≡p people are using Python on macOS
 17:23 <        heck> | yes thats my main target now
 17:23 <        fdik> | And the urgent need is to have it on Windoze
 17:23 <        heck> | what_?
 17:23 <        fdik> | yes.

#dev.log:

13:00 < heck> darthmama: regarding linker options for libpEpEngine.so, is there anything that prevents us from having RUNPATH=$ORIGIN set by default (for linux and mac build)?
13:03 < heck> darthmama: because this would allow us to have the whole pEp software stack (libseq/libetpan/libengine/libadapter) in a relocatable self contained package.
13:04 < heck> if the API (aka adpater) user for whatever reasons doesnt want or cant use the system wide installation of our libs
13:12 < heck> patrick: i have a binary pkg for the pEpPythonAdapter ready.
13:13 < heck> patrick: it is a prototype and it is only for linux_x86_64 / cpython3.7
13:13 < heck> patrick: verified to work on debian 10
13:15 < heck> patrick: if this is of any use already, please let me know.
13:20 <    patrick> | heck: yes, pls send it to me. Hartmut will probably be happy about that.
13:20 < patrick> hartmut: ^
...lines ommitted
13:34 <       fdik> | heck: with p≡p engine there are PER_USER_DIRECTORY and PER_MACHINE_DIRECTORY
13:35 <       fdik> | heck: this is for the databases
13:35 <       fdik> | heck: we don't influence how libraries are being found, with one exception in case you're building p≡p engine including SQLite3 amalgam.
13:38 < heck> fdik: actually just came the logical conclusion, that, if we want to include the whole lib-stack in the dist pkg of a pEpAdapter, we _never_ want to install anything system-wide (e.g. /usr/local/lib/)
13:40 < heck> fdik: because you only do that if you want to dist libs separately in a pkg that other pkg can depend on.
13:41 < heck> and since we dont dist anything without an adapter, there is no "libpEp-base" pkg that anything e.g. an adapter pkg could depend on
13:43 < heck> fdik: and also for the reason, that if we dist the shared libs (libSeq/libEtpan/libEngine) with every adapter, _and_ install system wide, they will overwrite each other and you en up having the wrong sys lib versions (peplibs) for your second last adapter install
13:44 < heck> fdik: we really have to define a concept for this problem.
13:45 < heck> fdik: we need a concept, because we dont do it the way most software projects do it, to depend on a "libpEp-base" pkg
13:49 <      Roker> | heck: ack.
13:51 < heck> fdik: but back to the pEpAdapter pkg that includes everything. just to be on the same page. its not more than 4 .so/.dll/.dylib files and a system.db file that has to be copied into $PER_MACHINE_DIR.
13:52 < heck> wont be "much" more ever... more or less depending on adapter, but thats the  subset we have right now
13:53 < heck> Roker: we need to clarifizzle our shizzle, no? :)
13:55 < heck> fdik: so, the only way to have several adapters (or versions, think test systems) installed is, to _not_ install them into system. Thats why we end uo either using LD_LIBRARY_PATH (user has to do) or RUNPATH
13:56 <      Roker> | heck: well, I am not a package maintainer. But I also got some ugly dpendency problems, also cyclic ones. ( fdik's solution to this was to add "make beinstall" to the pEpEngine, but it looks also like an ugly hack to me)
14:00 <       fdik> | heck: there's different possible use cases
14:00 <       fdik> | heck: our default use case is we've got an end user and either a Desktop adapter or an app
14:00 <       fdik> | heck: an extended use case is we've additionally the p≡p Python adapter
14:01 <       fdik> | heck: this adapter is lacking a feature: in case there's a desktop adapter it must use it (i.e. via JSON-RPC)
14:01 <       fdik> | heck: then we've got the more technical use cases
14:01 <       fdik> | heck: i.e. some scripting using p≡p Python adapter
14:01 <       Roker> | fdik: perhaps we should have a separate "-dev" package for testers and adapter developers?
14:01 <       fdik> | heck: and last but not least we've got the use cases for the active network components
14:02 < fdik> Roker: yes, that's one thing
14:02 <        fdik> | Roker, heck: just that the different operating systems are different, and therefore we'll need different solutions there, respectively
14:03 <       fdik> | heck: you're knowing the list of OS platforms and their sequence already ;_)
14:03 < heck> fdik: well the aspects that i tried to describe above are cross-platforms
14:03 < heck> cross-platform problems
14:04 <       fdik> | heck: there is no cross-platform solution with the exception “I have Python and no p≡p now I want to use p≡p in Python only”
14:04 <       Roker> | fdik: absolutely. That's why I hold off and let the experts (e.g. package maintainers for the several platforms) decide this question.
14:05 < heck> fdik: of course, but there are cross platform concepts, and everything i wrote above applies to all platforms i know.
14:06 < heck> fdik: so, we can solve a lot by defining how we conceptually solve these problems generally x-platform, and specifically where an exception or a specialization is needed
14:07 < heck> Roker: fdik: honestly, i am not an expert in packaging and making x-platform dists for product oriented software. either.
14:07 <       fdik> | heck: I don't share your conclusions
14:08 <       fdik> | heck: i.e. when it's about a Debian system installation there will be pEpEngine.deb
14:08 <       fdik> | heck: as dependency of pEpPythonAdapter.deb
14:08 <       fdik> | heck: depending on sequoia.deb
14:09 < heck> fdik: no, please. no
14:09 < heck> fdik: thats exactly what you said there wont be
14:09 <       fdik> | heck: no other way, because there will be i.e. pEpJNIAdapter.deb depending on pEpEngine.deb, too
14:09 <       fdik> | heck: it's platform dependent as I said
14:10 < heck> fdik: well i REALLY like to hear that, actually!
14:10 <       fdik> | heck: if there's package management by default in the OS we're going to use it
14:10 < heck> fdik: but until now, all i new was, that there will be a bin pkg containing the whole pep-stack AND we never distribute the enigne without adapter.
14:11 <       fdik> | heck: the topic exists i.e. on macOS and on Windows
14:11 <       fdik> | heck: because by default they don't have package management
14:11 < heck> fdik: sry but thats frustrating, no problem, but thats a bit frustrating now..
14:11 <       fdik> | heck: and there's the problem of Python packages
14:12 <       fdik> | heck: additionally, we have development binaries and we've got deployment
14:12 <       fdik> | heck: and they cannot be the same
14:12 <       fdik> | heck: I don't see any other option than analyzing use cases per OS and define what is best there, respectively
14:14 < heck> ...i need a coffebreak, right now.
...lines ommitted
16:25 < heck> patrick: hartmut: please find the self-contained pEpPythonAdapter python pkg on dragon /home/heck/public/pEp-2.0-cp37-cp37m-linux_x86_64.tgz
16:26 < heck> patrick: hartmut: the tgz contains the .whl pkg-file and a README.md containing complete installation instructions. please read.
16:30 <    patrick> | heck: great, thx.
16:34 <     andreas> | fdik: can you tell in wich Engine rev ENGINE-709 was fixed?
16:34 < andreas> 14:23 <andreas> Background: we decided to use 8e0723ca319a for the release as it is (close to) the last well tested (by service, keysync ++) version.
16:37 < heck> next thing i will do is trying to make the package work on all python3 versions, not just 3.7 because on CentOS/RHEL7 you will have python36 AFAIK.
16:39 < heck> maybe will work port this exact pkg now to MacOS first (3.7 only) depenging on how much effort it is to make it work on all python3 versions
17:00 <       fdik> | heck: don't know if we can support < 3.6
17:01 <       fdik> | heck: minimum 3.6 would be OK

Suggestions

Please add your suggestions after this line. Thanks a lot for your contribution.