Old, unmaintained and wrong: Building the engine on debian

Do not use this

These instructions are unmaintained and obsolete. Do not use them.

More comprehensive instructions

The Adapter Instructions page is carefully written and maintained and covers multiple systems, inclding debian 9 and 10.

This document is not completely consistent with that page, and describes a slightly different configuration involving more cases of installation from sources. Where the Adapter Instructions document applies, it should be preferred.

Introduction

This is a quick summary of the main steps I (positron) followed to set up the pEp engine plus enough software to test it, in a way which is convenient for playing with modifications. Not everything is clean and some points may be not explained clearly enough; feel free to ask if you need more information.

The instructions of this page assume a recent engine, from the master branch as of October 2021. Very different versions will require some tweaks.

System

This is GNU/Linux on x86_64, debian unstable, with the following package repos: sid, testing, stable oldstable and oldoldstable. It is mostly up-to-date as of late Summer 2021.

I have multiple GCC compilers installed, including cross-toolchains I use for other projects; the compilers I use for pEp are mostly gcc-10 and g++-9. I occasionally used x86_64-unknown-linux-gnu-gcc and x86_64-unknown-linux-gnu-g++ compiled from crosstool-ng, but I think those are no longer needed.

My shell is Bash. My user name is luca. My computer’s name is moore. My terminal is ROXterm – I do not particularly recommend ROXterm, however it supports ANSI terminal escape sequences and can save a “session” of open shells in different directories, one per tab, easy to reopen: this is useful to return to a session with everything ready to work on pEp sources, one terminal window with one tab per source directory.

However you decide to adapt these notes to your installation avoid whitespace in user names and path names, please! The world is complicated enough already.

Conventions and system description

I keep source directories under ~/pep-src , each one being in a direct subdirectory of pep-src .
I keep software installed from sources (such as pEp, but not only pEp) into multiple “prefixes” having the same structure as the standard /usr: there will be a subdirectory named bin for exacutables, one named lib for libraries and so on, with include, share and the like. The advantage is that it is easy to entirely remove, replace or disable such a prefix, without conflicting with the distribution installed on the machine: PATH and other environment variables can be defined so that non-standard prefixes take precedence. I have a bash function changing environment variables so as to make a prefix active. The function is named add-prefix and a version that works with no dependencies is on my personal web site in a “scratch” page . My pEp prefix is ~luca/usr-pep . Notice that there may be users different from luca on the system: at runtime they will each have their own ~/.pEp directory for engine databases and the like plus their own configuration and runtime libraries, but the software is executed from luca’s `usr-pep: this makes it easy to test software changes.

Challenges

The pEp Python adapter is precious for testing and exploring the system.

debian does not package a usable Python 3.8: it packages very early versions of 3.7, which break most current debian packages, and 3.9 which is not supported on the pEp side. The solution is building Python 3.8 from sources. In practice I also need to build Boost from sources in order to support that. pip3 will be the the pip3 which comes from Python 3.8. All of this of course ends up installed under ~luca/usr-pep .

`~luca/.bashrc`

This is in ~luca/.bashrc (non-luca users I use for testing have a symlink to ~luca/.bashrc instead of their own different file at ~/.bashrc ):

export ESC=$(echo -e '\033')
export BEL=$(echo -e '\007')
export GREEN=$ESC[0m$ESC[32m
export RED=$ESC[0m$ESC[31m
export REVERSE=$ESC[7m
export NOATTR=$ESC[0m
# This is for setting the title of an Xterm window:
export BEGINTITLE="${ESC}]0;"
export ENDTITLE="${BEL}"

# PROMPTCOLOR and the like can be defined.  I am ignoring all of this.

export PS1="\\[$NOATTR$PREFIXCOLOR\\]\$PROMPTPREFIX\\[$NOATTR$PROMPTCOLOR\\][\\u@\\[$NOATTR$HOSTCOLOR\\]\\h\\[$NOATTR$PROMPTCOLOR\\] \\w]\\$\\[$NOATTR\\] \\[$NOATTR$NONPROMPTCOLOR\\]"

title ()
{
    echo -en "${BEGINTITLE}$@${ENDTITLE}"
}

pwdrelative ()
{
    pwd | sed 's@.*/\([^/]*\)@\1@'
}

titlepwd ()
{
    title $(pwdrelative)
}

pepify ()
{
    PROMPTPREFIX="p"
    . ~/.bashrc
    export CC=gcc-9
    export CXX=g++-9
    # This is /home/luca, not $HOME: other users will take pEp from ~luca ,
    # but store their own data in ~/.pEp .
    export PR=/home/luca/usr-pep
    export PYTHONPATH=$PR/lib/python3.8/site-packages
    add-prefix $PR
    unset OBJC_INCLUDE_PATH C_INCLUDE_PATH CPLUS_INCLUDE_PATH INCLUDE_PATH; export INCLUDE_PATH=$PR/include
}

# Per-directory configuration.  This is useful with terminal sessions, where
# the last visited directory (and only that) is saved.
if test "${positron_initialization_finished}" != 'yes'; then
    positron_initialization_finished='yes'
    title="$(pwdrelative)"

    # Check for pEp
    if test "$HOSTNAME" = 'moore' \
            && echo "${PWD}" | grep -q "^/home/luca/pep-src"; then
        inapepdirectory=yes
        pepify
        echo "${GREEN}${REVERSE}initialised for p≡p in ${title}${NOATTR}"
    fi

    # pEp as luca
    if test "$inapepdirectory" = 'yes' \
            && test "$USER" = 'luca'; then
        case "${title}" in
            pEpJSONServerAdapter)
                title=json;;
            *)
                ;;
        esac

    # pEp as my testing user 'peter'
    elif test "$inapepdirectory" = 'yes' \
            && test "$USER" = 'peter'; then
        title="peter@$(pwdrelative)";
    fi
    title "${title}"
fi


# pEp, tentative
##########################################################################

# Recommended by pip3 installations.
export PATH="$HOME/.local/bin:$PATH"

This, in particular, ensures that when I restore a terminal session every tab contains a shell in the right directory, already “pepified” (with the environment set up), and with an understandable tab title.

Notice that $PR contains the pEp prefix.

Building source packages

This is what I have in ~/pep-src. Again, I keep source directories under ~/pep-src , each one being in a direct subdirectory of pep-src .

[luca@moore ~]$ \ls -d1 ~/pep-src/*/
/home/luca/pep-src/asn1c/
/home/luca/pep-src/boost_1_77_0/
/home/luca/pep-src/googletest/
/home/luca/pep-src/gtest-parallel/
/home/luca/pep-src/how-to-reproduce-570/
/home/luca/pep-src/libetpan/
/home/luca/pep-src/lib-pep-adapter/
/home/luca/pep-src/lib-pep-datatypes/
/home/luca/pep-src/lib-pep-transport/
/home/luca/pep-src/pep-engine/
/home/luca/pep-src/pep-engine-broken/
/home/luca/pep-src/pep-engine--stable/
/home/luca/pep-src/pEpForThunderbird/
/home/luca/pep-src/pep-jni-adapter/
/home/luca/pep-src/pEpJSONServerAdapter/
/home/luca/pep-src/pep-macos-adapter/
/home/luca/pep-src/pep-objc-adapter/
/home/luca/pep-src/pep-python-adapter/
/home/luca/pep-src/Python-3.8.11/
/home/luca/pep-src/pEpEngineSequoiaBackend/
/home/luca/pep-src/webserver/
/home/luca/pep-src/yml2/

It should be obvious what all of them are. They are all cloned git repositories, unless the description below explicitly speaks of a source tarball. The rest should follow one of the possible correct orders which satisfy dependencies.

Python 3.8 (from the source tarball):

[luca@moore ~/pep-src/Python-3.8.11]$ export C_INCLUDE_PATH=$PR/include:/usr/include:/usr/local/include:/usr/include/x86_64-linux-gnu; export CPLUS_INCLUDE_PATH="$C_INCLUDE_PATH"; ./configure --prefix=$PR --disable-optimizations --enable-shared && make -j32 && make -j32 install && pip3 install ipython && pip3 install bpython && pip3 install termcolor && $PR/bin/python3.8 -m pip install --upgrade pip

Make sure that SSL support and threading were actually installed; this may involve obvious dependencies such as libopenssl-dev, which should be present on development machines.

Boost 1.77 (from the source tarball):

# [luca@moore ~/pep-src/boost_1_77_0]$ ./bootstrap.sh --prefix=$PR --with-python-version=3.8 --with-python=$PR/bin/python3 && ./b2 link=shared runtime-link=shared cxxflags="-fPIC -pthread" install && ln -s "$PR/lib/libboost_python38.so" "$PR/lib/libboost_python3.so"

`pEpEngineSequoiaBackend`

pEpEngineSequoiaBackend is required instead of the discontinued sequoia FFI for recent versions of the engine (not officially released yet as of early 2022).

This is a wrapper library providing a stable FFI between Sequoia and the pEp engine. Building pEpEngineSequoiaBackend also downloads and builds part of sequoia, among other Rust dependencies.

≡[luca@moore ~/pep-src/pEpEngineSequoiaBackend]$ CARGO_TARGET_DIR=_build cargo build --release && mkdir -p "$PR/"{lib,lib/pkgconfig} && sed --in-place "s@^libdir=.*\$@libdir=$PR/lib@" _build/*/pep_engine_sequoia_backend.pc && cp -af _build/*/*.{so,d} $PR/lib/ && cp -af _build/*/*.pc $PR/lib/pkgconfig/ && echo GOOD

YML

x[luca@moore ~/pep-src/yml2]$ pip3 install --isolated --prefix $PR .

ASN1c

x[luca@moore ~/pep-src/asn1c]$ ./configure --prefix=$PR --disable-static --enable-shared  && make && make install

Libetpan

x[luca@moore ~/pep-src/libetpan]$ ./configure --disable-static --enable-shared --prefix=$PR CFLAGS='-fPIC -O2 -g' && make && make install

googletest

There must be no / after -DCMAKE_INSTALL_PREFIX=; otherwise this silly system will arbitrary insert a usr string. With this caveat:

[luca@moore ~/pep-src/googletest]$ cmake -DCMAKE_INSTALL_PREFIX=  && make install DESTDIR=$HOME/usr-pep

gtest-parallel

This does not require any configuration or building.

lib-pep-transport

This requires a build configuration file named local.conf in the source directory:

PREFIX=$(HOME)/usr-pep
YML2_PATH=$(PREFIX)/bin

Then run:

[luca@moore ~/pep-src/lib-pep-transport]$ make && make install

pEp engine

This requires a build configuration file named local.conf in the source directory:

PREFIX=$(HOME)/usr-pep
PER_MACHINE_DIRECTORY=$(PREFIX)/share/pEp

# CC=x86_64-unknown-linux-gnu-gcc
# CXX=x86_64-unknown-linux-gnu-g++
CC=gcc-10
CXX=g++-10

# In order to have a non-debug build, comment out the
#   DEBUG=placeholder
# line in Makefile.conf .  Undefining the variable here
# has no effect.

YML2_PATH=$(PREFIX)/bin

ASN1C_INC=-I$(PREFIX)/share/asn1c  # Very bizarre, but this is what asn1c does.

# These are just to avoid warnings due to a missing sequoia-openpgp.pc , and should
# work with either the old sequoia FFI or its recent replacement pEpEngineSequoiaBackend:
# compilation would still work without any of these definitions.
SEQUOIA_USELESS_FLAGS='-g'
SEQUOIA_CFLAGS=$(SEQUOIA_USELESS_FLAGS)
SEQUOIA_INC=$(SEQUOIA_USELESS_FLAGS)
SEQUOIA_LIB=-L$(PR)/lib
SEQUOIA_LDFLAGS=$(shell if ls "$$PR"/lib/libsequoia_openpgp_ffi.* > /dev/null 2> /dev/null; then echo -lsequoia_openpgp_ffi; else PKG_CONFIG_PATH="$$PR"/lib/pkgconfig pkg-config --libs-only-l pep_engine_sequoia_backend; fi)

GTEST_SRC_DIR=$(HOME)/pep-src/googletest/googletest
GTEST_INC_DIR=$(HOME)/pep-src/googletest/googletest/include
GTEST_PL=$(HOME)/pep-src/gtest-parallel/gtest_parallel.py

Then run:

[luca@moore ~/pep-src/pep-engine]$ make && make install && cp asn.1/libasn1.a ${PR}/lib/ && make dbinstall

Notice that

make test

works from the source directory as long as the database has been installed (see the dbinstall target above) but by default links against an installed library: this can easily mislead beginners like me who want to test their changes.

lib-pep-adapter

This requires a local.conf file in the source directory:

PREFIX=$(HOME)/usr-pep
DEBUG=1

Then:

[luca@moore ~/pep-src/lib-pep-adapter]$ make uninstall clean install

pep-python-adapter

This requires a local.conf file in the source directory:

PREFIX=$(HOME)/usr-pep
DEBUG=1

(It happens to be the same as in lib-pep-adapter.)

Then:

[luca@moore ~/pep-src/pep-python-adapter]$ make clean && make && make install

webserver (for the JSON adapter)

This requires a local.conf file in the source directory:

PREFIX=$(HOME)/usr-pep
NDEBUG=0

Then:

[luca@moore ~/pep-src/webserver]$ make -j uninstall clean && make -j install

pep-json-adapter

This requires a local.conf file, not in the source directory but in its server/ subdirectory:

PREFIX=$(HOME)/usr-pep
PEP_INCLUDE=$(PREFIX)/include
DEBUG=1

Then:

p[luca@moore ~/pep-src/pEpJSONServerAdapter]$ make -C server uninstall clean && make -C server -j install

pEp for thunderbird

I did not use the “setup” project. This will suffice to generate an xpi archive.

make

The xpi archive can be installed interactively. After the first installation the pEp addon can be replaced with the newly compiled version by:

make && cp build/pEp4Tb@pEp.security.xpi /home/luca/.thunderbird/*/extensions/pEp4Tb@pEp.security.xpi

In order to test, of course from a pepified shell:

pEp-mini-json-adapter -d0 -D & sleep 3 ; thunderbird -purgecaches &> /dev/null

Notice that this works for non-luca users as well: it is easy, for example, to have two pEp-for-thunderbird instances running under different Unix users, which can send messages to each other.

When debugging after such an installation, run thunderbird with the option -purgecaches .

This is important: the -purgecaches (only one -) option is always necessary, but not documented in most Thunderbird versions. (Thanks Jörg for teaching me about it.) Notice that Thunderbird silently ignores unknown command-line options (!).

Further information

Again, the Adapter build instructions page also contain information about how to build the engine. Thanks heck.