The xPack Build Box

A reproducible environment for building binary xPacks.

Edit

The xPack Build Box (XBB) is an elaborated build environment focused on obtaining reproducible runs while building cross-platform standalone binaries for GNU/Linux, macOS and Windows.

How it works?

XBB achieves repeatability and consistency by:

controlling the versions of the tools used during the build
compiling most dependencies from sources.

By strictly controlling the versions of the compiled sources and tools, it is possible to create build environments that use about the same tools on both GNU/Linux (Intel and Arm) and macOS, helping to obtain consistent results.

On GNU/Linux, to allow the builds to be performed on any distribution, the actual builds run inside Docker containers (Intel/Arm, 32/64-bit).

On macOS, the builds run on standard systems, without the need for custom tools.

Types of builds

From the point of view of the desired result, there are two types of builds:

local/native builds, intended for development and running on the local machine
distribution builds, intended for xPack binary distributions and running on most modern machines

The main use cases of XBBs are distribution builds, but they can be used for native builds as well.

XBB v5.0

The latest release, v5.0 from early 2023 represents a major milestone for the xPack project, since it is the first self-sustained release, which allows to build new binary xPacks using previous xPacks, and does not require any custom Docker images, or other compiled tools.

For convenience, the new build scripts are xPacks, with multiple build configurations, for all supported platforms, and all actions are defined uniformly as xPack actions.

There are three choices to build a binary.

Native builds with all tools from the host machine

In this case all build tools are expected to be available on the host machine.

xpm run install
xpm run build-native

The first command is used to install the helper project, which is a source xPack (@xpack-dev-tools/xbb-helper).

The second command runs the build.

These commands run the same on both GNU/Linux and macOS.

To generate the Windows binaries on GNU/Linux, use the separate action:

xpm run install
xpm run build-native-win

Native builds with the major tools coming from xPacks

Sometimes the native tools available on the host machine may be too old for building modern packages.

A simple solution is to install the main tools as binary xPacks. The selection of tools depends on the target platform.

The existing projects include separate build configurations for multiple platforms (darwin-x64, darwin-arm64, linux-x64, linux-arm64, linux-arm, win32-x64).

For example, to build the Intel GNU/Linux binaries, use:

xpm run install
xpm run install --config linux-x64
xpm run build --config linux-x64

Docker builds with tools coming from xPacks

An even more reproducible configuration can be achieved by using Docker containers, with the base system tools.

xpm run install
xpm run docker-prepare --config linux-x64
xpm run docker-build --config linux-x64

Writable helper scripts

For normal builds, the helper project, which is a source xPack, is downloaded and marked as read-only, to prevent damaging it.

For development purposes it might be necessary to update it; the xPack solution (inspired by the npm solution) is to download the helper repository into a separate location, and link it to the current project.

This is done in two steps, first a link from the central store to the helper repo is created. This ensures that multiple projects can conveniently use the writable helper.

In the second step, a link from the build project to the central store is created.

For details on the actual usage, please check the build projects.

Supported platforms

Generally, xPack binaries are available for the following platforms:

Windows Intel 64-bit
GNU/Linux Intel 64-bit
GNU/Linux Arm 32-bit
GNU/Linux Arm 64-bit
macOS Intel 64-bit
macOS Apple Silicon 64-bit

Note: Starting with 2022, support for macOS Apple Silicon was added; 32-bit support for Windows and GNU/Linux Intel was discontinued.

XBB supports creating binaries for all these platforms.

For the detailed supported versions, please check the specific release page.

GNU/Linux Arm binaries

Support for Arm binaries was added in v3.1, in early 2020.

The supported architectures are:

arm64v8 - the ARMv8 64-bit architecture Aarch64
arm32v7 - the ARMv7 32-bit architecture with hardware float (armhf)

macOS Apple Silicon

Support for Apple Silicon was added in v3.4, in late 2021.

Docker specifics

As with any Docker builds, the XBB builds run completely inside Docker containers, which start afresh each time they are instantiated.

To pass the folder with the build scripts in and the results out, it is usual to use a Work folder, for example:

$ docker run -it --volume "${HOME}/Work:/Host/Work" ilegeul/ubuntu:amd64-18.04-xbb-v3.4
root@831bc35faf9f:/# ls -l /Host/Work
total 175320
drwxr-xr-x  14 root root       448 Mar  7 19:47 arm-none-eabi-gcc-9.2.1-1.2
drwxr-xr-x 144 root root      4608 Mar  9 13:15 cache
drwxr-xr-x  34 root root      1088 Mar 26 11:22 openocd-0.10.0-14
drwxr-xr-x  12 root root       384 Oct 30 19:00 riscv-none-embed-gcc-8.3.0-1.1

In this simple configuration, the builds run with root permissions; with more elaborate configurations it is possible to start the Docker containers with user rights, but they are beyond the scope of this document.

How to use the XBB tools

Both on GNU/Linux and macOS, the XBB tools are installed as binary xPack dependencies in xpacks/.bin and are fully distinct from the system tools.

To access them, the application should update the PATH to prefer the xpacks/.bin path.

Common scripts

For consistency between projects, common scripts were grouped in the helper project.

After installing it with xpm install, the scripts are available from xpacks/xpack-dev-tools-xbb-helper.

There are many scripts in this location; the main ones are in the scripts folder. The scripts to build various dependencies are in the dependencies folder.

The `pkg-config-verbose` script

While running the configuration step, it is sometimes useful to trace how pkg-config identifies resources to be used during the build.

The standard pkg-config does not have an option to increase verbosity.

The workaround is to use a separate script that displays the received command and the response on the stderr stream.

This script is not specific to XBB, and can be used with any build.

For this, copy the file into .../xbb/bin or any other folder present in the PATH and pass the script name via the environment.

chmod +x /opt/xbb/bin/pkg-config-verbose
export PKG_CONFIG=pkg-config-verbose

No more TeX

Starting with v3.4, support for building the manuals was dropped, and the TeX tools are no longer needed.

Maintainer info

README-MAINTAINER

End of support for Linux distributions

To help decide what base version to use, and how long to keep support for it, a list of main Linux distributions was compiled, with the GLIBC versions:

end of support

Upgrade plans

RedHat maintenance support 1 for RHEL 7 ended in Aug. 2020, with maintenance support 2 to last until June. 2024. However XBB support for RHEL 7 is a tough requirement, and was discontinued starting with 2022.

Compatibility was moved up to Ubuntu 18 LTS (GLIBC 2.27), which also provides compatibility with RedHat 8 / Debian 10, which both use GLIBC 2.28.

The next upgrade step will be taken when the minimum Node.js becomes 18.x, which requires GLIBC 2.28, when the base system will most probably be Debian 10.

Credits

The xPack Build Box is inspired by the Holy Build Box.