Skip to main content

The xPack Project history

The xPack project received its name in 2017, but it actually originated in 2012, when it was named xcdl, and a Python experimental implementation of the eCos CDL was written.

The early years and the tight relation to µOS++​

The history of the xPack project was deeply intertwined with the history of µOS++, as it was initially intended to automate the building and running multi-platform unit tests for µOS++. Later, it was also used to build the binary xpm packages, which further validated the concept.

To avoid redundancy, the full story is not replicated here, so please read the history of µOS++ before continuing; the brief list of the important events is:

  • eCos CDL was the inspiration for a component based design (early 2000s).
  • In 2013, the GNU ARM Eclipse plug-ins (later rebranded as the Eclipse Embedded CDT) was the first attempt to find a solution that avoids generating make files by hand, but this prevented the building and running of unit tests in a scriptable environment.
  • In 2013, the experimental XCDL (eXtended Components Definition Language) project, inspired by eCos CDL, implemented in Python, validated the concept.
    note

    The major weakness of this XCDL implementation was the limitation introduced by having the build configuration metadata as a Python code: it could be edited only manually, and not via a GUI based IDE (compared, for example, to the C/C++ settings in Eclipse CDT, which writes back the XML configuration file)".

  • In 2014, the CMSIS Packs solution was evaluated, and an Eclipse plug-in was created to manage them. However the solution was later found to have too many design issues, and was abandoned.
  • In 2015, the yotta solution was considered, and the first XCDL packs based on yotta were tested. For various reasons, yotta was later abandoned by Arm.
    note

    As yotta was strongly influenced by npm, the npm ecosystem was considered a good starting point for a future solution.

  • In 2017, the first release of xpm was made available as a complementary solution to npm, and packages managed by it were named xPacks; xpm complements npm as the industry standard package manager for JavaScript packages, and uses the same npmjs.com as the hugely popular public package repository.
  • In 2019, xpm was extended to also support installing binary packages, and a new class of tools was created as xPack Binary Development Tools.
  • In 2022, xpm was used to build the binary packages, proving that the technology is standalone, with the newer versions of the tools being built with existing xPacks.
  • In 2023, with most of the tools in place, a first C/C++ project (@micro-os-plus/utils-lists) was fully tested via xpm, once again validating the whole concept.

The xPack project goals​

The project came to life out of a practical need to find a replacement for the Eclipse CDT managed build, with the additional requirement of being able to run unit tests in a scriptable environment.

note

In the beginning, it seemed easy, but it later proved to be quite a challenge and required defining sub-goals and achieving them in sequence.

Through trial and error, the main aspects that need to be addressed by such a solution were identified as:

  • The build (defining which files enter the build, the compiler/linker options, etc).
  • The application configuration (e.g. managing the values of the #define in a consistent way).
  • The tools (toolchains, debuggers, emulators, etc).

The XCDL experimental version​

The first experimental solution was XCDL, which was used in the µOS++ Second Edition. Essentially, some of the eCos CDL functionality was implemented in Python.

The result looked like this:

The implementation was quite easy, and worked pretty well.

However several issues were identified:

  • Using configuration files written in a programming language was convenient, but it had a major drawback: since parsing Python is not trivial, it prevented the configuration being edited with a GUI tool (similar to the Eclipse settings pages). In other words, the configuration files must be in a structured format; otherwise editing them by a machine is problematic.
  • Although the µOS++ SE was more or less modular, the project was basically a mono-repo, making it difficult to write unit tests for each module.
  • Running unit tests with several toolchains, not to mention different versions of the same toolchain, proved extremely difficult. Doing this on multiple platforms (macOS, Windows, GNU/Linux) was even more challenging.

The lessons learned from this experiment were:

  • If a configuration needs to be edited by a machine, it must be written in a structured format, with JSON being the preferred choice.
  • If each module is expected to be tested, it is better to use multiple small repositories, each with its CI tests.
  • If the project is split into multiple packages, a tool to automate the installation and management of dependencies is a great addition.
  • Toolchains and other tools need to be standalone and relocatable, allowing multiple versions to be installed in different folders on the same machine. Managing these multiple tools can also be automated with a tool.

Binary xpm packages​

The first binary tools prepared were arm-none-eabi-gcc, openocd, qemu and windows-build-tools. They were also used in Eclipse project. Initially, they were installed manually, and Eclipse was able to identify them in certain fixed location.

npm/xpm​

While searching for a solution to automate the installation of different source packages, npm appeared to be a close fit.

However, using npm directly proved to be an unfortunate solution, as npm's strategy for installing dependencies is specific to JavaScript, with hierarchical packages.

Therefore, it was decided that a new tool was needed, primarily to handle the linear dependencies of C/C++. This is how xpm was created. Although it preserves the general look and feel and design concepts of npm, it is not a replacement for npm.

Later, xpm was extended to automate the installation of toolchains and other binary archives.

As of now, there are toolchains for Arm, AArch64, RISC-V, and native (GCC & clang) that run on Windows, Linux, and macOS. It is probably fair to say that the tools part of the project is properly covered and fully automated.

Build system generators​

As for the build and configuration tools in the project, work is still in progress.

For the build, the initial idea was to rewrite the Python code in JavaScript, but a step-by-step approach was later considered.

In the first step, existing build system generators like CMake and meson were used, offering several advantages:

  • They allow tests to be added to the µOS++ projects immediately, without waiting for any special or future tools.
  • They provide a valuable opportunity to gain experience in organising projects with such tools.
  • They offer users a direct way to integrate the µOS++ source code libraries into their existing projects built with CMake or Meson.

To manage the multi-platform, multi-architecture, multi-version, and other varied tests, a tool to manage multiple build configurations was needed. Since each configuration requires its own tools, with different versions, it became clear that the build configurations must be managed by the same tool. Therefore, xpm was extended to handle multiple build configurations. Later, xpm actions were added to automate running sequences of commands, such as CMake or meson, to build and run the tests.

Future plans​

µOS++​

As of mid-2023, the project is quite advanced, but it is not yet complete.

In the future (2025), it is planned that all the µOS++ projects to be updated to use the multiple build configurations automated by xpm, ensuring everything is fully tested in µOS++.

With the experience gained from this step, the next step is to finalise the specifications for the xPack CDL, which will handle the configuration and build processes.

The tool will probably be called xcdl, and will use json files, with a content similar to this one:

As it can be seen, this file defines the include folders, the source files, and some preprocessor definitions that might be enabled while building the project.

This file, along with all other similar files from the packages that are part of the project, will provide the input data for a tree of components that can be rendered in a graphical tool like VS Code. This allows for project configuration in a manner similar to, or possibly even more advanced than, the Eclipse CDT settings. The hierarchical components reflect the logical structure of the application and enable the automatic management of dependencies between components and configuration options.

Visual Studio Code​

Additions to the VS Code xPack extension are planned with features to edit the tree of xPack CDL components and run the builds directly, without CMake/meson. This will be a true xPack-enabled tool and an alternative solution to Eclipse Embedded CDT.

However, the use of this tool should be optional. The entire functionality should also be available from a terminal and a plain text editor.

Conclusions​

As can be seen, the xPack project originated from the need to automate the workflow for existing open-source projects, particularly the execution of multiple tests for embedded projects, which often poses more challenges than developing the projects themselves.

To be continued...​