The xCDL packages

note

Work in progress.

To ensure a package is compatible with the xCDL component framework, it must adhere to specific rules set by the framework. Packages should be distributed in a format recognized by the component repository administration tool. Each package must contain a top-level xCDL metadata file that describes the package to the component framework. Additionally, there are specific portability requirements to ensure the package can be utilized across various host environments.

Moreover, the component framework offers several guidelines. Although it is not mandatory for packages to strictly follow all guidelines, adhering to them can streamline certain operations.

Packages

What is a package?

An xCDL package is essentially an xpm package that includes an xcdl.json file to store the xCDL metadata.

In other words:

1. A folder containing a valid xcdl.json file, with a top cdlPackage definition.
2. A gzipped tarball containing the folder mentioned above.
3. A URL that resolves to the gzipped tarball.
4. A [@<scope>]<name>@<version> that is published on a registry.
5. A Git URL that, when cloned, results in the folder mentioned above.

The definition is inspired by the npm Developer's Guide package definition and is intentionally Git and JavaScript-centric, as these technologies are considered mature and worth considering.

For the foreseeable future, the external representation of the xCDL metadata will remain based on JSON; the metadata documentation will use this syntax.

In addition to the xCDL metadata file(s), a typical package may contain the following:

• Several source files (.c/.cpp) and header files (.h): These files are used to create the project artifact (library or executable). Some source files may serve other purposes, such as providing a linker script.
• Exported header files: These define the interface provided by the package.
• Online documentation: This includes reference pages for each exported function.
• Test cases: Provided in source format, these allow users to verify that the package works as expected on their specific hardware and configuration.

It is conventional to include a per-package CHANGELOG file to track changes. This is especially valuable to end users who may not have convenient access to the source code control system used to manage the master copy of the package, making it difficult to determine what has changed. It can also be very useful to the main developers.

Not all packages need to contain all of these elements.

Some packages may not include any source code. It is possible to have a package that merely defines a common interface, which can then be implemented by several other packages, especially in the context of device drivers, or to use header-only C++ templates. However, it is still common to include some code in such packages to avoid duplicating shareable code across all implementation packages. Similarly, a package might have no exported header files, containing only source code that implements an existing interface. For example, an Ethernet device driver might implement a standard interface without providing any additional functionality.

Packages do not need to include online documentation, although this may affect their adoption. The same applies to per-package test cases.

Package layout

The component framework does not enforce any specific layout for the packages. The only constraints are:

• Each xCDL package must be stored in a separate folder; multiple packages cannot be in the same folder.
• Each xCDL package must have a top-level xCDL metadata file.

However, it is recommended to use a per-package folder layout that organizes the package contents based on functionality:

utils-lists-xpack.git
├── CHANGELOG.md
├── CMakeLists.txt
├── LICENSE
├── README.md
├── build
├── include
│   └── micro-os-plus
├── meson.build
├── package.json
├── src
│   └── lists.cpp
├── tests
├── website
└── xcdl.json

The names should be self-explanatory. The include sub-folder is used for the exported header files, the src sub-folder holds the source files, the tests sub-folder contains the unit tests and other test-related files, and the website sub-folder is used to build the project website, where the reference documentation is also expected to be.

Except for the name and location of the xcdl.json file, this folder layout is just a guideline and is not enforced by the component framework.

tip

Strictly speaking, the xCDL framework does not mandate that the content be an xpm package, and component authors can choose other strategies for dependency management.

However, the xCDL workflow benefits from xpm handling dependencies and build configurations. Therefore, it is highly recommended for xCDL packages to also be xpm packages, i.e., to have a package.json file in the top folder.

When the package is a source code library designed to be an xpm dependency of another project, the build, tests, and website folders should be added to the .npmignore file, as they are not needed at this stage.

Outline of the build process

The full build process is described in The xCDL Build Process page, but a summary is appropriate here. A build involves several folders:

1. Component repository: This is where all dependent package source code is held, along with xCDL metadata. For build purposes, a component repository is read-only. Application developers will only modify the component repository when installing or removing packages via the administration tool. When using xpm to manage dependencies, the component repository is located in the xpacks folder. Component authors will typically work with Git/local component repositories, which are read/write (via xpm link).
2. Local source tree: This is where artifact-specific files are located. The artifacts are built using the source files in this tree and selected source files from the component repository. Several artifacts can be constructed from one local source tree. In the simple case, these are the usual build configurations, like Debug/Release. In a more elaborate use case, it might be possible to build artifacts for different platforms (targets/boards) from the same source tree. For a given local source tree, there is a single xcdl-config.json file, containing the cdlConfigurations for each artifact to be built.
3. Build tree: Each configuration has its own build tree. The build tree contains the generated headers, intermediate files, object files, the final artifact, list files, map files, etc. Once a build is complete, the build tree contains no information that is useful for application development and can be wiped, although this would slow down any rebuilds following changes in the source tree.

The build process involves the following steps:

1. Configuration Setup: Given a configuration, the component framework is responsible for creating all the necessary folders required during the build. The configuration header files will be generated at this time. Depending on the host environment, the component framework will also generate makefiles or other build scripts for the various packages. Every time the configuration is modified, this step needs to be repeated to ensure that all option consequences take effect. Care is taken to avoid unnecessary rebuilds.
2. Compilation: All source files relevant to the current configuration are compiled. This involves a set of compiler flags initialized on a per-target basis, with each package able to modify these flags, and the user having the ability to override them as well. Care must be taken to avoid inappropriate target dependencies in packages intended to be portable. The component framework has built-in knowledge of how to handle C, C++, and assembler source files. The compilerSourceFiles property is used to list the files that should be compiled. All object files end up in the build tree.

Configurable source code

All packages should be fully portable to all target hardware, with the obvious exceptions of HAL and device driver packages. They should also be completely bug-free, require the absolute minimum amount of code and data space, be so efficient that CPU time usage is negligible, and provide numerous configuration options to give application developers full control over the behaviour. Configuration options are optional only if a package can meet the requirements of every potential application without any overhead.

This guide does not aim to explain how to achieve all of these requirements.

The xCDL component framework has important implications for the source code, including compiler flag dependencies, package interfaces versus implementations, and how configuration options affect the source code.

Compiler flag dependencies

Wherever possible, component authors should avoid dependencies on specific compiler flags, as these dependencies can impact portability. For example, if one package needs to be built in big-endian mode and another in little-endian mode, it will usually be impossible for application developers to use both packages simultaneously. Additionally, this removes the application developer's choice in the matter. It is far better for the package source code to adapt the endianness at compile-time, or possibly at run-time, although the latter will involve code-size overheads.

Package interfaces and implementations

The component framework provides encapsulation at the package level. Package A cannot access the implementation details of package B at compile-time. Specifically, if there is a private header file in a package's src sub-folder, this header file is completely invisible to other packages. Any attempts to bypass this by using relative pathnames beginning with ../.. are generally doomed to failure and should not be attempted. There are two ways in which one package can affect another: through the exported header files, which define a public interface, or via the xCDL configurations.

This encapsulation is a deliberate aspect of the overall xCDL component framework design. In most cases, it does not cause any problems for component authors. In some cases, enforcing a clean separation between interface and implementation details can improve the code. It also reduces problems when a package gets upgraded: component authors are free to make significant changes on the implementation side, including renaming every single source file. Care must be taken only with the exported header files and the xCDL data, as these have the potential to impact other packages. Similarly, application code cannot access package implementation details, only the exported interface.

Occasionally, the inability of one package to see the implementation details of another can cause problems. One example occurs in HAL packages, where it may be desirable for the architectural, variant, and platform HALs to share some information that should not be visible to other packages or application code. This may be addressed in the future by introducing the concept of friend packages, similar to how a C++ class can have friend functions and classes that are allowed special access to a class's internals. It is not yet clear whether such cases are sufficiently frequent to warrant introducing such a facility.

Source code and configuration options

Configurability typically involves source code that needs to implement different behaviours depending on the settings of configuration options. It is possible to write packages where the only consequence associated with various configuration options is to control what gets built, but this approach is limited and does not allow for fine-grained configurability. There are three main ways in which options could affect source code at build time:

1. The component code can be passed through a suitable preprocessor, either an existing one such as m4 or a new one specially designed with configurability in mind. The original sources would reside in the component repository, and the processed sources would reside in the build tree. These processed sources can then be compiled in the usual way.

   This approach has two main advantages. First, it is independent of the programming language used to code the components, provided reasonable precautions are taken to avoid syntax clashes between preprocessor statements and actual code. This would make it easier in future to support languages other than C and C++. Second, configurable code can make use of advanced preprocessing facilities such as loops and recursion.

   The disadvantage is that component authors would have to learn about a new preprocessor and embed appropriate directives in the code. This makes it much more difficult to turn existing code into components and involves extra training costs for the component authors. The extra definitions might also confuse document-generating utilities like Doxygen.

2. Compiler optimizations can be used to elide code that should not be present, for example:

   ...
   if (OS_INTEGER_NUMBER_UARTS > 0) {
       ...
   }
   ...

   If the compiler knows that OS_INTEGER_NUMBER_UARTS is the constant number 0, then it is a trivial operation to eliminate the unnecessary code. The component framework still has to define this symbol in a way that is acceptable to the compiler, typically by using a const variable or a preprocessor symbol. In some respects, this is a clean approach to configurability, but it has limitations:

   • It cannot be used in the declarations of data structures or classes, nor does it provide control over entire functions.
   • Additionally, it may not be immediately obvious that this code is affected by configuration options, which may make it more difficult to understand.
   • Even if the condition does not evaluate to true and the optimiser removes the code, it still requires the elided code to be syntactically correct, which is sometimes not possible due to missing references.

3. Existing language preprocessors can be used. In the case of C or C++ this would be the standard C preprocessor, and configurable code would contain a number of #ifdef and #if statements.

   #if defined(OS_DEBUG_INFRA_DEBUG_PRECONDITIONS)
       ...
   #endif
   
   ...
   
   #if (OS_INTEGER_NUMBER_UARTS > 0)
       ...
   #endif

   This approach has the advantage that the C preprocessor is a technology that is both well-understood and widely used. However, there are also disadvantages: the preprocessing facilities are rather limited, and some people (including ourselves) consider the technology to be unattractive, generally decreasing program readability.

Preprocessor definitions

The current component framework generates configuration header files with C preprocessor #defines for each option (typically, there are various properties which can be used to control this). It is up to component authors to decide whether to use preprocessor #ifdef statements or language constructs such as if. At present there is no support for languages which do not involve the C preprocessor.

C++11 constexpr

The second type of definitions that the component framework should support are C++11 constexpr definitions. These definitions are the typed equivalent of the preprocessor definitions, but with some significant differences:

• They are processed by the compiler, not the preprocessor; this has the advantage of allowing type checks.
• They can be grouped in namespaces, minimising the risk of name clashes.

namespace one
{
    constexpr int variable = 1234;
}

It is recommended to use constexpr expressions for all options providing values, and to limit the use of preprocessor definitions solely to control which parts of the code are included, i.e., definitions like #if defined(OS_INCLUDE_SOME_FUNCTIONALITY).

note

There is no support to define constexpr values; it'll be added in a later version.

Exported header files

A package's exported header files should specify the interface provided by that package and avoid any implementation details. However, there may be performance or other reasons why implementation details occasionally need to be present in the exported headers.

Configurability has a number of effects on the way exported header files should be written. There may be configuration options that affect the interface of a package, not just the implementation. It is necessary to consider nested #includes and how this affects package and application builds. A special case of this relates to whether or not exported header files should #include configuration headers. These configuration headers are exported but should only be #included when necessary.

note

To be clarified.

Configurable functionality

Many configuration options affect only the implementation of a package, not the interface. However, some options will affect the interface as well, which means that the options have to be tested in the exported header files. Some implementation choices, such as whether or not a particular function should be inlined, also need to be tested in the header file due to language limitations.

Consider a configuration option OS_INCLUDE_KERNEL_MUTEX_TIMEDLOCK which controls whether or not a function os_mutex_timedlock() is provided. The exported kernel header file os/kernel/kapi.h could contain the following:

#include <pkgconf/kernel.h>
...
#ifdef OS_INCLUDE_KERNEL_MUTEX_TIMEDLOCK
extern bool os_mutex_timedlock(os_mutex_t*);
#endif

This is a correct header file, as it defines the exact interface provided by the package at all times. However, it has a number of implications. First, the header file is now dependent on pkgconf/kernel.h, so any changes to kernel configuration options will cause os/kernel/kapi.h to be out of date, and any source files that use the kernel interface will need rebuilding. This may affect sources in the kernel package, other packages, and application source code.

Second, if the application uses this function but the application developer has misconfigured the system and disabled this functionality, there will be a compile-time error when building the application. Note that other packages should not be affected, as they should impose appropriate constraints on OS_INCLUDE_KERNEL_MUTEX_TIMEDLOCK if they use that functionality (although some dependencies like this may be missed by component developers).

An alternative approach would be:

extern bool os_mutex_timedlock(os_mutex_t*);

Effectively, the header file is now misrepresenting the functionality provided by the package. The first result is that there is no longer a dependency on the kernel configuration header. The second result is that an application file using the timedlock() function will now compile, but the application will fail to link. At this stage, the application developer still has to intervene, change the configuration, and rebuild the system. However, no application recompilations are necessary, just a relink.

Theoretically, it would be possible for a tool to analyse linker errors and suggest possible configuration changes that would resolve the problem, thereby reducing the burden on the application developer. No such tool is planned in the short term.

It is up to component authors to decide which of these two approaches should be preferred. Note that it is not always possible to avoid #include-ing a configuration header file in an exported one. For example, an option may affect a data structure rather than just the presence or absence of a function. Issues like this will vary from package to package.

Nested #include's

As a general principle, unnecessary #include directives should be avoided. A header file should only #include the header files essential for defining its interface. Including additional header files increases the likelihood that package or application source files will become dependent on configuration header files, leading to unnecessary rebuilds when minor configuration changes occur.

Including configuration headers

Exported header files should avoid #include directives for configuration header files unless absolutely necessary, to prevent unnecessary rebuilding of both application code and other packages when minor configuration changes occur. An #include is only required when a configuration option affects the exported interface or when it influences implementation details controlled by the header file, such as whether a particular function is inlined.

There are a couple of ways to address the issue of unnecessary rebuilding. The first approach would involve more intelligent handling of header file dependencies by the tools, particularly the compiler and the build system. This would necessitate changes to various non-xCDL tools. An alternative approach would be to support finer-grained configuration header files. For example, there could be a file pkgconf/libc/inline.h that controls which functions should be inlined. This could be achieved through relatively simple extensions to the component framework, but it complicates ensuring the correctness of package header files and source code. A C preprocessor #ifdef directive does not distinguish between a symbol not being defined because the option is disabled or because the appropriate configuration header file has not been #included. It is likely that a cross-referencing tool would need to be developed first to catch such issues before the component framework could support finer-grained configuration headers.

Package documentation

On-line package documentation should be published in a project web. The component framework imposes no special limitations: component authors can decide which version of the HTML specification should be followed; they can also decide on how best to cope with the limitations of different browsers. In general it is a good idea to keep things simple.

Docusaurus is a modern solution for creating documentation websites.

Test cases

Packages should typically include one or more test cases. This enables application developers to verify that a given package functions correctly on their specific hardware and within their particular configuration. Consequently, developers are more likely to investigate potential bugs in their own code rather than immediately attributing issues to the component authors.

Testing frameworks

xCDL does not include a specific testing infrastructure, allowing tests to utilise any C/C++ testing framework.

In addition to industry-standard frameworks such as Google Test, xCDL should also support lightweight frameworks like µTest++.

Running tests on the host

Where possible, tests should be written to be highly portable and capable of running on the host as command line applications. These applications should return a non-zero exit code if the test fails.

Continuous integration

The first choice for a testing platform is Google Actions, and xCDL should be easily integrated into the GitHub Actions workflows.

As a second choice, xCDL should provide support to collect information about all tests contributed by all packages available in a configuration and run all of them from a scriptable environment, in order to facilitate integration into continuous integration tools like Jenkins.

Semihosted tests

xCDL should also support running the same tests on actual hardware or simulators like QEMU, utilising the semihosting infrastructure.

TODO: update xCDL tests definitions

As of this writing, the support for building and running test cases via the component framework is under review and subject to change. Currently, each test case should comprise a single C or C++ source file that can be compiled with the package's compiler flags and linked as any application program. Each test case should employ the testing API defined by the infrastructure.

Host-side support

Occasionally, it would be beneficial for an xCDL package to include host-side support. This could involve an additional tool required to build the package, an application designed to communicate with the target-side package code and display monitoring information, or a utility necessary for running the package test cases, particularly for device drivers.

Currently, the component framework does not provide support for host-side software, and there are significant challenges related to ensuring portability across different host machines. This issue may be addressed in a future release. While custom build steps can sometimes be adapted to perform tasks on the host side rather than the target side, this approach is not recommended.

Making a package distribution

Developers of xCDL packages are advised to distribute their packages as xpm packages. Packages in this format can be robustly added to existing xCDL component repositories using the package administration tools.

To achieve this, packages must include a valid package.json file with the xpack property, as required by xpm.

The easiest way to consume xpm packages is to published them on the npmjs.com public repository.

For details, please see the xpm documentation.

Credits

The initial content of this page was based on Chapter 2. Package Organization of The eCos Component Writer's Guide, by Bart Veer and John Dallaway, published in 2001.

Also:

• The eCos Component Writer's Guide - 2. Package Organization