Continuous Integration (CI) for large scale package-based C, C++ projects with Conan2

While this work easily for third party dependencies, when dependencies are very stable, quick evolution in a large dependency graph when there are different developers working concurrently in different packages, can be more challenging.

With Conan 2 new tools it is possible to implement an efficient Continuous Integration process for large scale projects, computing what packages need to be built, and in what specific order and parallelism.

The concept of "products pipeline" will be introduced as an scalable approach that allows to focus on the business value while allowing efficiency. The dependency graph build order for every different product can be computed, for every different configuration (platform, compiler, build_type, etc). This is done efficiently, taking into account the versioning scheme and the package types (header-only, static library, shared library, application) with a explicit novel model of the dependencies effect in the packages binaries.

Then all the different computed graph "build-orders" can be merged into a single one to avoid duplicated builds. The final graph merged "build-order" can be distributed efficiently to different build agents, as it also specifies which package builds can be done in parallel. For projects with concurrent changes, using lockfiles is very important in this stage to guarantee consistency and reproducibility of the dependencies.

Finally, the concept of stages in the CI process using multiple server repositories and copying of packages across those repositories will be presented. This process is known as package promotions and is a known best practice in DevOps for other technologies, and is critical to avoid disruptions to developers and production systems that the CI builds could introduce.

The talk will present both the theoretical foundations and a full real example with available source code to demonstrate the presented functionalities.