Alan's Q3 UpdateOct 27, 2023
- Boost Website
- Boost Release Tools
- Boost Libraries
- C++ Github Actions
- Gray-box local search
Over the last quarter, we have been working intensely on MrDocs, a documentation generator for C++ projects. I’ve been overseeing and reviewing all the work done by the other contributors in the project. I’ve also been responsible for:
- setting up and maintaining CI for the project;
- MrDocs and LLVM release binaries;
- build scripts;
- setting up and integrating dependencies;
- setting up and deploying the Antora toolchains and documentation to the project website;
- working on supporting libraries; and
- fixing bugs.
These are some of the highlights of the work done in the last quarter:
- Refactor library layout and scripts so all the implementation files are in
- Automated deployment of demo documentation to the project website. The website is updated on every commit to the
masterbranches. The action uses the actual release package to generate the documentation. This PR involved fixing all scripts and directory layouts for the release package to be properly usable. The new project layout uses the FHS standard. CMake modules to create a target to generate the documentation for projects were included in the installation. The MrDocs executable and libraries are installed as separate CMake package components. Any usage of FetchContent has been removed from the CMake scripts. The documentation was refactored to reflect the changes. Included complete instructions in the documentation pages for installation and usage, describing all commands and options.
- Deployed new LLVM binaries to the website and updated CI. The binaries were regenerated for all platforms in 4 modes:
DebWithOpt. This involved fixing long-standing bugs related to conflicts in LLVM versions used by the project and providing new pre-built binaries on the website. In particular, the previous pre-built binaries used a special ReleaseWithDebInfo LLVM configuration that caused conflicts with MrDocs on MSVC, being used with a Debug CMake configuration variant by developers. This eliminated the need for the ad-hoc GitHub LLVM binaries release and for the special docker container we had been using so far.
- Added support for CMakePresets. This allowed us to simplify the build process for MrDocs and its dependencies, which was previously counting on long command line commands and
CMakeUserPresets.jsonfiles without a corresponding
CMakePresets.jsonfile. It was also a step towards the new installation instructions in the documentation. An example file for
CMakeUserPresets.jsonincluding all compilers was provided. The base
CMakePresets.jsonfile included a special vendor configuration to hide base configurations from Visual Studio.
- Added complete installation instructions for the project. The instructions were adapted so that all commands are relative to an enclosing directory containing all the dependencies and MrDocs. Included instructions for all steps considering package managers, installed dependencies, or binaries provided on the project website. The CMake scripts were adapted to make it easier to build the project in a variety of environments according to the instructions.
- Included a polyfill implementation of
mrdocs::Expected. This implementation is currently being used by MrDocs and support libraries.
- Refactored the project name from MrDox to MrDocs
MrDocs includes a support library that reimplements the Handlebars template engine in C++. This module is used to generate documentation from templates.
Over the last quarter, this is the MrDocs support library in which I have been investing most of my time. All the development started in this same quarter in July. It already supports all features from the original Handlebars test suite, including all mustache features. The library is already integrated with MrDocs to generate the documentation for the project website.
- Initial proposal of the C++ Handlebars library.
- Fixed and refactored code that relied on references to temporaries generated by the Dom.
- Included support for inverse blocks without helpers
- All features, specs, and tests from the original Handlebars test suite were then ported to C++ and are now passing: basic specs; partials and automatic indentation; whitespace control; block helpers, mustache blocks, and chained blocks; subexpressions; builtin helpers; private data; helper formats; track-ids mode; strict mode; util; mustache. As many handlebars features were undocumented in the original implementation, adjusting our handlebars implementation, which was only designed to handle basic templates, to pass all tests from the original Handlebars test suite involved multiple significant refactors and improvements to the entire codebase. The tests are a superset of the mustache specs, which are also passing. The previously available SafeString type because a regular
dom::Valuewith the implementation of the specs.
- Support for
dom::Functionimplementation type was provided to support this use case.
- Support for error handling via
Expected. All functions that might fail have a variant that throws exceptions and another that returns an
Expectedvalue. Helpers functions are also allowed to propagate errors via
- Fixed a bug that caused MrDocs to emit
\nafter the transition to C++ handlebars.
- The “find” helper was adjusted to support strings, arrays, and objects. This fixes a problem with the variants of similar but conflicting helpers that were categorized as container and string helpers.
After completing the Handlebars implementation, I also included complete unit tests for the DOM. Tests were included for all DOM types and many bugs have been fixed.
Throughout the process,
- the implementation was completed for classes that were placeholders and existing bugs have been fixed
- the API was documented
Scopeobject was extended to support all types of alternative syntax to evaluate expressions.
- Value types: support for integers and floating point numbers.
- Value types: support for all dom::Value operations using the native
Scopefunctions that might fail were adjusted to return
MrDocs includes a support library for unit tests. The library was initially adapted from the Boost.URL unit tests and extended to support the needs of MrDocs.
I had previously implemented a smaller system for the Handlebars unit tests which was then integrated with the boost.url test suite library. Features from the handlebars test suite library were ported to the boost.url test suite library, including the expression decomposer and the diff algorithm for golden master tests.
With this integration, Handlebars tests were then listed among any other tests in the library. These tests were later complemented with regular MrDocs tests.
The decomposer has later been improved for integral comparison operators.
In this last quarter, the Boost website went beta on https://www.preview.boost.org/. Among the many support projects for the website, I’ve been helping the most on the
cppalliance/site-docs, which includes the Boost website documentation as an Antora project. Its components represent the “User Guide”, “Contributor Guide”, and “Formal Review” sections of the website.
Since the inception of the project, I’ve been overseeing and reviewing all the work done by the other contributors to the project. I’ve also been responsible for:
- setting up and maintaining CI for the project;
- coordinating with
cppalliance/temp-siteon content uploaded to AWS buckets;
- build scripts to be reused by the release tools and previews;
- writing sections of the documentation that require technical knowledge;
- developing custom Boost/Antora extensions, such as the Boost Macro extension;
- maintaining the Antora toolchain and templates; and
- adjusted Boost libraries to match formats expected by the website.
The Antora playbooks were recently adjusted to initially contain no content sources, now that the Antora-enabled build process also implemented by me was deployed in the official Boost release process.
As in other quarters, the Boost Library in which I have been investing most of my time is Boost.URL, since it’s our most recently accepted library. The library is in maintenance mode since our focus shifted to MrDocs, but considering how recent it is, there is a constant demand for work fixing bugs and improving the documentation. In Boost.URL, I’ve been responsible for:
- upgrading CI, mostly coordinating with the C++ Github Actions;
- maintaining, simplifying, and updating build scripts;
- integrating more spec tests, such as the Ada tests included more recently;
- including more examples, such as the more recent sanitize-URL example;
- fixing documentation content that is out of date; and
- fixing bugs.
Besides bugs, the library was recently refactored to remove what was previously called a “header-only” mode and deprecated several aliases, which caused some small friction in this last quarter. These are some of the highlights of the work done in the last quarter:
- Extended fuzz testing.
Fuzzwas included as a new clang factor in CI. The process was adjusted so that the corpus is properly reused from and stored in GitHub action archives. CMake scripts were refactored to include CMake options that control the fuzzer parameters. Fuzzers were included for each of the grammar rules for URLs.
IPv6addrz. This is an issue where a valid
IPv6addrzwasn’t being considered an
IP-literal. IPv6addrz includes a
ZoneIDat the end, delimited by an encoded
ipv6_addressclass is unmodified, as the mapping from the
std::uint32_tis dependent on the application context. The original
ZoneIDcan be obtained from the url_view but the library is agnostic about it.
- Included GDB pretty printers and documentation. All available URL components are now pretty printed in GDB. A developer mode was also included which prints the URL components in a format corresponding to the internal URL string offsets.
- Updated the content of both the documentation and README.adoc so that they match current best practices. The documentation in README.md contained dated and incorrect information, while the quickbook documentation was missing important information and contained bad practices.
Some relevant bug fixes were:
- Enforced that appropriate CMake BOOST_INCLUDE_LIBRARIES are set according to the build options. The previous implementation included these extra libraries whenever they were available, which caused problems for other libraries that depended on Boost.URL via
- Fix and include unit-tests for issues #755 and #446
- We disabled and re-enabled drone caching. This is related to a bug where Drone would attempt to cache the
b2binary, which would cause conflicts. This was fixed with PR #123 in boostorg/boost-ci.
- Remove quickbook references to variant. Links and references to
variantin both the
.xmlfiles were removed as the
variantalias had been deprecated in 96438f6.
- Updated javadoc deprecated references. This fixed a mistake where the Javadoc for many deprecated aliases included references to the deprecated alias
boost::core::string_viewinstead of the correct deprecated aliases
@refprefixes with backtick for references in javadocs. This caused an issue in the documentation since several aliases were deprecated in favor of symbols from Boost.Core.
- Refactored variable name in url::set_params. This issue was causing an error in the documentation for
- Fix bug where
string_viewconstructor would require a non-
url_view_baseas input. This cased the constructor to reparse the
string_viewfrom an already parsed
- Fix bug where
std::string_views. This would make the query include values in the underlying std::string beyond the expected string_view.
- Updated changelog for boost release.
Besides Boost.URL, as usual, I’ve been overseeing and fixing smaller issues with other boost libraries, such as Boost.Docca, Boost.StaticString, and helping with libraries by other contributors when asked for assistance, as in a more recent case with Boost.Outcome.
In particular, we had to fix a smaller issue in Boost.Docca that was also affecting Boost.URL. The issue involved Boost.Docca’s dependence on a deprecated version of Doxygen that is no longer supported by the Boost toolchain.
Boost Release Tools
Over the last quarter, I’ve been working on the integration of toolchains I developed into the Boost Release Tools to add support for features desired for the new website.
Some of the highlights of the work done in the last quarter:
- Introduced support for libraries with Antora documentation into the official Boost release process. Deployed new docker containers that include NodeJS, Gulp, and the Antora toolchain. With this enhancement, each library now can function as an Antora component within an Antora master project hosted in a separate repository (https://github.com/cppalliance/site-docs). This master project repository also contains additional components, such as the user guide, contributor guide, and a dedicated component for the review process. In a subsequent phase of the release process, this Antora documentation is seamlessly merged with the pre-existing in-source documentation, which has been generated using various other tools. When a library is “Antora-enabled”, the release process will automatically generate the Antora documentation and publish it with the documentation of other libraries. No
b2scripts are required to generate the documentation for the library. All Antora-enabled libraries use the same master Antora UI template that matches the design of the boost website. Antora playbooks were adjusted to initially contain no content sources, now that the Antora-enabled build process in deployed in the official Boost release process.
- Deployed new container to
boostorg/boostto support the new release process.
- New archive variants for boost. Add extra archive variants such as
boost-source. These variants can reduce expenses with JFrog download bandwidth, provide users with archives that are simpler to use, and provide docs-only archives for the website. The new MakeBoostDistro.py script includes parameters to determine what types of files should be included in the distribution. All other functions are adapted to handle these requirements accordingly. Switching to source-only downloads would save Boost $1000 per month.
C++ Github Actions
C++ Github Actions is a project I created and have been maintaining since the second quarter of the year. It is a collection of reusable Github Actions for any C++ project that needs to be tested on a variety of compilers. Both MrDocs are Boost.URL are currently using these actions in their CI.
The project includes actions to:
- Generate a Github Actions Matrix for C++ projects;
- Setup C++ compilers;
- Install and setup packages;
- Clone Boost modules;
- Run complete CMake and
- Generate changelogs from conventional commits;
- Generate summaries; and
- Generate time-trace reports and flame graphs
These actions include a myriad of options and features.
- The “setup-*” actions include logic to detect, install, and cache dependencies, which can be used by the CMake and
- Individual options and actions attempt to set up a wide variety of compilers on different platforms, including MSVC, GCC, Clang, MinGW, AppleClang, and Clang-CL.
- Actions that generate reports include a multitude of tools and options to analyze changes, time traces, and coverage.
Since then, these actions have been adapted as needed to support the needs of MrDocs and Boost.URL, which have also been using conventional commits. Here’s a recent summary report generated for Boost.URL by the CI workflow: https://github.com/boostorg/url/actions/runs/6512424067
The project documentation also uses the Antora UI template we have been maintaining for all other projects: https://alandefreitas.github.io/cpp-actions
Gray-box local search
On 13 September 2023, the following paper I co-authored was published:
Lopes, R.A., Freitas, A.R.R. Gray-box local search with groups of step sizes. Soft Computing. p. 1-14 2023 https://doi.org/10.1007/s00500-023-09129-1
The paper was accepted on 18 August 2023 and published on 13 September 2023.
While the paper is more aligned with my educational background than my daily C++ Alliance tasks, it is one more paper that carries the C++ Alliance in the affiliation, contributing to its reputation.