A long long time ago, after years and years of fighting with software that doesn’t build, I made a life changing decision. Any and every software I write must download, install and setup its dependencies without requiring user interaction. All of them. Allways. When combined with other build best-practices, not pasting dependencies in your source tree and locally installing your dependencies, the rule has led to some interesting CMake usage.
This post isn’t the standard “modern CMake” fair. It is a collection of funky and unorthodox techniques I find cool and useful. That is it, nothing more. For these to work, I expect the user has 3 and only 3 tools. Conan, CMake and a compiler.
ExternalProject
Lets start with something simple. I use ExternalProject in virtually every new project. If the world was a perfect place, every project would provide a Conan recipe. If it didn’t, it would at least provide a CMake file and an install target.
The world is not a perfect place.
Not even close.
Some people still use autotools and expect us t- [redacted rant about autotools - The Editor].
I use ExternalProject_Add for many third party libraries. It is the perfect last resort when all else fails. Personally, I rather like it. Here’s how I use it.
At the top of your project, you will need to include ExternalProject and set a link directory. Furthermore, to make things simple down the line, I like to output my binary packages into predefined folders, creatively called bin and lib. I also like to gather up my dependency IDE projects and the miscellaneous CMake targets into a nice IDE folder called, you guessed it, Dependencies.
cmake_minimum_required(VERSION 3.14)
project(my_project CXX)
include(ExternalProject)
link_directories(${CMAKE_CURRENT_BINARY_DIR}/lib)
# Organize unrelated targets to clean IDE hierarchy.
set(DEPENDENCY_FOLDER "Dependencies")
# This will also clean up the CMake ALL_BUILD, INSTALL, RUN_TESTS and ZERO_CHECK projects.
set_property(GLOBAL PROPERTY USE_FOLDERS ON)
set_property(GLOBAL PROPERTY PREDEFINED_TARGETS_FOLDER ${DEPENDENCY_FOLDER})
# Output binary to predictable location.
set(BINARY_OUT_DIR ${CMAKE_BINARY_DIR}/bin)
set(LIB_OUT_DIR ${CMAKE_BINARY_DIR}/lib)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${BINARY_OUT_DIR})
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${LIB_OUT_DIR})
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${LIB_OUT_DIR})
foreach(OUTPUTCONFIG ${CMAKE_CONFIGURATION_TYPES})
string(TOUPPER ${OUTPUTCONFIG} OUTPUTCONFIG)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_${OUTPUTCONFIG} ${BINARY_OUT_DIR})
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY_${OUTPUTCONFIG} ${LIB_OUT_DIR})
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY_${OUTPUTCONFIG} ${LIB_OUT_DIR})
endforeach(OUTPUTCONFIG CMAKE_CONFIGURATION_TYPES)
# clang-format, just because <3 clang-format
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/.clang-format
${CMAKE_CURRENT_BINARY_DIR}/.clang-format COPYONLY
)Lets use some third-party libraries. First off, the misbehaving SG14 repository, which doesn’t provide an install target (a PR is open to fix this). How does one install a header-only library without an install target? Simple, copy the header files.
set(LIB_INCLUDE_DIR ${CMAKE_CURRENT_BINARY_DIR}/include)
target_include_directories(${PROJECT_NAME} PUBLIC ${LIB_INCLUDE_DIR})
ExternalProject_Add(sg14_ext
GIT_REPOSITORY https://github.com/WG21-SG14/SG14.git
CONFIGURE_COMMAND ""
BUILD_COMMAND
COMMAND ${CMAKE_COMMAND} -E copy_directory <SOURCE_DIR>/SG14
${LIB_INCLUDE_DIR}/sg14
INSTALL_COMMAND ""
UPDATE_COMMAND ""
)Here, we override most of CMake’s COMMANDS, so the project doesn’t copy files every time you change something in your sources.
Lets also download and install another useful library, units by Nic Holthaus. This one does provide install support, so ExternalProject_Add will automatically install it in our local prefix. Nice.
ExternalProject_Add(units_ext
GIT_REPOSITORY https://github.com/nholthaus/units.git
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}
-DCMAKE_PREFIX_PATH=${CMAKE_CURRENT_BINARY_DIR}
-DBUILD_TESTS=Off
UPDATE_COMMAND ""
)As you can see, we can pass on CMake options to the library. Here we give it our local install path and respectfully ask not to build unit tests.
Finally, to get these libraries to download and install, we must set a dependency from our target to the libraries. We will also group these projects in the Dependencies folder mentioned earlier.
# Pretty IDE folder
set_target_properties(sg14_ext units_ext PROPERTIES FOLDER ${DEPENDENCY_FOLDER})
# This sets a dependency from our project to the libs.
add_dependencies(${PROJECT_NAME} sg14_ext units_ext)ExternalProject has many more features. It can download compressed archives, local directories, SVN and the likes. You can provide it with branches or tags. But ultimately, the basics remain the same. Download something, copy or install it, and add a dependency from your project to the new targets.
One Trick To Rule Them All
Lets get real. What I am about to show you isn’t thoroughly documented. I cannot recall exactly where I read this (I believe it was some gtest documentation), but it has been a game changer for me. I am unsure this is a deliberate CMake feature, so lets keep it hush hush just in case Kitware “fix” this.
One main issue I bump into time and time again is ordering. When you get creative with your build, you often need to run external commands and what-not, which can become difficult to order correctly. Especially if you just want those to run before your build. Well, it so happens you can fix that quite easily with CMake include. If you add includes at the top of your CMakeLists.txt, before anything else, the included scripts will execute before the rest of your build.
Lets look at a simple example : running Conan from CMake without requiring user input. I know the Conan team doesn’t encourage this practice (sorry guys!), but it does come in useful when calling a CMake project from ExternalProject_Add for example. Of course, the appropriate solution is to publish a Conan recipe. Sometimes laziness wins ;)
CMakeLists.txt
include(${CMAKE_CURRENT_SOURCE_DIR}/CMakeLists.conan.txt)
cmake_minimum_required(VERSION 3.10)
project(my_awesome_project CXX)In the CMakeLists.conan.txt, we can use execute_process to run external commands. This is very handy.
CMakeLists.conant.txt
cmake_minimum_required (VERSION 3.10)
project(conan-setup NONE)
execute_process(COMMAND conan install ${CMAKE_CURRENT_SOURCE_DIR} --build missing -s build_type=Debug)
execute_process(COMMAND conan install ${CMAKE_CURRENT_SOURCE_DIR} --build missing -s build_type=Release)Now, when a user calls CMake setup, Conan will download and install the necessary dependencies. Sweet.
NASM And A Custom Linker
One of my hobby projects is a 4K intro (4K as in 4KB executable). This sort of project is a great way to experiment with things you wouldn’t normally get to play with in typical software. As always, I stick to my rule of thumb and don’t want to install external tools when building it. This lead me down an interesting path. Lets dive in!
First off, I needed an assembler. Fortunately, and unbeknownst to me, CMake does support NASM. You can enable it with the following lines, and CMake will process .asm files appropriately.
set(CMAKE_ASM_NASM_COMPILER ${NASM_PATH})
enable_language(ASM_NASM)However, the user must install NASM. That is a big no-no! So lets download and install NASM in a local build tools directory for the user, and provide that path to CMake. First, we’ll include the appropriate setup file. Next, in that setup file, we will download, extract and locally install NASM. Ezpz.
CMakeLists.txt
include(${CMAKE_CURRENT_SOURCE_DIR}/CMakeLists.nasm.txt)CMakeLists.nasm.txt
cmake_minimum_required (VERSION 3.10)
project(nasm-download NONE)
set(BUILD_TOOLS_DIR ${CMAKE_CURRENT_BINARY_DIR}/build_tools)
set(NASM_VER "2.13.03")
set(NASM_LINK "https://www.nasm.us/pub/nasm/releasebuilds/${NASM_VER}/win64/nasm-${NASM_VER}-win64.zip")
file(DOWNLOAD ${NASM_LINK} ${BUILD_TOOLS_DIR}/nasm.zip)
execute_process(
COMMAND ${CMAKE_COMMAND} -E tar xzf ${BUILD_TOOLS_DIR}/nasm.zip
WORKING_DIRECTORY ${BUILD_TOOLS_DIR}
)
set(NASM_PATH "${BUILD_TOOLS_DIR}/nasm-${NASM_VER}/nasm.exe")Now, we can use NASM in our project without worrying about a user (me) screwing up tool paths and what-not.
One strange thing about 4K intros is they use a custom linker/compressor to achieve such small executable sizes (amongst other insane tricks). This was a difficult nut to crack, since I haven’t found any CMake option that exposes your compiler’s linker path. Obviously this is a niche scenario. Before diving into the details, lets download and setup crinkler, the compressor/linker we will use.
CMakeLists.txt
include(${CMAKE_CURRENT_SOURCE_DIR}/CMakeLists.crinkler.txt)CMakeLists.crinkler.txt
cmake_minimum_required (VERSION 3.10)
project(crinkler-download NONE)
set(BUILD_TOOLS_DIR ${CMAKE_CURRENT_BINARY_DIR}/build_tools)
set(CRINKLER_VER "20a")
set(CRINKLER_LINK "http://www.crinkler.net/crinkler${CRINKLER_VER}.zip")
file(DOWNLOAD ${CRINKLER_LINK} ${BUILD_TOOLS_DIR}/crinkler.zip)
execute_process(
COMMAND ${CMAKE_COMMAND} -E tar xzf ${BUILD_TOOLS_DIR}/crinkler.zip
WORKING_DIRECTORY ${BUILD_TOOLS_DIR}
)
execute_process(
COMMAND ${CMAKE_COMMAND} -E copy ${BUILD_TOOLS_DIR}/crinkler${CRINKLER_VER}/crinkler.exe
${BUILD_TOOLS_DIR}/link.exe
)The linker is ready to be used, now what? To get MSVC to change its linker, we will need a props file. These files are specific to Visual Studio, and can customize your IDE solution to a greater extent than CMake. In it, we will simply add the build_tools executable path to the generated solution. MSVC will use the custom linker, as it is called link.exe. Yes, that is pretty hacky.
crinkler.props
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ImportGroup Label="PropertySheets" />
<PropertyGroup Label="UserMacros" />
<PropertyGroup>
<ExecutablePath>$(ProjectDir)build_tools\;$(ExecutablePath)</ExecutablePath>
</PropertyGroup>
<ItemDefinitionGroup />
<ItemGroup />
</Project>Now we have quite a few things to add to our main CMakeLists file to get this all to work. In addition, crinkler options are provided through linker flags, in this case LINK_FLAGS_RELEASE. Here is what it looks like.
CMakeLists.txt
# Somewhere, set this.
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /MANIFEST:NO")
# These are crinkler options.
set_target_properties(${PROJECT_NAME} PROPERTIES LINK_FLAGS_RELEASE "/NODEFAULTLIB /SUBSYSTEM:WINDOWS
/ENTRY:release_main /DYNAMICBASE:NO /LARGEADDRESSAWARE:NO /CRINKLER /HASHTRIES:300 /COMPMODE:SLOW /ORDERTRIES:4000
/PROGRESSGUI /RANGE:opengl32 /TRANSFORM:CALLS /NOINITIALIZERS /UNALIGNCODE /REPORT:crinkler_report.html")
# User the props file in your generated solution.
set_target_properties(${PROJECT_NAME} PROPERTIES VS_USER_PROPS ${CMAKE_CURRENT_SOURCE_DIR}/crinkler.props)In Guga’s famous words, we’re done! Our project is ready to roll, and all a user has to do is call cmake .. && cmake --build . The project will download and locally install NASM and Crinkler. The generated solution will use crinkler when compiling in release.
Python!? Eww
Recently, I’ve been working on a project that generates some header files and documentation in markdown format. I was scratching my head at what best scripting language to use for this task. Python is always a goto, but it does break the “rule of 3” [tools required to build a project]. Truth is, I use Python so rarely it is painful and slow to get up to speed with it. With std::filesystem in C++17, I find writing small parsers and such a breeze in C++. So after half an hour googling about python directory manipulation, I decided to give it a shot with C++. Specifically, I wanted RAII to generate scopes for namespaces, functions and classes in the header.
Using the previously mentioned trick, we will use a separate CMake file to setup and build our code generation tool. CMake will then run that tool, which will output headers in our base source tree. The main CMake project will then use those headers to create an INTERFACE library with an install target. Third-party users and our gtest target will consume this interface library.
The generator project is a typical CMake project. It uses ExternalProject as mentioned previously. The source file comes from generator_src/*. The only difference here is a POST_BUILD custom command to run the built tool. Here is the important line.
CMakeLists.generator.txt
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_CURRENT_BINARY_DIR}/bin/${PROJECT_NAME}
)Once ran, the tool outputs the header in ../include which is our generated header-only library directory. It also generates documentation in ../readme.md. We can now create an INTERFACE library, as one normally would, using the headers.
CMakeLists.txt
include(${CMAKE_CURRENT_SOURCE_DIR}/CMakeLists.generator.txt)
# Usual CMake things...
file(GLOB_RECURSE HEADER_FILES "${PROJECT_SOURCE_DIR}/include/${PROJECT_NAME}/*.hpp")
add_library(${PROJECT_NAME} INTERFACE)
target_sources(${PROJECT_NAME} INTERFACE
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>
$<BUILD_INTERFACE:${HEADER_FILES}>
)
target_include_directories(${PROJECT_NAME} INTERFACE
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
)This creates a consumable header-only CMake package, which we then proceed to use in a gtest executable.
CMakeLists.txt
##
# Tests
##
include(GoogleTest)
option(BUILD_TESTING "Build and run tests." On)
if (${BUILD_TESTING})
enable_testing()
# Conan is love.
include(${CMAKE_BINARY_DIR}/conanbuildinfo_multi.cmake)
conan_basic_setup(TARGETS)
set(TEST_NAME ${PROJECT_NAME}_tests)
file(GLOB_RECURSE TEST_SOURCES "tests/*.cpp" "tests/*.c" "tests/*.hpp" "tests/*.h" "tests/*.tpp")
add_executable(${TEST_NAME} ${TEST_SOURCES})
# We link here.
target_link_libraries(${TEST_NAME} PRIVATE ${PROJECT_NAME} CONAN_PKG::gtest)
gtest_discover_tests(${TEST_NAME})
add_dependencies(${TEST_NAME} ${PROJECT_NAME})
# Build options ...
endif()I use Conan to install and link with gtest, because why not. We link with ${PROJECT_NAME} to consume the generated library. I wont show the technique I use to generate the install target, as I have been made aware it is now outdated and overly complex (merci Guillaume). I am still in the process of porting and experimenting with more recent CMake install scripts.
We are done. Third-party users can easily link with the interface library. When they use ExternalProject_Add, the library will be generated and everything should be dandy. Our users didn’t have to install Python or some other obscure scripting languages like chef.
Final Boss, Shaders & Custom Targets
The include trick I’ve shown previously isn’t the only, or even the “right” way to execute custom CMake commands before or after building your software. However, it is the least painful. custom_targets are the worst. I hate them and stay as far away as I can. However, after weeks of banging my head on them and their stamp files, I have a nice small CMake snippet I use to copy random files to my build directory. In this case, I will be copying shaders and binary data, if and only if, they have changed.
First, we will go through files in a specificied directory and generate STAMPS. These are empty files, which CMake uses to determine if our shaders have changed.
##
# Copy data on build.
##
set(DATA_IN_DIR ${CMAKE_CURRENT_SOURCE_DIR}/data)
set(DATA_OUT_DIR ${BINARY_OUT_DIR}/data)
set(STAMP_DIR ${CMAKE_CURRENT_BINARY_DIR}/stamps)
file(GLOB_RECURSE DATA_FILES "${DATA_IN_DIR}/*")
set(STAMP_FILES "")
foreach(FILE ${DATA_FILES})
get_filename_component(FILENAME ${FILE} NAME)
get_filename_component(FILE_PATH ${FILE} REALPATH DIRECTORY)
file(RELATIVE_PATH FILE_OUTPUT_RPATH ${DATA_IN_DIR} ${FILE_PATH})
set(STAMP_FILE ${STAMP_DIR}/${FILENAME}.stamp)
add_custom_command(
OUTPUT ${STAMP_FILE}
COMMAND ${CMAKE_COMMAND} -E make_directory ${STAMP_DIR}
COMMAND ${CMAKE_COMMAND} -E make_directory ${DATA_OUT_DIR}
COMMAND ${CMAKE_COMMAND} -E touch ${STAMP_FILE}
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${FILE}
${DATA_OUT_DIR}/${FILE_OUTPUT_RPATH}
DEPENDS ${FILE}
)
list(APPEND STAMP_FILES ${STAMP_FILE})
endforeach()We generate stamp files for each file in out data directory, and we add a custom_command which depends on the input file and outputs a stamp and copies the file when changed. We store the stamp files in a list.
Now, all we need is a custom_target that “represents” these files. We will make our main project dependent upon this target, just like we do for ExternalProject targets. When building our main project, CMake will look at the custom_target to determine if it is still up-to-date. If not, it will execute the previously set custom_command of changed files, which in turn will copy it to our output directory and generate a new up-to-date stamp file.
At least, that is how I understand it.
file(GLOB_RECURSE SHADERS "${DATA_IN_DIR}/shaders/*.glsl")
# The custom target
add_custom_target(my_shaders
SOURCES ${SHADERS}
DEPENDS ${STAMP_FILES})
# The main project depends on this target, so CMake will update it when needed.
add_dependencies(${PROJECT_NAME} my_shaders)And that’s it! On build time, modified shaders or assets will be copied to your build directory. This is a simple example, but should get you started when you need to copy files at build time.
Conclusion
I hope you learned a thing or two. I didn’t show much Conan, as it is super easy to use. Going forward, the need for ExternalProject should diminish with the advent of competent dependency managers. There will always be a package or 2 that misbehaves, or projects that need more creative solutions. These CMake snippets should come in handy when you are faced with such scenarios.