ci_helloworld

Description

This repo provides a simple example for how to setup various CI services as well as integrating analysis tools into these services. These tools should be used as part of a comprehensive Software Development Process (SDP) and can also be used as a starting template for any C or C++ application. The following CI tools are used, providing testing support for Windows, Cygwin, Linux and macOS

• Travis CI
• AppVeyor

The following checks are performed:

• Doxygen
• Git Check
• Astyle
• Clang Tidy
• CppCheck
• Codacy
• Coverity Scan
• Codecov
• Coveralls
• Google Sanitizers
• Valgrind
• GCC Compiler Tests
• Clang Compiler Tests
• XCode Compiler Tests
• Visual Studio Compiler Tests

The following real world projects use a variety of these techniques as part of their SDP:

• Bareflank Hypervisor
• JSON
• Neovim
• GSL

Dependencies

Although this repo can be made to run on most systems, the following are the supported platforms and their dependencies:

Ubuntu 16.10 (or Higher):

sudo apt-get install git build-essential cmake

Windows (Cygwin):

setup-x86_64.exe -q -P git,make,gcc-core,gcc-g++,cmake

Windows (Visual Studio):

Install the following packages:

• Visual Studio SDK 10
• Visual Studio 2017
  • Check "Desktop development with C++"
  • Check "C++ CLI / Support"
  • Check "Standard Library Modules"
• CMake v3.6+
• Git for Windows

macOS:

• XCode 7.3+
• CMake v3.6+

Compilation / Testing / Installation

To compile and install this example, use the following instructions:

GCC / Clang

git clone https://github.com/ainfosec/ci_helloworld.git

mkdir ci_helloworld/build
cd ci_helloworld/build

cmake ..

make
make test

Visual Studio 2017 (NMake)

git clone https://github.com/ainfosec/ci_helloworld.git

mkdir ci_helloworld/build
cd ci_helloworld/build

cmake -G "NMake Makefiles" ..

nmake
nmake test

Visual Studio 2017 (MSBuild)

git clone https://github.com/ainfosec/ci_helloworld.git

mkdir ci_helloworld/build
cd ci_helloworld/build

cmake -G "Visual Studio 15 2017 Win64" ..

msbuild ci_helloworld.sln
ctest

XCode 7.3+

git clone https://github.com/ainfosec/ci_helloworld.git

mkdir ci_helloworld/build
cd ci_helloworld/build

cmake ..

make
make test

Analysis Tools

The following provides a description of all of the analysis tools that have been integrated into the CI services used by this project including an explanation of how it works.

Doxygen

The CI is setup to check for missing documentation using doxygen. Unlike most of the analysis tools used in this project, there is no make target for doxygen, and instead it is run using doxygen manually with the following script:

- doxygen .doxygen.txt
- |
  if [[ -s doxygen_warnings.txt ]]; then
    echo "You must fix doxygen before submitting a pull request"
    echo ""
    cat doxygen_warnings.txt
    exit -1
  fi

This script runs doxygen against the source code and any warnings are placed into a file called doxygen_warnings.txt. If this file is empty, it means that the doxygen analysis passed, and all of the code is documented based on the settings in the .doxygen.txt configuration file. If this files is not empty, the test fails, and prints the warnings generated by doxygen.

Git Check

git diff --check provides a simple way to detect when whitespace errors has been checked into the repo, as well as checking when end-of-file newlines are either missing, or contain too many. More information about this check can be found here. This check is extremely useful for developers when PRs contain modifications unrelated to their specific changes.

- |
  if [[ -n $(git diff --check HEAD^) ]]; then
    echo "You must remove whitespace before submitting a pull request"
    echo ""
    git diff --check HEAD^
    exit -1
  fi

This check simply runs git diff --check, which returns with an error if the check fails. If this occurs, the diff is displayed for the user to see.

Astyle

Source code formatting is a great way to keep a consistent look and feel with the code. The problem with source formatting is, unless everyone is using it, developers PRs will contain modifications unrelated to their specific changes, or worse, attempting to fix source formatting periodically will destroy your repo's git history each time you format the source. Therefore, if source formatting is to be used, it should be checked on every code diff to ensure the formatting is correct.

For this example, we use Astyle, but Clang Format will also work. To support Astyle in a simple way, we provide a make target that allows the developer to format their source code by simply running make format. To do this, we must first get Astyle:

list(APPEND ASTYLE_CMAKE_ARGS
    "-DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}"
)

ExternalProject_Add(
    astyle
    GIT_REPOSITORY      https://github.com/Bareflank/astyle.git
    GIT_TAG             v1.2
    GIT_SHALLOW         1
    CMAKE_ARGS          ${ASTYLE_CMAKE_ARGS}
    PREFIX              ${CMAKE_BINARY_DIR}/astyle/prefix
    TMP_DIR             ${CMAKE_BINARY_DIR}/astyle/tmp
    STAMP_DIR           ${CMAKE_BINARY_DIR}/astyle/stamp
    DOWNLOAD_DIR        ${CMAKE_BINARY_DIR}/astyle/download
    SOURCE_DIR          ${CMAKE_BINARY_DIR}/astyle/src
    BINARY_DIR          ${CMAKE_BINARY_DIR}/astyle/build
)

This cmake logic uses ExternalProject_Add to automatically download Astyle, compile it for your platform, and install it into your build directory so that it can be used by our custom make target. Note that we use our own patched version of Astyle that changes the build system from Astyle's custom set of Makefiles to a CMake build system for simplicity.

list(APPEND ASTYLE_ARGS
    --style=1tbs
    --lineend=linux
    --suffix=none
    --pad-oper
    --unpad-paren
    --break-closing-brackets
    --align-pointer=name
    --align-reference=name
    --indent-preproc-define
    --indent-switches
    --indent-col1-comments
    --keep-one-line-statements
    --keep-one-line-blocks
    --pad-header
    --convert-tabs
    --min-conditional-indent=0
    --indent=spaces=4
    --close-templates
    --add-brackets
    --break-after-logical
    ${CMAKE_SOURCE_DIR}/include/*.h
    ${CMAKE_SOURCE_DIR}/src/*.cpp
    ${CMAKE_SOURCE_DIR}/test/*.cpp
)

if(NOT WIN32 STREQUAL "1")
    add_custom_target(
        format
        COMMAND ${CMAKE_SOURCE_DIR}/bin/astyle ${ASTYLE_ARGS}
        COMMENT "running astyle"
    )
else()
    add_custom_target(
        format
        COMMAND ${CMAKE_SOURCE_DIR}/bin/astyle.exe ${ASTYLE_ARGS}
        COMMENT "running astyle"
    )
endif()

To create our custom astyle make target, we use the above CMake code. This points CMake to the resulting astyle binary depending on the platform, and provides astyle with the formatting options and source files specific to this project.

- cmake -DENABLE_ASTYLE=ON -DCMAKE_CXX_COMPILER="g++-6" ..
- make
- make format
- |
  if [[ -n $(git diff) ]]; then
    echo "You must run make format before submitting a pull request"
    echo ""
    git diff
    exit -1
  fi

Finally, to verify on each PR that a code change adheres to our Astyle configuration, we add the above code to our Travis CI script. This creates our make format target and executes it to format the code. If make format formats the code, a diff will be created which git diff can be used to detect. If no diff is created, it means all of the source adheres to our Astyle configuration, and the test passes.

Clang Tidy

Clang Tidy provides static analysis. Support for this tool starts by adding the following to the CMakeLists.txt:

set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

This tells CMake to record all of the compilation instructions used to compile your project including flags and definitions. Clang Tidy will use this information to statically analyze your project the same way it was compiled. The advantage to this approach is a significant improvement in accuracy. The main disadvantage to this approach is Clang Tidy is not good at statically analyzing files that do not show up in this compilation database (such as header files). For this reason, if you want to analyze a header, it has to be included by a source file that is included in the compilation database.

list(APPEND RUN_CLANG_TIDY_BIN_ARGS
    -clang-tidy-binary ${CLANG_TIDY_BIN}
    -header-filter=.*
    -checks=clan*,cert*,misc*,perf*,cppc*,read*,mode*,-cert-err58-cpp,-misc-noexcept-move-constructor
)

add_custom_target(
    tidy
    COMMAND ${RUN_CLANG_TIDY_BIN} ${RUN_CLANG_TIDY_BIN_ARGS}
    COMMENT "running clang tidy"
)

Finally, Clang Tidy is given its own make target to simplify its use. Here we tell the run-clang-tidy-4.0.py script which clang tidy binary to use, as well as which checks to perform, and what header files to include, which is all of them. We turn off the -cert-err58-cpp test because it triggers code from catch.hpp, and -misc-noexcept-move-constructor because it is still buggy in 4.0. Note that we choose a specific version of Clang Tidy which is important because each new version of Clang Tidy fixes bugs and adds new checks, resulting in different results depending on which version you use.

- cmake -DENABLE_CLANG_TIDY=ON -DCMAKE_CXX_COMPILER="g++-6" ..
- make
- make tidy > output.txt
- |
  if [[ -n $(grep "warning: " output.txt) ]] || [[ -n $(grep "error: " output.txt) ]]; then
      echo "You must pass the clang tidy checks before submitting a pull request"
      ech