Benchmark Tests

Description:

The benchmark test provides data points to help in the checking of application performance and establishes whether changes to code base, improved or deteriorated the performance.

The CI system is responsible of running the benchmark tests regularly.

Details:

  • Can run against any build type

  • Executed by CI-pipeline

  • Written in pure shell to run on targets with very limited environment

To run the benchmark test from within the source (Developer):

First ensure the build is present that shall be tested. Debug is assumed as default.

./ci/build.sh [Release]

    Run the benchmark test suite or parts of it against actual build. Default
    is to run all benchmark tests.

    Usage: run_benchmark.sh [option] [build type]

    Options:

    -h|--help:                  print this help

    Build Type:

    Debug   - Run benchmark test or interactive shell against the Debug Build
    Release - Run benchmark test or interactive shell against the Release Build
    ...     - Any build type located in build/ can be provided

    <Debug> is the default build type, if nothing is specified.

    Examples:

    $>run_benchmark.sh # run all benchmark tests on Debug build

    $>run_benchmark.sh Release # run all benchmark tests on Release build

Run on a Target (Integrator):

To run the benchmark test install all content in tests/benchmark to a custom location on the target system. Currently it is not packaged and installed by samconf build system itself.

Prerequisites :

  • Samconf is installed on target system

  • Samconf benchmark test package is installed

  • At least a writeable directory to store results and for intermediate files

Running the Test Suite:

To execute the benchmark tests run the installed bench_test.sh script. Here is an example script to for benchmark test integration.

./test/benchmark/bench_test_env.sh

Extended configuration

Depending on the installation and configuration it could be necessary to adjust some settings. The following Environment variables can be used:

#!/bin/sh

export DIST_DIR="${DIST_DIR-"${BUILD_DIR}/dist"}"
export LD_LIBRARY_PATH="${BASE_DIR}/build/deps/lib:${DIST_DIR}/usr/local/lib"
export PATH="${PATH}:${DIST_DIR}/usr/local/bin/"
export BENCHMARK_DIR="${BENCHMARK_DIR-"$(dirname "$0")"}"
export BENCHMARK_RESULT_DIR="${BENCHMARK_RESULT_DIR-"$BUILD_DIR/result/benchmark_results"}"
export SAMCONF_SIGNATURE_KEY="${PUBLIC_KEY}"