Testing Idempotence and Convergence for Infrastructure as Code
(Accompanying Material and Test Results)
Waldemar Hummer, Florian Rosenberg, Fábio Oliveira, Tamar Eilam
This Web page contains accompanying material for the paper
'Testing Idempotence and Convergence for Infrastructure as Code'. The data listings on
this page provide in-depth insights into the testing approach, contain detailed reports
of the test results, and list some identified bugs.
To briefly introduce the problem domain, we refer to the abstracted system model depicted in
Figure 1. The core entity is denoted automation, which represents the logic of a piece of
Infrastructure as Code (IaC, see here
for a brief overview).
An automation consists of multiple automation tasks,
which are implemented by some script code and are supposed to cause potential state
changes. From the runtime perspective, an automation run represents the entire
execution of an automation, which itself consists of multiple task executions. Each
task execution effects a set of concrete state transisions which are actually
measurable in the system by comparing the pre-state before execution with the
post-state after execution. Automations and tasks are configured by parameters
which steer the execution of the IaC logic at runtime.
Figure 1: Domain Model of IaC Automation Scripts
Description of Tables and Metrics
In our evaluation we performed systematic testing for idempotence and convergence,
based on a selected set of publicly available, community-maintained
Chef cookbooks. The full evaluation results
are listed under the tab "Evaluation Test Results".
For each Chef cookbook tested with our approach, the evaluation results include the
three types of tables below:
- List of test cases: This table reports on the configuration of each test case (in terms of executed and repeated tasks) and the identified test result (success true/false). Corresponding to the semantics of Chef cookbooks, a result of success=true means that the entire execution went fine, whereas success=false is reported as soon as any of the executed tasks (i.e., Chef resources) in the automation returned an error.
- List of automation tasks: This table lists all tasks (resources) defined in the cookbook, and provides details about their execution, including:
- the name and source information (line number and name of the file in which the task is defined),
- success ratio (percentage over all executions),
- list of convergent state properties. Each array entry contains the following values: (1) the name of the state property, (2) the value of the property, (3) the percentage of occurrence (1.0 means that the property value was observed in 100% of the post-states observed after executing this task), (4) the number of executions in which the value occurred as post-state, and (5) the total number of post-states observed for this task. (That is, (3) measures the ratio of (4) to (5).)
- whether the tests indicate that the task is idempotent or not.
- List of non-idempotent tasks: For each identified non-idempotent task, we report additional details (note that the rows in this table are structured hierarchically, hence there may be several subsequent rows belonging together):
- identifier of the task (linking to the respective table of automation tasks),
- name and source information,
- identifier of the test case in which the task was executed (linking to the test cases table),
- for each test case, the task execution numbers in the sequence of executed tasks (usually tasks are executed more than once within a test case),
- for each execution, the start time at which the task was executed, relative to the start of the entire test case,
- for each execution, the returned success result, and
- for each execution, the state changes effected in the system.