Testing Idempotence and Convergence for Infrastructure as Code
(Accompanying Material and Test Results)

Waldemar Hummer, Florian Rosenberg, Fábio Oliveira, Tamar Eilam

Contact: http://dsg.tuwien.ac.at/staff/hummer/

This Web page contains accompanying material for the paper 'Testing Idempotence and Convergence for Infrastructure as Code'. The data listings on this page provide in-depth insights into the testing approach, contain detailed reports of the test results, and list some identified bugs.

System Model

To briefly introduce the problem domain, we refer to the abstracted system model depicted in Figure 1. The core entity is denoted automation, which represents the logic of a piece of Infrastructure as Code (IaC, see here for a brief overview). An automation consists of multiple automation tasks, which are implemented by some script code and are supposed to cause potential state changes. From the runtime perspective, an automation run represents the entire execution of an automation, which itself consists of multiple task executions. Each task execution effects a set of concrete state transisions which are actually measurable in the system by comparing the pre-state before execution with the post-state after execution. Automations and tasks are configured by parameters which steer the execution of the IaC logic at runtime.

Domain Model of IaC Automation Scripts
Figure 1: Domain Model of IaC Automation Scripts

Description of Tables and Metrics

In our evaluation we performed systematic testing for idempotence and convergence, based on a selected set of publicly available, community-maintained Chef cookbooks. The full evaluation results are listed under the tab "Evaluation Test Results". For each Chef cookbook tested with our approach, the evaluation results include the three types of tables below:
  1. List of test cases: This table reports on the configuration of each test case (in terms of executed and repeated tasks) and the identified test result (success true/false). Corresponding to the semantics of Chef cookbooks, a result of success=true means that the entire execution went fine, whereas success=false is reported as soon as any of the executed tasks (i.e., Chef resources) in the automation returned an error.
  2. List of automation tasks: This table lists all tasks (resources) defined in the cookbook, and provides details about their execution, including:
  3. List of non-idempotent tasks: For each identified non-idempotent task, we report additional details (note that the rows in this table are structured hierarchically, hence there may be several subsequent rows belonging together):