Testing on Physical Machines with kitchen-static

This article shows how to work with Test Kitchen on physical machines using the kitchen-static Driver. If you need to deliver a product (bundle of server and software) instead of just configuration, some tasks cannot be run on virtual machines alone but need testing on actual hardware.

The need for testing on physical machines comes from a current project, where physical servers are part of a large industrial solution and are provisioned with Chef.

As with all projects, one should keep the Dev/Prod Parity Principle in mind, which states that development and production environment should be as similar as possible. For example, physical systems include components which are not available in your usual VMware- or AWS-based environment: Installation of vendor specific drivers, configuration of hardware components (ILO, BMC, …) or even setting certain types of Windows Network Teaming.

kitchen-static

Over a year ago, this kitchen driver was created to allow our first steps on HP Gen9 servers. It was easy enough, as Test Kitchen already comes with the proxy driver. Upon checking its source code it became clear that this driver was outdated and poorly maintained. As it was obvious that we are dealing with a total niche case here, the decision to maintain a custom driver was easy.

Installation of the driver is very straightforward as it has no external dependencies and is available on RubyGems:

gem install kitchen-static

Now, just change your kitchen.yml accordingly. I prefer having my platform specifics in a separate file kitchen.static.yml:

driver:
  name: static
  hostname: 198.51.100.14

If you have a machine at the given IP, your usual kitchen converge and kitchen verify commands work as intended.

Setup and Tear Down

As we are working with a physical machine which does not just vanish or pop into existence, we have to take care of not getting the machine messed up. After all, kitchen create and kitchen destroy just change a value in your local .kitchen directory and do not interact with the actual machine at all.

One decentralized solution are Kitchen Lifecycle Hooks. They have been added in Kitchen 1.23 and allow running local or remote commands on certain stages of Kitchen execution. We can use them to do our setup and tear down actions:

driver:
  name: static
  hostname: 198.51.100.14
  
lifecycle:
  post_create:
    - remote: setup.ps1
      environment:
        SOME_PARAMETER: value
  pre_destroy:
    - remote: Start-Process -FilePath C:\Windows\System32\Sysprep\Sysprep.exe -ArgumentList "/generalize /oobe /shutdown /quiet"

The logical execution steps are:

• kitchen create: Add the IP address to the suite state file in .kitchen only
• right after create: Run a script setup.ps1 which is present on the remote machine
• … develop, test, retry …
• right before destroy. Remotely trigger a SysPrep (reset to factory defaults for Windows
• kitchen destroy: Remove the IP address from the state file

Of course, SysPrep takes time so you will now sit there and bite your nails for some minutes until you can retest.

Alternative for Restoring Baseline Configuration

While I have not benchmarked it, I think using System Restore Points instead of a full SysPrep might be quicker for Windows systems. These work using Restore-Computer -RestorePoint X in Powershell, where X is the ID of a Restore point you set up on initial deployment of the machine.

Linux-based Systems

For Ubuntu/Debian based systems there is no centralized, standard procedure to reset them to default as far as I know. You can try the tool Resetter here, which also has a command line interface.

Centralized Management of Machines

As soon as you have a bigger team involved in writing components for physical machines or have automated testing pipelines, the 1:1 approach of this post does not scale any more.

The logical solution is to have a central pool of machines and a mechanism to centrally hand out test machines, do queueing and standardized restore procedures. This functionality was introduced in version 0.10.0 of kitchen-static and will be the topic of another blog post.