The State of Target Mode in 2020

After my blog series on the agentless use of Chef dubbed “Target Mode” between August and October 2019, it is time to review what happened since.

Updated 14th October 2020

Update (14th October 2020)

With the release of Chef 16.6, improvements to enable Remote Ohai have been included into the Chef core. After talking with the Chef engineers, discussing my PoC and some adjustments for stability, the Chef Target Mode should now be usable by everyone.

That being said, while Ohai can collect remote system data, the support for remote capable resources has not changed. If you are fine with writing your own custom resources using the Train interface, you now have a completely new approach to managing remote systems though.

Target Mode in a Nutshell

Target Mode (abbreviated TM) made its first appearance in Chef 15.1 as an experimental and undocumented feature. The idea behind it is that not every machine will have (or even support) a Chef agent, but might still be desired to be in a certain state. To enable this, the Chef developers rely on the Train library, which is the transport backend for Chef’s popular InSpec product.

This approach makes it possible to have a pluggable backend infrastructure from standard SSH/WinRM to user-supported ones like Telnet, AWS SystemsManager, or even exotic ones like USB/Serial interfaces and REST APIs.

Relying on a line-based protocol, any existing Chef resource that just executes commands could be made “Target Mode Ready” by simply swapping local execution to remote execution. When TM debuted, all resources within Chef who used this pattern were made available. But it’s only a handful of them.

The Current State

After some time has passed, I got curious about what had changed since my initial adventures.

There has been no documentation upgrade, so how to even start with TM will remain a mystery to anyone who did not find third party blog posts. Just before this blog post, I checked if any of those initial code sections were touched since - only to find them unchanged in almost two years.

Is that a bad thing? Not necessarily. When I did some POC work on Target Mode in late 2019 I found it to be pretty straightforward and stable already. Following the path of just swapping the backend by some helper libraries and doing easy case distinctions, the code does not leave much room for error.

But one thing is problematic about this: if you look at the the code where remote system data gets collected you find it very static. For a system that is supposed to do an inventory run to determine, what the current state is and how it relates to desired, that is a big gap.

To iterate once again: There is no built-in remote discovery except for four OS-related bits of info. And no way to add any more.

The Dirty Way

I will not go into details here, because it is no solution for anything but exploration. But with some Ruby metaprogramming you can hook into that method from the outside and inject your own discovery data into that mock Ohai object. This enables you to work with remote data exactly in the same way as with a local system. But in reality, you should not need to have that knowledge. Things should just be there. Or even an interface to get them yourself.

How would Chef be able to solve this?

Extending Train Capabilities

The Train library already includes some minimalistic discovery functionality, as you can see in the get_ohai_data_remotely snippet of the Chef Client. As Train already has a plugin system for backends, this would be pretty straightforward: Add a new plugin type with platform filters (only run Cisco related filters on Cisco devices and such), cycle over matching plugins on connecting and pull it into the Chef Client as remote data.

This approach is the one I took in my PoC, where I had to do some black magic to modify the Chef client to do this. If you discount for the black magic, this is a 10 line modification of Chef and 1 additional plugin definition in Train.

Pros:

• lightweight
• authoring of plugins by users, for any platform
• limited modification of existing core products

Cons:

• every discovery needs to be written from scratch

Changing Ohai

While the Train extension is suitable for exotic platforms (read: non-Chef supported like industrial devices or networking gear), it misses out on a huge chance if the remote system is just standard Linux. As Chef already has a powerful on-system discovery tool with Ohai, this would be the second route: making Ohai work on remote systems.

Unknown to most, this approach was taken by Chef themselves under the nickname “OhBye”. It is hidden deep in the GitHub repositories, only to be pointed out if you ask Chef developers about remote discovery. Lifting the vail today, so you can look at OhBye yourself.

This takes a similar approach to the Chef Target Mode: adding an additional parameter to Ohai, adding a new second parameter to Ohai plugins (for the backend to use: local or remote), and then issuing the commands on there.

The code for OhBye was committed in March 2019, around the same time as Target Mode appeared. If you compare it to the current Ohai implementation, you can see it was never merged though. Looking at the code, this is clearly just a Proof of Concept as there are still commented-out calls to binding.pry in it.

By hooking into Ohai the way it did, it would leverage all platform detection features of Ohai and instantly enable things like Solaris support for example. With some more flexibility, there’s no reason against adding discovery for network devices and similar as it would just execute vendor-specific commands over some backend and parse the results.

Apart from being a dead-end at this point in time, the extensibility of this approach and instant functional gain is impressive. While the backend switching logic would have to be modified to dynamically use train plugins (which already return their URI scheme anyway), this is promising.

Pros:

• uses the whole existing discovery infrastructure on supported platforms
• could be enabled to support additional platforms

Cons:

• extensive change to a core product, just like Chef TM was

For my special cases, even enabling non-line-based backends like my train-rest plugin would be ideal. This would not only enable users to author Target Mode resources and get their state but also put it into external integration like the recently announced ServiceNow CMDB connector. For enterprise customers, that would close a gap in their compliance stance.

Summary

So should you use Target Mode in 2020? I think the answer is a clear “no” as it is still too experimental and the discovery issue need to be solved.

As soon as that is the case, partners can start authoring vendor-specific kits though. While Chef would need to update, merge, and support things like OhBye or the Train discovery plugin format, the community could go ahead and write resources specific to their customers.

This would open up a whole new marketplace where specialized partners could sell and support their resource packs. From our customers, I know that this would be a welcome addition to the Chef ecosystem.