Configure Device42 to be a Puppet External Node Classifier (E.N.C.)

Introduction: Understanding is Powerful

The modern tech ecosystem is comprised of a litany of tools, frameworks, solutions, and philosophies that seek to empower us in our work. Tasks that were substantial, confusing, or banal can be abstracted into autonomous clarity by using these tools. Puppet is one such powerful tool used by System Administrators, Technicians, and Developers alike to definitively configure and maintain their network architecture.

Puppet animates the configuration of each node on your network. To do this, Puppet needs to know some key information:

  • What each node is supposed to be.
  • What each node actually is.
  • Which operations workflow is needed to configure each node properly.

Being responsible for each device in your network is a big task, and Puppet offers a several strategies to tackle that role. The adage goes, “with great power comes great responsibility”. But let’s be frank; great responsibility requires great understanding. With power comes responsibility, and without understanding, power is impossible.

All the information one might need to understand a network can be found in a Configuration Management Database (CMDB) application like Device42. Conveniently, Device42 has an integration with Puppet that will fully map out a Puppet network, delivering a single point of truth about your network as configured by Puppet. You can learn more about the Device42 Puppet Integration here.

With the complete picture of our network, we can extend the power of these two tools by utilizing Device42 as an External Node Classifier (ENC) for Puppet. Before we do that, let’s understand why Puppet Classes are desirable and how Device42 can interact with them.

Configuring each device on your network on a node-by-node basis would be difficult to understand, irresponsible, and likely impossible at scale. It is far better to simplify the work of configuration by classifying each of your nodes with Puppet Classes. Classes are named units of Puppet code that perform configuration work when applied to a node. They ensure consistency and allow for code reuse across the network.

By default, classes are applied to nodes based on node definitions in the main site manifest (site.pp) on the Puppet master server. Instead of managing the classes of your nodes across text files on your Puppet master server, we want to use an external source to manage which classes are applied to which nodes. To do this, we will configure Device42 to be the External Node Classifier for Puppet.

External Node Classifiers are simple and powerful if you think of them in terms of their input and output. They accept the name of a node as their input and return the class hierarchy for that node as output. By using Device42 as our Puppet ENC, we can easily manage our Puppet classes from a single interface, bringing these two powerful technologies closer together.

So, let’s get started!

Part 0: Setup

Environment

This tutorial and integration was written using Puppet version 4.8 and Puppet Server version 2.7.2. The Python version used for the ENC script is 2.7.10.

The Device42 Puppet Integration is also used in this blog to map the Puppet agents to Device42 directly. In addition to this, we will use the ENC script I wrote while developing and researching this integration. It can be found here.

Device42 Custom Field

First, we will need to create a field on our Device42 instance where we can store node classifications. For this we will use a custom field that stores JSON data. Navigate to Tools>Custom Fields and click “Add Device Custom Field”.

Creating a new custom field to contain our ENC data.

You can name your custom field whatever you like, but it needs to store data as JSON. For our purposes, we’ll name the field “node_classes”. Let’s keep “Log for API” checked so that we can store a log of whenever a node’s class changes, which is already a feature not provided by vanilla Puppet.

Adding the new “node_classes” JSON field.

Once you save your new custom field, it should be available on all your devices.

As you’ve likely noticed, the “node_classes” field stores node class hierarchy in JSON as opposed to YAML which is the expected output format of an ENC for Puppet. This discrepancy will be handled by the ENC script implemented below.

At this point, if you’re already using the Puppet Device42 integration, then your Puppet agent nodes are available inside Device42. If you aren’t, you can either set up that integration, which is simple but beyond the scope of this blog, or manually add a device on D42 with a device name that matches the name of the node used by Puppet. The key to remember is the input/output of our ENC calls for the node name as input and the class hierarchy as output, so making sure the node name matches between Puppet and Device42 is important.

Note that the names of these two servers match between the Puppet master server and Device42.

Lastly, let’s put a placeholder in the node_classes field so that we can test against it in the next section. Edit one of your device’s node_classes field and give it the value of:

{
  "classes": {
    "example": { 
      "param": "example_param" 
    } 
  }, 
  "environment": "production" 
}

When it’s all set, your device page should look like this:

Editing the custom field “node_classes”.

Part 1: ENC Script

Puppet ENC scripts are simple piping between the Puppet master and an external node classification source, in our case Device42. I have created a GitHub repository that contains an implementation of this ENC script for our use here.

Clone the d42-puppet-enc repository onto your Puppet master and make sure to give it executable permissions (chmod +x {filename}). If you’re using a special Unix user/group for running Puppet, give ownership of d42-puppet-enc-integration to that user as well.

Our ENC script is a simple Python program that expects to be run from the command line, and can be tested from there during development.

First, we should ensure we have the requirements found in requirements.txt installed, or install them by running:

$ pip install -r requirements.txt

Next we need to fill in the relevant information in the settings.yaml file. That includes

  • IP address of your D42 appliance.
  • Username + Password for your D42 appliance for authentication.
  • The name of the custom field we created earlier that contains node classification on each device.
Simple settings.yaml example file.

Now that your settings.yaml file is filled out, lets do a quick overview of the main script file that does the real work of the ENC: d42_enc_fetcher.py.

The main ENC script d42_enc_fetcher.py does a few things, but its main task is to accept the node name input and return node classification output as valid YAML. The steps to accomplish this are:

  • 1: Parse input arguments containing the node name or any input flags. Done in main().
  • 2: Query Device42 through the https://{d42_host}/api/1.0/devices/name/{device_name}API endpoint to get that device’s data. Done in fetch_node_classification().
  • 3: Select the device’s custom field containing the ENC data as set in the settings.yaml file. Done in process_d42_node_enc() and top_level_classes_reducer().
  • 4: Return the ENC data as yaml to stdout. Done in the if __name__ == “__main__": check section.

We can test the ENC script from the command line like this:

$ d42_enc_fetcher.py {name_of_device} --verbose

This will return the placeholder ENC we set on Device42 during Part 0:

Everything looks good.

Part 2: Puppet Master

Now that we’ve verified that the ENC script works as expected, we can try plugging it into Puppet master, designating it as the ENC authority. This is a simple matter of editing your puppet.conf configuration file, which can be found in /etc/puppetlabs/puppet/puppet.conf.

As per the “Writing ENC” documentation straight from Puppet, all it takes is the change of a couple values in the [master] section of the puppet.conf file. Simply add these 2 configuration lines, pointing towards wherever you placed the d42_enc_fetcher.py file:

[master]
  node_terminus = exec
  external_nodes = /etc/puppetlabs/puppet/d42-puppet-enc/d42_enc_fetcher.py

Once changing that, restart your Puppet server and Puppet services on your master node like this:

$ service puppetserver restart 
$ service puppet restart

Try testing the new configuration on one of your nodes managed by Puppet. You will likely get a could not retrieve catalog from remote server error. This is because the ENC returned a class “example” which does not exist on our Puppet.

If a class you apply to a node in D42 does not exist, you will get this error.

We’ll fix this error in the next section where we show a fully working, albeit simple example of using Device42 as Puppet’s ENC.

Part 3: Proof of Concept

Puppet classes must exist before they can be used, obviously, and our ENC has returned a class “example” which our Puppet does not know about. Let’s create this class so that we can move forward.

Our example class will simply create a new file in our /tmp/ directory containing some text we set within Device42.

Depending on where your codedir for Puppet is, you’ll need to set up a new class module with the “example” class. My codedir is found at /etc/puppetlabs/code.

Structure of the /etc/puppetlabs/code directory

Your codedir may look different from this, and if so just follow along as how it would make sense to do given your configuration.

In our case, we have a directory environments which contains class configurations for each environment we manage with Puppet. Since our node’s environment is set to “production” within “node_classes” on Device42, we’ll create our class module in the environments/production/modules/ directory. Then we’ll create a file init.pp within the example class module which will define what the class is and what it needs to do.

$ mkdir -p /etc/puppetlabs/code/environments/production/modules/example/manifests 
$ touch /etc/puppetlabs/code/environments/production/modules/example/manifests/init.pp

Inside of init.pp let’s put the following contents:

# class example with params
class example ( $param = 'default') {
    file { '/tmp/example_txt':
        ensure  => 'present',
        replace => 'no',
        content => "Greetings from D42 Puppet ENC!!!\n  here is the param you asked for: $param",
        mode    => '0644',
    }
}

What this does is tell Puppet that a param named “$param” is expected to be found on a class named “example”, and it will have a default value of “default” if it is not found. Then, it creates a file at /tmp/example_txt with content that uses the $param parameter.

Now, any node you add the “example” class to inside Device42 will have this /tmp/example_txt file created on it. Let’s test it to make sure. Make sure the following class hierarchy is applied to your node on Device42 in the custom field we created initially:

{"classes": {"example": { "param": "example_param" } }, "environment": "production" }

And to test it, run this in your agent node’s terminal:

$ puppet agent --test 
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for puppet.device42.pvt
Info: Applying configuration version '1507564494'
Notice: Main site.pp running
Notice: /Stage[main]/Main/Notify[Main site.pp running]/message: defined 'message' as 'Main site.pp running'
Notice: /Stage[main]/Example/File[/tmp/example_txt]/ensure: defined content as '{md5}1c2b83659a6d4b419ac58444544ec18d'
Notice: Applied catalog in 0.16 seconds

As you can see from the Notice:/Stage[main/Example/File[/tmp/example_txt]/ensure line from the output of the last command, Puppet ensured that the file existed and put the specified content within it. Let’s check that to make sure…

/tmp/example_txt contains the data from the param of the “example” class, set inside of D42!

Success!

Conclusion:

As you can tell, this example is very basic and only means to demonstrate the most fundamental functionality of this ENC. But, with this baseline proof of concept behind us, we can begin managing our Puppet node classifications through Device42. Managing your node classes in a single place is a boon to simplicity and certainty about your network’s configuration.

If you have any questions, feedback, or want to share what you’ve done with your Device42 Puppet ENC, leave a comment below. I’d love to hear from you!