I just spent an embarrassing amount of time trying to figure out why some resources in one of my puppet class where not being executed. All of these were inside of an if branch that was checking if another class was defined. It was defined (wouldn’t be much of a blog post if that wasn’t the case), there were not errors, it just wasn’t happening. Turns out that defined does not work the way I thought it did. From Puppet’s documentation (https://docs.puppetlabs.com/references/latest/function.html#defined):
Checking whether a given resource has been declared is, unfortunately, dependent on the parse order of the configuration
Because defined happens during the parse step and not another step the order in which your resources are declared matters. To give you a concrete example, I have two classes, php and imagemagick. The imagemagick class installs the imagick php extension for you if php is defined. How nice of it, right? This has always worked perfectly. Until my most recent manifest where I had something like:
before => Class['imagemagick'],
Which looks good. Php has to happen before imagemagick so it should be defined. But, because imagemagick is parsed before php, php isn’t actually defined for imagemagick, so nothing inside of my if was run during the apply. To make this actually work it needed to like like:
before => Class['imagemagick'],
And now it works. Ridiculous. Hope this saves someone some time.
Drops at 14:00 and 15:00.
For a long time puppet apply lines were added to server crontab files and we were content with the trade-off between timely updates and server resources used. More recently, while looking at some of our monitoring graphs, we noticed that one of our core business activities took a noise dive every hour, on the hour. When did puppet run? Every hour, on the hour. Time for a change.
My goal was the make so that there was a big red button to press whenever we wanted to run puppet, which is largely whenever we want to push code. Automatic deploys would be really nice but FDA regulations pretty much force something similar to waterfall. Anyway, Jenkins play button is close enough to the big red button so I went with that – also already had it installed. That left the following problems to overcome: running commands in production, getting around the DMZ, running puppet with sudo privileges, and doing it all in a timely manner. Turns out, this actually wasn’t terribly difficult. Here’s what I had to do.
- Set up Puppet. I’m assuming you’ve already got this part.
- Set up Jenkins. Again, assuming you’ve done this, but if not, it’s probably in your package manager.
- Bonus, secure Jenkins so random web traffic can’t access it. I used HAProxy rules to only forward the request to Jenkins if it originated inside of the office.
- Install the Publish Over SSH plugin. This will allow you to ssh into whatever server you already have punched through the DMZ. I’ll refer to this server as Smuggler here for short.
- Install pdsh on Smuggler.
- Run ssh-keygen on Smuggler if you haven’t already.
- In Jenkins, add Smuggler (Manage Jenkins -> Configure System) with it’s public key.
- Distribute Smuggler’s public key to every DMZed server you want to run puppet on. You can do this with ssh-copy-id or just copy the key into .ssh/authorized_keys on the servers you want to log into.
- On ever server you want Smuggler to log into, run visudo – sudo visudo is a weird sort of command – and change the following:
- Comment out “Defaults requiretty”. This makes it so you can run sudo from ssh in one command.
- Add a line with, “user ALL = (ALL) NOPASSWD: /usr/bin/puppet”. Replace user with the correct username. That gives that user the ability to run puppet as sudo without a password prompt.
- Create or edit a project in Jenkins and add a new ssh build step.
- Select your server to connect to.
- In the execute command part of the ssh step you want to set up a pdsh command. You’ll need two parameters for this,
- -R exec. This basically tells pdsh to execute a command for every server. If you don’t have any atypical ssh options, you could say -R ssh.
- -w <targets>. This is where you specify what to log into. You’ll need a comma separated list hosts or IPs. Thankfully you can use a range in the form of [01-16]. To ssh into IPs 192.168.0.1 to 192.168.0.5 and 192.168.0.100 to 192.168.0.115 you would say, “-x 192.168.0.[1-5],192.168.0.1[01-15]”.
- The last part of the pdsh command you need is the command to run. If you used exec you’ll end up with something like, “ssh %h sudo puppet apply myManifest.pp”, or “ssh %h sudo puppet agent –no-daemonize –onetime”, or something like that. The %h substitues the servers host into the command.
- All together, you end up with something in your execute field that looks something like:
- pdsh -R exec -w 192.168.0.[1-5],192.168.0.1[01-15] ssh %h sudo puppet apply myManifest.pp
With that in place, we are no longer wasting cycles having puppet accomplish nothing. As a bonus we can also modify puppet modules, stage code (we build rpms and use yum to deploy our code), or whatever else we need to do without fear of puppet sending it out before we’re ready. Double bonus, if we screw up we don’t have to wait for the next puppet run to deploy our fix.