For a long time puppet apply lines were added to server crontab files and we were content with the trade-off between timely updates and server resources used. More recently, while looking at some of our monitoring graphs, we noticed that one of our core business activities took a noise dive every hour, on the hour. When did puppet run? Every hour, on the hour. Time for a change.
My goal was the make so that there was a big red button to press whenever we wanted to run puppet, which is largely whenever we want to push code. Automatic deploys would be really nice but FDA regulations pretty much force something similar to waterfall. Anyway, Jenkins play button is close enough to the big red button so I went with that – also already had it installed. That left the following problems to overcome: running commands in production, getting around the DMZ, running puppet with sudo privileges, and doing it all in a timely manner. Turns out, this actually wasn’t terribly difficult. Here’s what I had to do.
- Set up Puppet. I’m assuming you’ve already got this part.
- Set up Jenkins. Again, assuming you’ve done this, but if not, it’s probably in your package manager.
- Bonus, secure Jenkins so random web traffic can’t access it. I used HAProxy rules to only forward the request to Jenkins if it originated inside of the office.
- Install the Publish Over SSH plugin. This will allow you to ssh into whatever server you already have punched through the DMZ. I’ll refer to this server as Smuggler here for short.
- Install pdsh on Smuggler.
- Run ssh-keygen on Smuggler if you haven’t already.
- In Jenkins, add Smuggler (Manage Jenkins -> Configure System) with it’s public key.
- Distribute Smuggler’s public key to every DMZed server you want to run puppet on. You can do this with ssh-copy-id or just copy the key into .ssh/authorized_keys on the servers you want to log into.
- On ever server you want Smuggler to log into, run visudo – sudo visudo is a weird sort of command – and change the following:
- Comment out “Defaults requiretty”. This makes it so you can run sudo from ssh in one command.
- Add a line with, “user ALL = (ALL) NOPASSWD: /usr/bin/puppet”. Replace user with the correct username. That gives that user the ability to run puppet as sudo without a password prompt.
- Create or edit a project in Jenkins and add a new ssh build step.
- Select your server to connect to.
- In the execute command part of the ssh step you want to set up a pdsh command. You’ll need two parameters for this,
- -R exec. This basically tells pdsh to execute a command for every server. If you don’t have any atypical ssh options, you could say -R ssh.
- -w <targets>. This is where you specify what to log into. You’ll need a comma separated list hosts or IPs. Thankfully you can use a range in the form of [01-16]. To ssh into IPs 192.168.0.1 to 192.168.0.5 and 192.168.0.100 to 192.168.0.115 you would say, “-x 192.168.0.[1-5],192.168.0.1[01-15]”.
- The last part of the pdsh command you need is the command to run. If you used exec you’ll end up with something like, “ssh %h sudo puppet apply myManifest.pp”, or “ssh %h sudo puppet agent –no-daemonize –onetime”, or something like that. The %h substitues the servers host into the command.
- All together, you end up with something in your execute field that looks something like:
- pdsh -R exec -w 192.168.0.[1-5],192.168.0.1[01-15] ssh %h sudo puppet apply myManifest.pp
With that in place, we are no longer wasting cycles having puppet accomplish nothing. As a bonus we can also modify puppet modules, stage code (we build rpms and use yum to deploy our code), or whatever else we need to do without fear of puppet sending it out before we’re ready. Double bonus, if we screw up we don’t have to wait for the next puppet run to deploy our fix.