Building AWS EC2 instances with Puppet Enterprise

Building aws ec2 with puppet

Puppet Enterprise has a useful system called Cloud provisioner which will manage the creation, deletion and puppet classification of virtual machines or “instances”.  This cloud provisioner system supports three well known virtualization / cloud systems;  Amazon Web Services, Google Compute Engine and VMware.  This document is going to cover the building aws ec2 with puppet on Amazon Web Services.  I will cover the commands Puppet Enterprise provides and give an example script for you to use on your own systems.

How does it work?

Amazon Web services or AWS (I will refer to it as AWS from now on)  comes with an API (Application Programming Interface” which allows you to create EC2 instances “Programmatically”.  Puppet Enterprise utilizes this API and allows us to build instances, optionally classify them as a puppet group and very usefully install the puppet agent.  This means potentially you can have your instances being built and being managed by puppet in one step.


Setting up the prerequisites

Get Puppet Enterprise

You will need a puppet master server running Enterprise on once of your EC2 instances.  An m3.medium is powerful enough for testing and probably for some production setups  For this example I am using a Red Hat Enterprise 6.5 as my operating system – but you can do this very easily with Centos or Ubuntu et al.

This can be obtained here  which is a pain as you can’t just “wget” it to your server so if you do want to wget the install from the command line then use this link .


 Create a .Fog File

I am not sure what fog means and its an odd name for a file. Anyway suffice to say you need a .fog file to allow puppet to authenticate your connection to cloud systems in this case aws. It simply contains your API credentials.

So within your users home directory create a .fog file.  In my case the user I am using is the AWS supplied ec2-user, but you can use whichever account you want.

touch ~/.fog

Add the following entries to the empty .fog file remembering to replace it with your own AWS access codes

:default:
:aws_access_key_id: AKI4567NP4XATwrtcvfgWA
:aws_secret_access_key: kZ6G5KRU1TWZxlrdgN+xLMWjMO-SfCSS4N5izO

Note I just made up the above key so I hope that it doesn’t actually refer to someones real account 🙂


Create a Puppet Friendly AWS  EC2 Security Group

AW2 Ec2 Security Groups

Create a security group, mine is called PuppetHosts and has traffic / firewall rules as shown below. The rules shown will ensure that your newly create instance can talk to the puppet master and you can SSH. Naturally the rules can be tightened by specifing actual IP addresses rather than using the broad “Anywhere” as I have.

puppet-aws-security-group

Optionally Add a Puppet Group

If you are going to classify your nodes you need a group to add them to. So from your Puppet enterprise console please greate a group. I have added a one called my_aws_nodes
Puppet AWS


Building an AWS Instance

This is very easy.  The only tricky part is typing all of the command options on one line at the command prompt.  You can run it from a shell script to make this less cumbersome as I have done in the following example.

so create a file in your home directory called aws_build_instance.sh


touch ~/aws_build_instance.sh

edit it and enter the following text.

NOTE: You will need to make some customizations to your environment and these are all found in the section between #Parameters for building a host and #End Parameters


#!/bin/sh

#Parameters for building a host

#This is the SSH key you want to inject into your instance to allow you to login.
KEYNAME=KITSWEBKITCHENKEY_AWS

#Set the security group you wish this instance to belong to. This group should allow for puppet traffic 8140 and 61613
SECURITYGROUP=PuppetHosts

#Set the ami image you want e.g Redhat Ubuntu
AMI=ami-42718735

#Set the type / Size of Instance
INSTANCETYPE="m3.medium"

#Set the Region where this instance will reside

REGION="eu-west-1"

#Set the availability Zone

AVAILABILITY_ZONE="eu-west-1a"

#End Parameters

#Build The Options String to Create the Node
options="--region=eu-west-1"
options=$options" --availability-zone=eu-west-1a"
options=$options" --image=$AMI"
options=$options" --keyname=$KEYNAME"
options=$options" --type=$INSTANCETYPE"
options=$options" --security-group=$SECURITYGROUP"

echo $options
puppet node_aws create $options

save the file and run it

./aws_build_instance.sh

if all goes to plan you should see output like so:


Notice: Creating new instance ...
Notice: Creating new instance ... Done
Notice: Creating tags for instance ...
Notice: Creating tags for instance ... Done
Notice: Launching server i-02efd430 ...
#################
Notice: Server i-02efd430 is now launched
Notice: Server i-02efd430 public dns name: ec2-23-125-44-97.eu-west-1.compute.amazonaws.com
ec2-23-125-44-97.eu-west-1.compute.amazonaws.com


Nice!

Optionally Puppet Classify your Instance

If you are going to manage your instance with Puppet its a good idea to add it to a group so the customisation and classes can be added. This is pretty easy.

So create a file called


touch ~/aws_classify_instance.sh

edit it and enter the following text ( You will need to alter it to match your own environment. In particular the AWSINSTANCE parameter as this names the server you wish to classify)


#!/bin/sh
#Parameters for Classifying a host

#Enter the dns/host name of the instance you just built
AWSINSTANCE=ec2-23-125-44-97.eu-west-1.compute.amazonaws.com
PUPPETMASTER_CLOGIN=qets.puppetAdmin@quindell.com #Puppet Console U/Name
PUPPETMASTER_CPASSWD=Sunsh1ne #Puppet Console P/Word
PUPPET_MASTER=ec2-54-75-253-106.eu-west-1.compute.amazonaws.com
ENC_SERVER=localhost

#Enter the NODE group of the server you are

options='--insecure'
options=$options' --node-group=$NODEGROUP'
options=$options' --enc-server=$ENC_SERVER'
options=$options' --enc-port=443'
options=$options' --enc-auth-user=$PUPPETMASTER_CLOGIN'
options=$options' --enc-auth-passwd=$PUPPETMASTER_CPASSWD'
options=$options' $AWSINTANCE'

echo $options
puppet node classify $options

Now run it – if all is good you will get a simple response like so

Notice: Contacting https://localhost:443/ to classify ec2-23-125-44-97.eu-west-1.compute.amazonaws.com
complete

That is that your instance is now classified.  When you connect it  to puppet all you software will be delivered and installed.

Install the Puppet Agent

We can script and push the puppet install.  And this section will cover this. We need two files one to run the installer and a second

Create a file called aws_install_pupp.sh

touch ~/aws_install_puppet.sh

Now edit the file


#!/bin/sh
#The puppet agent cert name
PUPPET_AGENT_CERT=ec2-23-125-44-97.eu-west-1.compute.amazonaws.com

#The installer answer file which will be uploaded
PUPPET_ANSWER_FILE=agent_install_answers.txt

#The login of the install (root for some like Centos or ec2-user if it is an official Amazon AMI)
SSH_LOGIN=root

#The path to your ssh key that was injected in the instance (for authentication)
SSH_KEY_PATH=/home/ec2-user/.ssh/kitswebkitchen_aws.pem

#Puppet Installation Path (where the install of puppet lives )
PUPPET_INSTALL_PLOAD=/home/ec2-user/binaries/puppet-enterprise-3.3.1-el-6-x86_64.tar.gz

#Puppet Server
PUPPET_SERVER=ec2-54-75-253-106.eu-west-1.compute.amazonaws.com

options="--install-script=puppet-enterprise"
options=$options" --login=$SSH_LOGIN"
options=$options" --puppetagent-certname=$PUPPET_AGENT_CERT"
options=$options" --server=$PUPPET_SERVER"
options=$options" --keyfile=$SSH_KEY_PATH"
options=$options" --installer-answers=$PUPPET_ANSWER_FILE"
options=$options" --installer-payload=$PUPPET_INSTALL_PLOAD"
options=$options" $PUPPET_AGENT_CERT"

echo $options
/usr/local/bin/puppet node install --debug $options

Now create a second file which will be the answer file which will automate the agent install. Edit the file and add the following answers. This will ensure the installer runs silently on the box and doesn’t fail.


q_all_in_one_install=n
q_puppet_cloud_install=y
q_vendor_packages_install=y
q_database_install=n
q_puppetdb_install=n
q_puppet_cloud_install=n
q_puppet_enterpriseconsole_install=n
q_puppetmaster_install=n
q_puppetagent_server=ec2-54-75-253-106.eu-west-1.compute.amazonaws.com
q_puppetagent_install=y
q_puppetagent_certname=`curl http://169.254.169.254/latest/meta-data/hostname`
q_install=y

Save the file and then run the installer we created earlier

./aws_install_puppet.sh

This will kick off the puppet installer agent.

Comments are closed.