Charles Engelke’s Blog

December 28, 2012

Provisioning a Server with CloudFormation

Filed under: Uncategorized — Charles Engelke @ 8:47 pm

In my first post on AWS CloudFormation I talked about how to create a machine instance with specific properties. That’s very useful. But what I really like about CloudFormation is how it lets me declaratively provision my new server with necessary software, content of my own, and even running services. I’m going to cover that in this post. But fair warning: there’s a trick needed to make it work. I don’t feel that it’s clearly documented by Amazon, and it took me a while to figure it out. I’ll cover that near the end of this post.

I said that CloudFormation Resources contain Type and Properties keys. But they can optionally have another key: Metadata. Any metadata object defined here can be retrieved using the CloudFormation API. A new instance can retrieve the metadata, and, if it’s the right kind of object, provision the instance according to that specification. The “right kind of metdata” object for this is an AWS::CloudFormation::Init resource. I think you can declare that as another resource and reference it by name in the metadata, but for now we will just put it directly in the metadata. We defined our new server resource last time as:

"NewServer": {
    "Type": "AWS::EC2::Instance",
    "Properties": {
        "ImageId": "ami-1624987f",
        "InstanceType": "t1.micro",
        "KeyName": "cloudformation"
    }
}

Now we can add the needed Metadata property:

"NewServer": {
    "Type": "AWS::EC2::Instance",
    "Properties": {
        "ImageId": "ami-1624987f",
        "InstanceType": "t1.micro",
        "KeyName": "cloudformation"
    },
    "Metadata": {
        "AWS::CloudFormation::Init": {
            provisioning stuff goes here
        }
    }
}

What kind of things can you specify in the configuration? The full documentation is here, but let’s complete an example. We will provision a web server with static content that’s stored in an S3 object. That’s pretty simple provisioning: install the Apache web server using the yum package manager, fetch my zipped up content from S3 and expand it in the right place, and start the httpd service. Here’s an AWS::CloudFormation::Init object that will do all that:

"AWS::CloudFormation::Init": {
    "config": {
        "packages": {
            "yum": {
                "httpd": []
            }
        },
        "sources": {
            "/var/www/html": "https://s3.amazonaws.com/engelke/public/webcontent.zip"
        },
        "services": {
            "sysvinit": {
                "httpd": {
                    "enabled": "true",
                    "ensureRunning": "true"
                }
            }
        }
    }
}

It’s pretty obvious what most of this does. The packages key lets you specify a variety of package managers, and which packages each one should install. We’re using the yum manager to install httpd, the Apache web server. The empty list as the value of the httpd key is how you specify that you want the latest available version to be installed. You can also use the apt package manager, or Python’s easy_install or Ruby’s rubygems package managers here. The sources key gives a URL (or local file name) for a zip or tgz file containing content to fetch and install. The key (/var/www/html here) is the directory to expand the fetched file to. Finally, the services key lists the services to run on boot. The ensureRunning key value of true specifies that the service should start on every boot. The only tricky part of the services key is that it has one value, always called sysvinit, and that key has the actual services as its children. Putting this all together gives the following template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Create and provision a web server",
    "Resources": {
        "NewServer": {
            "Type": "AWS::EC2::Instance",
            "Properties": {
                "ImageId": "ami-1624987f",
                "InstanceType": "t1.micro",
                "KeyName": "cloudformation"
            },
            "Metadata": {
                "AWS::CloudFormation::Init": {
                    "config": {
                        "packages": {
                            "yum": {
                                "httpd": []
                            }
                        },
                        "sources": {
                            "/var/www/html": "https://s3.amazonaws.com/engelke/public/webcontent.zip"
                        },
                        "services": {
                            "sysvinit": {
                                "httpd": {
                                    "enabled": "true",
                                    "ensureRunning": "true"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

I’ve made the zip file at that URL public, so you can copy this template and try to launch it yourself. Remember, you need to have created a key pair named cloudformation first. Did you try it? Did you notice that all this new stuff had no effect at all? The httpd package wasn’t installed, there is nothing at /var/www/html, and there’s no httpd service running. I had the hardest time figuring out what was wrong, but it turned out to be simple. The Amazon Linux AMI doesn’t do anything with this metadata automatically. You have to run a command as root to have it provision the instance according to the metadata:

/opt/aws/bin/cfn-init -s WebTest --region us-east-1 -r NewServer

The cfn-init utility is the program that understands the metadata and performs the steps it specifies, and the Linux AMI doesn’t run it automatically. If you log on to your new instance and run this command, though, it will do it all for you. You will have to replace WebTest in the command with whatever name you give the stack when you create it. If you’re running in a different region than us-east-1, change that part of the command, too. The -r NewServer option gives the name of the resource containing the metadata you want to use; we called that NewServer in the template above.

That’s nice, but not yet what we wanted. We want CloudFormation to handle the provisioning itself. To do that we have to get the new instance to run the cfn-init command for us when it first boots. And that’s what the UserData property of an instance lets us do. We can just put a simple shell script as the value of the UserData key to make that happen:

#!/bin/sh
/opt/aws/bin/cfn-init -s WebTest --region us-east-1 -r NewServer

Well, as you might guess, it’s not quite that simple. The value of the UserData key has to be a base 64 encoded string of this shell script. There’s a built-in CloudFormation function to base 64 encode a string, and we will use that:

UserData: {
    "Fn::Base64": "#!/bin/sh\n/opt/aws/bin/cfn-init -s WebTest --region us-east-1 -r NewServer\n"
    }

Note the \n characters to terminate each line. Put this in as a property of the server, giving the complete template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Create and provision a web server",
    "Resources": {
        "NewServer": {
            "Type": "AWS::EC2::Instance",
            "Properties": {
                "ImageId": "ami-1624987f",
                "InstanceType": "t1.micro",
                "KeyName": "cloudformation",
                "UserData": {
                    "Fn::Base64": "#!/bin/sh\n/opt/aws/bin/cfn-init -s WebTest --region us-east-1 -r NewServer\n"
                }
            },
            "Metadata": {
                "AWS::CloudFormation::Init": {
                    "config": {
                        "packages": {
                            "yum": {
                                "httpd": []
                            }
                        },
                        "sources": {
                            "/var/www/html": "https://s3.amazonaws.com/engelke/public/webcontent.zip"
                        },
                        "services": {
                            "sysvinit": {
                                "httpd": {
                                    "enabled": "true",
                                    "ensureRunning": "true"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

If you create a stack called WebTest with this template, you should get a new instance already running the Apache web server, with a couple of pages of content installed and already available. Give it a try. For me, at least, this was a success!

There are still a lot of rough edges. What if you don’t want to call your new stack WebTest? What if you want to run it in other regions? How about dealing with protected resources? Creating resources that interact with each other? Getting better reports of how to access resources that are created? Letting the user specify parameters to control the stack? I’ll cover some of that in future posts.

December 27, 2012

Windows Printing to an Airport Extreme Connected Printer

Filed under: Uncategorized — Charles Engelke @ 11:34 am

[Update: Got Windows 8 or RT? Mark Allibone has published an update on how to do this at his blog!]

Want to print from your Windows 7 PC to a USB printer connected to Apple’s Airport Express? Well, you can do what Apple says:

  1. Install Apple’s Bonjour for Windows
  2. Run the Bonjour Printing Wizard, answering its questions one by one
  3. Print!

And that works. At least it did for me. For some definition of “works”:

  • It showed the correct printer, but selected a driver for a different printer (that didn’t work at all)
  • It was easy to switch to the right driver, which worked
  • But it would only print black-and-white to my color laser printer
  • And would only print one job. Subsequent print jobs from the same or any other PC or Mac did nothing until you turned the printer off and back on.

Or, you could do what ended up working for me. The key points of my solution are:

  • Do not install any Apple software on your Windows PC
  • Do not pay any attention to anything Apple says regarding printing from your Windows PC

Instead, just use the regular Windows 7 Install Printer wizard. There are a lot of steps, but they’re easy.

  1. Select Devices and Printers from the Windows Start menu
  2. Click Add a Printer
  3. Select the Add a local printer option (yes, it’s not local, but that’s Microsoft for you)
  4. Click Create a new port, and select Standard TCP/IP Port from the drop-down list, and click Next
  5. Fill in the Hostname or IP address with the address of your Airport Extreme router. That’s probably 10.0.1.1, but you can check it by running the ipconfig command from a command prompt and looking for the Wireless LAN’s Default Gateway address. Leave Port name at whatever it fills in, uncheck Query the printer and automatically select the driver to use, and click Next.
  6. The wizard will say it’s Detecting the TCP/IP port. It should find the device. If not, you probably entered the wrong IP address. Check it and try again. If it still fails to detect it, don’t worry about it and continue anyway.
  7. Select Network Print Server (1 Port – USB) from the Standard Device Type list. The default Generic Network Card would probably work okay, but I didn’t try it. Click Next.
  8. Select your printer’s Manufacturer from the list, then select your specific printer from the Printers list, then click Next. If your printer isn’t there, you’ll have to download a driver and use the Have Disk… option.
  9. Fill in a Printer name, or leave the name it fills in for you alone. Click Next.
  10. Decide whether to share the printer or not. Since other devices on your network can print directly to the Airport Extreme, why bother to share it? I selected Do not share this printer and clicked Next.
  11. Decide whether to Set as the default printer, and try to Print a test page, then click Finish.

This worked for me on two different Windows 7 PCs. They now print in color, and jobs submitted after they print also print.

December 19, 2012

Learning about AWS CloudFormation

Filed under: Uncategorized — Charles Engelke @ 5:05 pm
Tags: , , ,

I’ve been using Amazon Web Services as the infrastructure for some products for a while now. A big advantage of running in the cloud is being able to automate creating, updating, and destroying servers. So far, we’ve been doing this by writing scripts. Now it’s time to move up to the next level of sophistication and use their CloudFormation service instead. That not only supports automated launching and provisioning an new servers, it supports automatically creating a whole bunch of interconnected services all at once. And it’s declarative, specifying where you want to end up, instead of procedural, specifying how to get there.

CloudFormation looks pretty simple at first, but I’ve found out that it really isn’t. You need to handle a lot of details, and the documentation isn’t always clear (to me), nor even always complete (as far as I can tell). And there aren’t enough complete examples. So, as I learn about it I’m going to blog about what I discover.

I’ll start with the simplest case I need handled: launching and provisioning a single server. I have to write a template specifying what I want CloudFormation to do, and then use CloudFormation to create the stack defined by that template.

CloudFormation templates are documented in the Working With Templates section of the AWS CloudFormation User Guide. Each template is a JSON document representing an object (basically, key/value pairs). The general format of that JSON object is:

{
   "key1": "value1",
   "key2": "value2",
   ...
   "keyN": "valueN"
}

The order of the key/value pairs is irrelevant. Another JSON document with the same pairs in a different order would be considered to represent the same object. Note that the keys are quoted strings, separated from the values with a colon. Key/value pairs are separated (not terminated) with commas. And values can be quoted strings, as shown here, or numbers, or JSON objects themselves. They can also be arrays, which are comma-separated lists of values enclosed in square brackets. Don’t worry too much about the details; we’ll see all of this in the examples.

CloudFormation template are JSON objects with some of the following keys:

  • AWSTemplateFormatVersion
  • Description
  • Parameters
  • Mappings
  • Resources
  • Outputs

All of these keys are optional except for Resources. Resources are the things that CloudFormation is going to create for you, so if there are none, there’s not much point to having a template anyway.

Although the AWSTemplateFormatVersion is optional, and there’s only ever been one version declared so far, I’m always going to include it. The only legal value for it is “2010-09-09″. The Description is also optional, but again, I’ll always include it to help me keep track of what I’m trying to do. So my template is going to start taking shape:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Create a basic Linux machine",
    "Resources": {
        something needs to go here!
    }
}

I need to fill in the Resources section with at least one key/value pair. The key is going to be my logical name for the resource. It can be just about anything (I haven’t pushed the limits though), because CloudFormation doesn’t care. I’ll just call this NewServer. The value is always an object with Type and Properties keys. The possible types are listed in the Template Reference section of the User Guide. To create an EC2 instance, use a Type of AWS::EC2::Instance.

The Properties object contains different possible keys for different resource types. The possible keys for AWS::EC2::Instance are listed in that section of the Template Reference in the User Guide. Only two keys are required: ImageId, which is the ID of the AMI to use for the new instance, and InstanceType, which tells what kind of instance to launch. Actually, in my experience I’ve found I can omit the InstanceType and it defaults to m1.small, but that may be a bug, not a real feature. The documentation says InstanceType is required, so I always include it.

I want to launch a standard 64-bit, EBS-Backed Amazon Linux instance in the US-East-1 region. According to the Amazon Linux AMI web page, the ImageId is ami-1624987f. I’ll save money by using a t1.micro instance. Putting all this together, I get the following template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Create a basic Linux machine",
    "Resources": {
"NewServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "ami-1624987f",
"InstanceType": "t1.micro"
} }
    }
}

Now to create this stack. Log in to the AWS Management Console and select CloudFormation. (If you’ve never used it before, you’ll be walked through a few sign-up steps to verify your identity. A few minutes later, you’ll be able to use the console.) It currently looks like this:

Image

I made sure I was in the right region (N.Virginia showing in the upper right corner), clicked Create New Stack, then filled in the blanks. I put my template in a file called cf.json, and selected it for upload:

Image

Then I clicked Continue. I had the option to enter some tags, which would be applied to the stack and to every resource it created. I just clicked Continue. Finally, I had a confirmation box:

Image

I clicked Continue, and my stack started building. I closed the acknowledgment window and looked at the console. The upper part showed all my stacks. There was only the one I just created. When a stack is selected, the bottom part shows its properties. I selected the Events tab for the screen capture below:

Image

Eventually, CloudFormation finishes, either successfully or with an error. In that latter case, it will usually roll back all the steps it took automatically. Otherwise you can click Delete Stack to get rid of everything it created.

In this case, everything worked. The Resources tab lists everything that was created. That’s just the NewServer resource, which is an AWS::EC2::Instance. It also shows me the ID of that instance. If I want to log in to that server I’ll have to look up its address in the EC2 section of the console. However, I’m not going to have much luck with that because I did not specify a key pair when creating the machine, so it’s impossible for anyone to connect via ssh.

Oops.

KeyName was an optional property I could have specified, but didn’t. The reason it’s optional is that you very well might want to create an instance nobody could log in to. That’s not true in our case, so I fixed it. First, I cleaned up the stack I created that I can’t use. I selected it in the console and clicked Delete Stack. The stack and every resource it created was destroyed. Next, I went back to the template and specified a KeyName value. It had to already exist as a Key Pair in the US-East-1 region. I happened to have one there named cloudformation, so I used it. The updated template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Create a basic Linux machine",
    "Resources": {
"NewServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "ami-1624987f",
"InstanceType": "t1.micro",
"KeyName": "cloudformation"
} }
    }
}

Repeating the steps above I got a running Linux machine. This time, that machine was associated with the cloudformation key pair, so I could  log in via ssh. Success!

Instead of the console, I could have used the cfn-create-stack command line tool. Or I could have written a program that invoked the REST API for CloudFormation. Each method looks about the same to AWS, and gets the same result.

But what’s the point? I could have created this instance directly with the EC2 console, or command line tools, or REST API. And it would have been at least as easy. Easier, in fact, in my opinion. That’s because I haven’t tapped into the real powers of CloudFormation yet:

  • Provision created servers with specified packages, files, software, etc.
  • Create (and manage) multiple resources that work together

I’ll get started on those more useful, and more interesting, things in my next post. But before I go, I’d better remember to go back to the CloudFormation console and delete the stack I created, so I don’t keep paying for that server.

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 52 other followers