Learn how to use infrastructure as code to create services in AWS using Terraform.
Part one of a four-part series.
In this blog, I will describe how to build the core infrastructure in Amazon Web Services (AWS) to support our Continuous Integration platform.
We will refer to this as the “Landing Zone,” which will consist of components like VPCs, Subnets, Gateways and Routing Tables, Security and Identity Management, and more.
The platform architecture I’ll build to support the CI system is represented in the following diagram.
A Look Ahead
At the end of this series, the final platform will consist of a Jenkins cluster with a dedicated Master and multiple slaves configured in an autoscaling group, a GitLab code repository deployed in a HA configuration with external Elastic File System (EFS) volumes, a hosted Redis cluster, a multi-AZ Postgresql database, and an Elastic Container Registry to store Docker images.
The Jenkins Master and GitLab servers sit behind an application load balancer with public access, allowing developers to remotely manage the code base and manage the Jenkins pipelines.
The cluster of application servers will consist of an autoscaling group of Jenkins masters, Jenkins slaves and Gitlab servers deployed in private subnets behind an application load balancer. The autoscaling configuration will maintain one Jenkins master, two Jenkins Slaves, and two GitLab instances at any given time.
During deployment and initial configuration, only a single GitLab server will be deployed. Once GitLab has been configured for high availability, we will scale up the autoscaling group from one to two servers to provide redundancy.
For secure external access to the internet from instances within the private subnets, a NAT gateway will be deployed into a public subnet. The private instance will use the NAT gateway to get out to the internet for things such as package and patch installs.
Finally, a Linux bastion host will be deployed in a public subnet to allow access to EC2 instances in the private subnets from external users.
The entire platform will be scripted and stored as infrastructure as code so that it can be recovered quickly or deployed in another region in the event of a disaster.
Deployment Part 1
The deployment I’ll build in this initial part of the blog series will consist of the following services. This will provide the foundation for the service I’ll be adding later in this blog series.
- One Virtual Private Cloud (VPC) – A VPC is required. It enables you to launch AWS resources into a virtual network that you’ve defined. This virtual network closely resembles a traditional network that you’d operate in your own data center.
- Two public subnets – These subnets will be used to deploy any resource which will be accessed from the internet, such as our bastion host and Application Load Balancers. These subnets will also provide external internet access for our private resources via the NAT Gateway, which we will deploy here. The two subnets will be assigned to two separate availability zones, so we can deploy resources across each, providing for high availability. Note that Availability Zones (AZs) are physically isolated from one another.
- Two private application subnets – These two subnets, each assigned to unique AZs, will house our application servers. Servers in a private subnet (not attached to an Internet Gateway) will not be directly accessible from the internet and therefore are safe from external attacks. Access to these servers will only be available through a load balancer virtual IP address.
- Two private database subnets – These two subnets, each assigned to unique AZs, will house our database instances. Databases in a private subnet (not attached to an Internet Gateway) will not be directly accessible from the internet and therefore are safe from external attacks. Database access will be very secure, restricted via AWS Security Groups to only allow the GitLab server to access them.
- Internet Gateway (IG) – The internet gateway allows our bi-directional internet access to and from services within our VPC. Note that only resources in our “public” subnets will have access to an internet gateway.
- NAT Gateway (NAT) – Network Address Translation (NAT) gateway enable services in our private subnets to connect to the internet or other AWS services, but prevent the internet from initiating a connection with those instances. This is needed for package and patch installs on our private EC2 instances.
- Route Tables – These tables are a set of rules or routes that govern where traffic is routed. We attach a routing table to each subnet. Our public subnets will have a route to the Internet Gateway and our private subnets will have a route to the NAT Gateway.
To get started building this environment, you will need the AWS CLI and terraform installed locally on your workstation and create an IAM access key for an AWS user with administrator access.
We will not cover how to install these tools or get into details of terraform syntax. To learn more about terraform, see terraform.io. To learn more about AWS CLI, see AWS CLI. We will focus on a practical application of these tools only.
An IAM user can be setup from the AWS console and will require programmatic access in order to use CLI commands. It is recommended to first create a group with Administrative access credentials, then add a new user to this group. As you create the group, save the generated access key which will be required when configuring the AWS CLI.
See below for a few screenshots modeling the creation of an administrative terraform user with programmatic access:
As mentioned before, save your Access key ID and Secret access key. A downloadable csv file containing the credentials is also provided when creating the AWS user.
These keys will be used when initializing your AWS CLI for access to your AWS account.
In addition to the administrative account, you will need a second user created with access to Elastic Container Registry used to push images from Jenkins to the Elastic Container Registry (ECR).
Create another user using the same procedure as above, but do not add the user to a group.
Simple create a user with Programmatic access and attach two policies: AmazonEC2ContainerServiceFullAccess, AmazonEC2ContainerRegistryPowerUser. Use the User ID: “ecr-remote-pusher”. When complete download and save the Access Keys, which will later be used to create a Jenkins security credential.
Install Terraform, AWS CLI and GIT
Installation of the tools is beyond the scope of this series, but links to the software are provided. After installing the AWS CLI, GIT CLI and terraform CLI, complete the following tasks:
- Verify AWS CLI installation
- Verify terraform CLI installation
- Verify GIT CLI installation
- Configure an AWS CLI profile to be used with terraform to access the AWS account
- Validate command-line access to your AWS account
As a good practice, we would also recommend installing git command-line tools locally and have access to a code repository where you can store, protect and version control the scripts you will be developing.
Verify AWS CLI installation:
Verify terraform Installation:
Verify git Installation:
Configure AWS CLI profile:
Verify AWS CLI:
Once the command-line tools are installed and verified, we can start to build out the AWS infrastructure using terraform scripts.
We will begin by configuring the VPC, Subnets, Internet Gateway, NAT Gateway, and Route Tables. This will give us our landing zone, a foundation to begin to build out the applications, databases, security and load balancing infrastructure.
First, we must create a working directory where we will build our terraform scripts on your workstations. Once this is created and you are in that working directory, begin by creating the following terraform scripts to build out the base infrastructure in you AWS account.
Terraform the AWS Infrastructure
Here we are setting up a variety of variables that will be referenced in our other terraform scripts. You want to use variables for configurations that may either change from environment to environment or where you use the same value in multiple locations/scripts.
Important: Review this file and make changes appropriate to your environment. Ensure you change the profile value to the profile name you created with the command “aws configure –profile <name>” previously. You will need to generate an SSH Key on your local system using ssh-keygen, Putty or another tool.
This procedure is beyond the scope of this blog. If you need guidance on creating ssh keys on Windows or Linux, there are many resources available on the internet that will step you through this process.
variables.tf
vpc.tf
ig.tf
subnets.tf
nat_gw.tf
rt.tf
Here we are defining two route tables. One will be used to route to the internet gateway for the public subnets. The other route table will be used to route internet traffic to the NAT Gateway from the private subnets.
This will protect our private subnets from direct internet access, but allow egress to the internet for patches, software installs, and more.
Now that we have the terraform scripts prepared in a working directory, we can initialize terraform, which will evaluate your scripts and download any providers needed to execute the scripts.
Running terraform init, we can see that the “AWS” provider gets installed.
terraform init
The next step is to run a “terraform plan”. Terraform plan will validate the syntax prior to applying the configuration to your AWS account.
Note that it reads all .tf files and is set to create 17 new AWS resources based on the scripts in your working directory.
terraform plan
If the “plan” did not report any errors, you are set to apply the plan to your AWS account. Run the following command and then you can log into the AWS console and validate that all of the resources were successfully created.
Note that if any errors are reported, you can run “terraform destroy” to delete everything that was created, fix your scripts and again run “terraform apply”.
Also, here is a little trick to save some time… By adding -auto-approve option to your terraform commands, you do will not be prompted for confirmation (i.e. terraform apply -auto-approve).
terraform apply
A quick review of the subnets in our AWS console indicates that our first script was a success. We now have a VPC with 2 Private, 2 Public and 2 DB subnets.
All other resources were also created, which can be verified from various screens in the AWS console.
Final Thoughts
That was a quick introduction to using infrastructure as code to create services in Amazon Web Services (AWS). Here is what we accomplished so far:
- Created users and credentials with programmatic access to your AWS account
- Installed terraform, git and AWS CLI tools
- Created terraform scripts to build out our AWS Landing Zone
- Executed the terraform plan and terraform apply to validate and create AWS services
- Validated the services were built within our account
More to Come…
Bookmark this post and come back for part two in the series to learn how we will extend the platform with additional AWS services and build our Jenkins, GitLab servers.
Later, in parts three and four, I will show you how to use these tools to deploy containerized docker applications to Fargate.
Here is a sneak peek at the future state of the platform as we continue to build this out in the coming parts:
