Journey to Kubernetes - Part 2 of 7 - Infrastructure
Welcome back to the second part of the journey to the kubernetes cluster. In this article I'll show you which cloud hosting I picked to build the cluster on and how I did it.
> Archived article from 2018, from the old nutellino.it: commands and versions may be out of date.
> Machine translation from Italian: the original is here.
I want to build it automatically!
Welcome back to the second part of the journey to the kubernetes cluster. In this article I'll show you which cloud hosting I picked to build the cluster on and how I did it.
You'll find all the code for this guide in part 2 of this repository: https://github.com/nutellinoit/journey-to-kubernetes
The tools I used are:
- Terraform
- Route 53
- Some bits of the https://github.com/hobby-kube project
Let's start with the machine layout. On Hetzner cloud a VM with 2 virtual cores, 4Gb of ram and 40Gb of disk costs exactly:
[Hetzner Price: image not recovered]
5.98 euros, they're practically giving it away. I took exactly 10 of these machines.
I split them up logically like this:
- 1 Nginx server doing the reverse proxy to the services exposed by Kubernetes via
NodePort. This Nginx server will also have acme.sh installed for issuing SSL certificates, and Rancher to get an overview of the cluster we'll install later. - 8 Kubernetes servers, 3 masters and 5 workers
- 1 HaProxy server, to do the reverse proxy and balancing of the mail server ports
The first part of the job is creating the virtual machines and setting up the internal VPN for private traffic.
Why the internal VPN?
Hetzner cloud is cheap, but it only gives you a public ip and no way to have private ips between the machines. Because of this, all the cluster's internal traffic is easy to sniff. To get around the problem we use Wireguard, a secure, encrypted internal mesh network. (https://www.wireguard.com/)
First of all, to work via API on hetzner cloud, you need to go into your project and create the API key to use:
[Hetzner screenshot: image not recovered]
Let's give our key a description and save it somewhere safe.
Let's also remember to add our public ssh key in the panel, so that Hetzner doesn't create a root password but adds our key to the servers for logging in.
In my case it's in
cat .ssh/id_rsa.pub
[image: image not recovered]
This key, and above all the name it gets, is what we'll need to use in terraform to add the key to the servers automatically.
With these steps done, we're ready to create our terraform script.
Let's start with the variables file:
/* SERVER NUMBERS */
variable "hosts_kube" {
default = 8
}
variable "hosts_nginxrancher" {
default = 1
}
variable "hosts_haproxy" {
default = 1
}
/* HOSTNAMES FORMAT */
variable "hostname_format_kube" {
default= "kube-staging-%d"
type = "string"
}
variable "hostname_format_nginxrancher" {
default= "nginx-rancher-staging-%d"
type = "string"
}
variable "hostname_format_haproxy" {
default= "haproxy-staging-%d"
type = "string"
}
## VM type
variable "hcloud_type" {
default = "cx21"
type = "string"
}
variable "hcloud_type_nginxrancher" {
default = "cx21-ceph"
type = "string"
}
variable "hcloud_type_haproxy" {
default = "cx21-ceph"
type = "string"
}
variable "image" {
type = "string"
default = "ubuntu-16.04"
}
/* HCLOUD SETTINGS */
variable "hcloud_token" {
default = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
variable "hcloud_ssh_keys" {
type = "list"
default = ["name@example.com"]
}
variable "hcloud_datacenter" {
type = "list"
default = ["fsn1-dc8", "fsn1-dc14"]
}So we define the number of servers, the type, the hostname format, the server image we'll use and the datacenter location details. Note that hcloud_datacenter is a list of two zones of the same datacenter, so we get more availability in case one of the two has a failure.
Let's also remember to put our Hetzner API key in the hcloud_token variable
After that we define the variables for AWS route 53
/* AWS DNS SETTING AND KEYS */
variable "region" {
default = "us-east-1"
}
variable "ttl" {
default = 300
}
variable "zone_id" {
default = "XYZZZZZZZZZZ"
}To use terraform with our AWS account we first need to run the awscli tool from the command line and configure access:
aws configureWith these variables we're saying to work on the dns zone XYZZZZZZZZZZ; you'll find this value inside amazon route 53 in the hosted zones info. Hosted Zone ID
[image: image not recovered]
We also define which packages to install on the nodes during provisioning
/* PACKAGES INSTALLED DURING PROVISIONING */
// HAPROXY
variable "apt_packages_haproxy" {
type = "list"
default = ["ansible","python"]
}
// NGINX REVERSE
variable "apt_packages_nginxrancher" {
type = "list"
default = ["nginx","git-core","apache2-utils"]
}
// KUBE NODES
variable "apt_packages_kubes" {
type = "list"
default = ["ansible","python","open-iscsi","software-properties-common"]
}Main Script
Now let's move on to the main definition of our terraform script.
In this phase we'll use the variables defined so far to handle the deploy
Let's start by initializing the hcloud provider with our Hetzner API key
/*INIT PROVIDER HCLOUD*/
provider "hcloud" {
token = "${var.hcloud_token}"
}Next we define the creation of all 10 nodes that will then be used for the reverse proxy, the kubernetes nodes and the haproxy server
/*KUBERNETES NODES PREP*/
resource "hcloud_server" "host_kube" {
name = "${format(var.hostname_format_kube, count.index + 1)}"
datacenter = "${element(var.hcloud_datacenter,count.index)}"
image = "${var.image}"
server_type = "${var.hcloud_type}"
ssh_keys = ["${var.hcloud_ssh_keys}"]
count = "${var.hosts_kube}"
provisioner "remote-exec" {
inline = [
"while fuser /var/lib/apt/lists/lock >/dev/null 2>&1; do sleep 1; done",
"apt-get update",
"apt-add-repository -y ppa:ansible/ansible",
"apt-get install -yq ufw ${join(" ", var.apt_packages_kubes)}",
"curl https://releases.rancher.com/install-docker/17.03.2.sh | sh",
"apt-get -y install linux-image-extra-$(uname -r)",
"docker run --name enable_lio --privileged --rm --cap-add=SYS_ADMIN -v /lib/modules:/lib/modules -v /sys:/sys:rshared storageos/init:0.1"
]
connection {
type = "ssh"
user = "root"
private_key = "${file("~/.ssh/id_rsa")}"
}
}
}
/*HAPROXY SERVER INSTALL*/
resource "hcloud_server" "host_haproxy" {
name = "${format(var.hostname_format_haproxy, count.index + 1)}"
#location = "${var.hcloud_location}"
datacenter = "${element(var.hcloud_datacenter,count.index)}"
image = "${var.image}"
server_type = "${var.hcloud_type_haproxy}"
ssh_keys = ["${var.hcloud_ssh_keys}"]
count = "${var.hosts_haproxy}"
provisioner "remote-exec" {
inline = [
"while fuser /var/lib/apt/lists/lock >/dev/null 2>&1; do sleep 1; done",
"apt-get update",
"apt-get install -yq ufw ${join(" ", var.apt_packages_haproxy)}"
]
connection {
type = "ssh"
user = "root"
private_key = "${file("~/.ssh/id_rsa")}"
}
}
}
/*NGINX REVERSE PROXY INSTALL*/
resource "hcloud_server" "host_nginxrancher" {
name = "${format(var.hostname_format_nginxrancher, count.index + 1)}"
#location = "${var.hcloud_location}"
datacenter = "${element(var.hcloud_datacenter,count.index)}"
image = "${var.image}"
server_type = "${var.hcloud_type_nginxrancher}"
ssh_keys = ["${var.hcloud_ssh_keys}"]
count = "${var.hosts_nginxrancher}"
provisioner "remote-exec" {
inline = [
"while fuser /var/lib/apt/lists/lock >/dev/null 2>&1; do sleep 1; done",
"apt-get update",
"apt-get install -yq ufw ${join(" ", var.apt_packages_nginxrancher)}",
"curl https://releases.rancher.com/install-docker/17.03.2.sh | sh",
"curl https://get.acme.sh | sh",
"git clone https://github.com/nutellinoit/lets-ssl-wizard-nginx.git"
]
connection {
type = "ssh"
user = "root"
private_key = "${file("~/.ssh/id_rsa")}"
}
}
}This last section defines all the packages to install via remote-exec, and all the information about the Hetzner nodes passed in through variables.
Finally, based on the machines created earlier, we define the dns records for these virtual machines
resource "aws_route53_record" "nginx-rancher-staging" {
zone_id = "${var.zone_id}"
name = "nginx-rancher-staging"
type = "A"
ttl = "${var.ttl}"
records = ["${hcloud_server.host_nginxrancher.*.ipv4_address}"]
}
resource "aws_route53_record" "haproxy-staging" {
zone_id = "${var.zone_id}"
name = "haproxy-staging"
type = "A"
ttl = "${var.ttl}"
records = ["${hcloud_server.host_haproxy.*.ipv4_address}"]
}
resource "aws_route53_record" "kube-stage" {
// same number of records as instances
count = "${var.hosts_kube}"
zone_id = "${var.zone_id}"
name = "kube-staging-${count.index+1}"
type = "A"
ttl = "${var.ttl}"
// matches up record N to instance N
records = ["${element(hcloud_server.host_kube.*.ipv4_address, count.index)}"]
}This way we're using the ipv4_address output from creating the vms with the hcloud provider in the previous steps to create all the matching DNS records via aws route53.
Finally, we use the wireguard terraform module to install the VPN automatically on our 10 freshly created nodes
/*VPN MODULE*/
module "wireguard" {
source = "./wireguard"
count = "${var.hosts_nginxrancher+var.hosts_kube+var.hosts_haproxy}"
connections = "${concat(hcloud_server.host_nginxrancher.*.ipv4_address,hcloud_server.host_kube.*.ipv4_address,hcloud_server.host_haproxy.*.ipv4_address)}"
private_ips = "${concat(hcloud_server.host_nginxrancher.*.ipv4_address,hcloud_server.host_kube.*.ipv4_address,hcloud_server.host_haproxy.*.ipv4_address)}"
hostnames = "${concat(hcloud_server.host_nginxrancher.*.name,hcloud_server.host_kube.*.name,hcloud_server.host_haproxy.*.name)}"
}This way we invoke the module in the wireguard subfolder, passing it the outputs of the other definitions as input variables. Note how all the servers' ipv4 addresses have been concatenated so the ips have a defined order.
A CLOSER LOOK AT THE VPN
In this series of guides we'll see how to install kubernetes using Kubespray. With Kubespray we can define a few variables, including the network CIDR of the plugin used for intra-cluster communication (in our case it will be flannel).
To make all the network traffic go through the VPN, we also need to define the Kubernetes network CIDR in our wireguard module.
The two parameters are in the wireguard/main.tf file and they are:
- overlay_cidr defines the overlay network that Kubernetes will create
- vpn_iprange are the ips that will be used to create the VPN interfaces
variable "overlay_cidr" {
type = "string"
default = "10.233.0.0/16"
}
variable "vpn_iprange" {
default = "10.0.1.0/24"
}The script in action!
Now let's try running everything we've prepared; at the end we should have our nodes ready and properly configured, with the right dns associated and the vpn network working.
[asciicast: image not recovered]
And with that we're ready for the next steps!
See you next time!
$ cat ~/archivio/journey-to-kubernetes-parte-2-di-7-infrastruttura.md
WARNING: this file is from 2018.
I wrote it, but a me from a few years ago: it may no longer reflect what I think or what I can do.
NOTE: machine translation from Italian, the original is here.
Continue? [y/N]