Cloud-init

Figure 1100. Introduction and reference Slide presentation

Figure 1101. In a nutshell Slide presentation
  • Initial system configuration

  • Standardized installation related tasks:

    packages, users, ssh keys, networks, hostnames, custom scripts


Figure 1102. Post server creation steps Slide presentation
  • Individual CRUD file operations

  • Adding users

  • Supplying ssh user and host keys.

  • System hardening: Firewall rules, security policies, logging / monitoring

  • Installing packages

  • Enabling and starting services

  • System upgrade + reboot

  • Arbitrary command execution


Figure 1103. DIY using Bash by Terraform's user_data attribute Slide presentation
# main.tf

resource "hcloud_server" "nginxServer" {           
  name         = "hello"
  image        = "debian-13"
  server_type  = "cx23"
  ssh_keys     = ...

  # Post server creation init script
  user_data    = file("myinit.sh")
}
#!/bin/bash

# myinit.sh

aptitude update && aptitude -y upgrade     
├── main.tf
├── ... 
└── myinit.sh

exercise No. 18

Automatic Nginx installation

Q:

Follow Figure 1103, “DIY using Bash by Terraform's user_data attribute ” and create a bash script which:

  • Installs the Nginx web server by its nginx package.

    Follow How to Install Nginx on Debian 13 installing and configuring the ufw firewall alongside as well. You'll thus do not need a Hetzner firewall. Do not forget to enable ssh access.

  • Uses aptitude for updating the package list and updating all installed packages.

Tip

  • Automated installation clashes with interactive user input: Use the aptitude install --assume-yes option allowing for automated package installation and upgrades.

  • There is a similar option when enabling your ufw firewall.

  • Execute your initialization script beforehand line by line on an existing machine interactively. You'll find errors more quickly thereby avoiding time consuming terraform apply cycles

Figure 1104. Basic Cloud-init facts Slide presentation
  • Server post-creation install steps

  • YAML based syntax

#cloud-config
package_reboot_if_required: true
package_update: true
package_upgrade: true

packages:
  - nginx

runcmd:
  - systemctl enable nginx
  - systemctl start nginx
  - rm -rf /var/www/html/*
  - >
    echo "I'm Nginx @ $(curl -4 https://ifconfig.me) 
    created $(date -u)" > /var/www/html/index.html

Figure 1105. cloud-init file validation Slide presentation

exercise No. 19

Cloud-init for starters

Q:

Follow Hetzner's Cloud-init introduction and assemble a corresponding cloud-init script. Use the package_reboot_if_required: true directive in favor of an explicit reboot command.

Use the Hetzner WEB Gui for manual server creation. Copy-paste your cloud-init script into the respective text field.

Tip

Do a syntax validation of your Cloud-init script beforehand. For this purpose install cloud-init on your local machine. Beware: Executing cloud-init requires root privileges, sudo may be your friend here.

Without prior validation you'll likely end up in tedious debugging struggles.

Figure 1106. Terraform triggering Cloud-init Slide presentation
# main.tf
resource "hcloud_server" "web" {
  name         = var.server_name
       ...
  user_data = file("userData.yml")
}
#cloud-config

# userData.yml

packages:
  - nginx
runcmd:
  - systemctl enable nginx
  - rm /var/www/html/*
  - >
    echo "I'm Nginx @ $(curl -4 https://ifconfig.me) 
    created $(date -u)" > /var/www/html/index.html
├── main.tf
├── ... 
└── userData.yml

exercise No. 20

Cloud-init by Terraform

Q:

Re-use your Cloud-init script from Cloud-init for starters . This time follow Figure 1106, “Terraform triggering Cloud-init ” using Terraform for executing Cloud-init rather than Hetzner's WEB GUI.

Figure 1107. Using template file interpolation Slide presentation
# main.tf
resource "local_file" "user_data" {
  content         = templatefile("tpl/userData.yml", {
  loginUser              = "devops"
                    })
  filename        = "gen/userData.yml"
}

resource "hcloud_server" "mySrv" {
  ...
  # Using gen/userData.yml
  user_data    = local_file.user_data.content
}
# tpl/userData.yml
...
users:
  - name: ${loginUser}
    ...
# gen/userData.yml
...
users:
  - name: devops
     ...

Figure 1108. Yaml syntax quirk 1: Missing indentation Slide presentation
resource "tls_private_key" "host" {
  algorithm   = "ED25519"
}
...
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private =
       tls_private_key.host.private_key_openssh
     host_ed25519_public  = ...
                })
  filename = "gen/user_data.yml"
}
...
ssh_keys:
  ed25519_private: |
    -----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA...
-----END OPENSSH PRIVATE KEY-----
# E1: File gen/user_data.yml is not valid YAML.
  in "<unicode string>", line 19, column 1:
    b3BlbnNzaC1rZXktdjEAAAAABG5vbmUA ...

Figure 1109. Yaml missing indentation resolution Slide presentation
# main.tf
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = 
       indent(4, tls....host.private_key_openssh)
     ...})
  filename = "gen/user_data.yml"
}
# gen/usr_data.yml
ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣...
␣␣␣␣...
␣␣␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1110. Yaml syntax quirk 2: Wrong indentation Slide presentation
# gen/usr_data.yml

ssh_keys:
  ed25519_private: |
␣␣␣␣-----BEGIN OPENSSH PRIVATE KEY-----
␣␣␣␣␣␣b3BlbnNzaC1 ...
␣␣␣␣␣␣...
␣␣␣␣␣␣...
␣␣␣␣␣␣-----END OPENSSH PRIVATE KEY-----
# /etc/ssh/ssh_host_ed25519_key
...
ssh_keys:
  ed25519_private: |
-----BEGIN OPENSSH PRIVATE KEY-----
␣␣b3BlbnNzaC1 ...
␣␣...
␣␣...
␣␣-----END OPENSSH PRIVATE KEY-----

Figure 1111. Cloud-init configuration file validation Slide presentation
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {})
  filename = "gen/user_data.yml"
}
#> cloud-init schema --config-file  gen/user_data.yml

Figure 1112. Watch out for your enemies! Slide presentation
root@hello:~# journalctl -f
May 06 04:41:20 hello cloud-init[898]: Cloud-init v. 22.4.2 finished at Mon, 06 May 2024 04:41:20 +0000. Datasource DataSourceHetzner.  Up 11.78 seconds
   ...
May 06 04:46:16 hello sshd[927]: Invalid user abc from 43.163.218.130 port 33408
May 06 04:46:17 hello sshd[927]: Received disconnect from 43.163.218.130 port 33408:11: Bye Bye [preauth]
May 06 04:46:17 hello sshd[927]: Disconnected from invalid user abc 43.163.218.130 port 33408 [preauth]
   ...
May 06 04:50:54 hello sshd[930]: fatal: Timeout before authentication for 27.128.243.225 port 59866
   ...
May 06 04:52:45 hello sshd[933]: Invalid user cos from 43.163.218.130 port 59776
   ...
May 06 04:53:04 hello sshd[935]: Invalid user admin from 194.169.175.35 port 51128
May 06 04:53:49 hello sshd[937]: User root from 43.163.218.130 not allowed because not listed in AllowUsers
May 06 04:53:49 hello sshd[937]: Disconnected from invalid user root 43.163.218.130 port 50592 [preauth]

exercise No. 21

Working on Cloud-init

Q:

We continue our exercise series ??? by adding a Cloud-init configuration:

  1. Follow Figure 1106, “Terraform triggering Cloud-init ” and Figure 1106, “Terraform triggering Cloud-init ” to create simple web server.

    Tip

    You will have to extend your current firewall configuration allowing inbound traffic to port 80.

    On success pointing your web browser of choice to http://<your server's IP> should result in something similar to:

    I'm Nginx @ "95.217.154.104" created Sun May 5 06:58:37 PM UTC 2024
  2. With respect to Figure 1112, “Watch out for your enemies! ” inspect the output of journalctl -f on your own server for a while. Then modify your current sshd configuration:

    On success the following sequence should be possible:

    $ ssh -v devops@95.217.154.104
    ...
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey ❶
    debug1: Next authentication method: publickey
    ...
    Last login: Sun May  5 19:21:12 2024 from 217.245.243.187
    devops@hello:~$ sudo su - ❷
    root@hello:~# hostname
    hello

    ❶

    password is not among the list of allowed authentication methods.

    ❷

    User devops may execute sudo commands by virtue of his membership in group sudo.

    On contrary ssh root login must be prohibited:

    $ ssh root@95.217.154.104
    root@95.217.154.104: Permission denied (publickey).
  3. Read the cloud-init documentation and/or related tutorials to:

    • Currently your (most likely outdated) cloud provider supplied distribution does not get upgraded on installation time:

      $ ./bin/ssh 
      ...
      devops@hello:~$ sudo su -
      root@hello:~# apt update
      Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
      Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
      ...
      Reading package lists... Done
      Building dependency tree... Done
      Reading state information... Done
      6 packages can be upgraded. Run 'apt list --upgradable' to see them.
      
      # apt list --upgradable
      Listing... Done
      less/stable-security,stable-security 590-2.1~deb12u2 amd64 [upgradable from: 590-2]
      libc-bin/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      libc-l10n/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]
      libc6/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales-all/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]

      Modify your Cloud-init configuration to upgrade your distribution at server creation time. If so required your system should also reboot.

    • Install and configure fail2ban limiting ssh failed connection attempts.

      Tip

    • Install the plocate file indexer package and initialize it.

    On success all packages should be up to date:

    $ ./bin/ssh 
    ...
    devops@hello:~$ sudo su -
    root@hello:~# apt update
    Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
    Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
    ...
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    All packages are up to date.

    Failed login attempts should be banned: Keep a second login open in advance when trying to simulate login failures! You should then see a report similar to:

    root@hello:~# fail2ban-client status sshd
    Status for the jail: sshd
    |- Filter
    |  |- Currently failed:	2
    |  |- Total failed:	14
    |  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
    `- Actions
       |- Currently banned:	2
       |- Total banned:	2
       `- Banned IP list:	170.64.133.30 213.136.94.219

    Searching for file name components should work like e.g.:

    root@hello:~# locate ssh_host
    /etc/ssh/ssh_host_ed25519_key
    /etc/ssh/ssh_host_ed25519_key.pub
Figure 1113. Problem: Duplicate known_hosts entry on re-creating server Slide presentation

Problem of repeated terraform apply:

$ ssh root@128.140.108.60
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

Figure 1114. Solution: Generating known_hosts ... Slide presentation
resource "local_file" "known_hosts" {
  content         = join(" "
                        ,[ hcloud_server.mySrv.ipv4_address
                        , tls_private_key.host.public_key_openssh ]
                    )
  filename        = "gen/known_hosts"
  file_permission = "644"
}
157.180.68.60 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...XXgAC3eaueHMQ8ePtbnw

Figure 1115. ... cloud-init config using yamlencode / JSON Slide presentation
resource "local_file" "user_data" {
  content  = templatefile("tpl/userData.yml", {
     host_ed25519_private = tls_private_key.host.private_key_openssh
     host_ed25519_public  = tls_private_key.host.public_key_openssh
     devopsUsername       = hcloud_ssh_key.loginUser.name
               ...
  filename = "gen/user_data.yml" ...
#cloud-config
${yamlencode(
  { ...
    ssh_keys = {
      ed25519_private = host_ed25519_private
      ed25519_public  = host_ed25519_public
    } ...
  }
)}
users:
  - name: ${devopsUsername} ...

Figure 1116. ... and ssh wrapper Slide presentation
main.tf tpl/ssh.sh
resource "local_file" "ssh_script" {
  content = templatefile("tpl/ssh.sh", {
    ip=hcloud_server.hello.ipv4_address
  })
  filename        = "bin/ssh"
  file_permission = "755"
  depends_on      = [local_file.known_hosts]
}
#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile= \
  "$GEN_DIR/known_hosts" devops@${ip} "$@"

exercise No. 22

Solving the ~/.ssh/known_hosts quirk

Q:

Extend Working on Cloud-init generating both a bin/ssh wrapper, a bin/scp wrapper and a corresponding gen/known_hosts file to be used by the two former scripts.

Avoiding ssh host key issues on login we require the creation of a known_hosts file among with each server being created:

157.180.78.16 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJDd+x7b80BM97rTU4RCM/CP+K7u4QAqx8ulDdXm9JDv

We assume this file being generated at gen/known_hosts. A corresponding ssh bash script shall be generated at the same level in bin/ssh:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" devops@157.180.78.16 "$@"

# end of script

This way invoking ./bin/ssh will instruct the ssh client to read gen/known_hosts and thus avoiding unknown host key issues. The "$@" allows for optional user provided parameters.

For generating this script we start from a corresponding tpl/ssh.sh template file:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile="$GEN_DIR/known_hosts" ${devopsUsername}@${ip} "$@"

# end of script

We then let Terraform use this template replacing ${devopsUsername} and ${ip} by your deployment's actual values.

As shown in Figure 1125, “Parent module vs. sub module context ” point to your child module's local tpl path by using ${path.module}:

resource "local_file" "ssh_script" {
  content = templatefile("${path.module}/tpl/ssh.sh", {
    ...
}

Likewise create a bin/scp script to be used for file transfer purposes as well:

#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

if [ $# -lt 2 ]; then
   echo usage: .../bin/scp ... devops@157.180.78.16 ...
else
   scp -o UserKnownHostsFile="$GEN_DIR/known_hosts" $@
fi

# end of script

Since copying objects requires at least a source and a destination parameter this script checks for the presence of at least two extra user supplied values by means of if [ $# -lt 2 ].

Figure 1117. Failsafe console login Slide presentation
#cloud-config

chpasswd:
  list: |
    root:bingo9wingo
  expire: False

Figure 1118. Avoiding Yaml parsing issues Slide presentation
#cloud-config

write_files:
  - content:
        ${base64encode(private_key_pem)}
    encoding: base64
    path:  /etc/nginx/snippets/cert/private.pem

Note

Congrats to paultyng