Automation With Ansible

Written on 8 December 2020. Tagged with automation, self-hosting.

For many years, I have been renting a cheap dedicated server to self-host a bunch of services, including this website.

Since day one, I have been managing the server remotely using SSH, from upgrading to new Debian releases to installing packages and tweaking configuration files.

The inevitable happened

Although I thought about it a few times, I never invested time into learning and configuring automation tools to manage the server because it seemed overkill for a hobby.

Eventually, I ran into an issue while rebooting after a kernel upgrade: the server failed to come back up because of the aging power supply unit, according to technical support.

Long story short, the only thing left for me to do was to install a new server from scratch. However, I learned my lesson and I finally started a project to automate the installation.

Project overview

To get the job done, I am combining different tools:

Ansible to describe the server configuration in roles,
Molecule to test the main role in a container,
Testinfra to write unit tests in Python,
Moby to build a base image for the test container,

The project root has a conventional layout for anyone familiar with Ansible:

project
- roles
  - bob
  - system
- Dockerfile
- hosts
- install.yml
- Makefile
- README.md

A self-documented Makefile helps running common tasks:

$ make
env        Prepare environment
lint       Lint project files
play       Run install.yml playbook
test       Test system role

With Molecule, I am able to test the main role before applying it onto the server. Tests are executed inside a local container to verify its actual state:

$ make test
cd roles/system && MOLECULE_NO_LOG="false" molecule --verbose test
[...]
INFO     Running default > verify
INFO     Executing Testinfra tests found in roles/system/molecule/default/tests/...
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
plugins: testinfra-6.1.0
collected 17 items

molecule/default/tests/test_dns2tcp.py ..                                [ 11%]
molecule/default/tests/test_dovecot.py ..                                [ 23%]
molecule/default/tests/test_monit.py ..                                  [ 35%]
molecule/default/tests/test_nginx.py ..                                  [ 47%]
molecule/default/tests/test_openssh.py ..                                [ 58%]
molecule/default/tests/test_openvpn.py ..                                [ 70%]
molecule/default/tests/test_phpfpm.py ..                                 [ 82%]
molecule/default/tests/test_postfix.py ..                                [ 94%]
molecule/default/tests/test_users.py .                                   [100%]

============================== 17 passed in 5.23s ==============================
INFO     Verifier completed successfully.
[...]

In about three minutes, Molecule creates a new container, configures it with the role being tested, checks if the role is idempotent, executes unit tests and finally destroys the container. Pretty impressive!

When all tests pass, I can change the server configuration with Ansible:

$ make play
ansible-playbook --verbose --inventory hosts --ask-become-pass --ask-vault-pass install.yml
BECOME password:
Vault password:

PLAY [install server] **********************************************************
[...]
PLAY RECAP *********************************************************************
ok=42   changed=0   unreachable=0   failed=0   skipped=0   rescued=0   ignored=0

Confidence is key

I invested time in this automation project and I must say I am quite happy with the result:

my server configuration is safe because the project is versioned with Git and pushed to a private repository on GitHub,
the services I am hosting can be installed on a new server in a matter of minutes in case something goes wrong,
when Debian stable is released, it will come really handy to be able to execute tests inside a local container,
it is always rewarding to learn about new tools and techniques.

Now that I feel confident, I can upgrade the kernel and reboot the server anytime!