How I deploy Kubernetes clusters easily with Ansible and RKE2.

Lately at $DAYJOB I’ve been tasked with deploying a couple of new Kubernetes clusters. I chose to use RKE2, to keep in line with the other clusters we run.

The problem

I’m lazy and I don’t like to do the same things over and over again because I will inevitably mess something up in the process, so I like to have things automated.

The solution

I’m using Ansible to deploy the clusters. Being even lazier than what you may think, I use the open source RKE2 Ansible Role from the fine folks at Labyrinth Labs, so I just need to write a playbook and an inventory file.

Some notes:

• We use cilium as CNI, so I added a custom manifest to the role to disable kube-proxy.
• We use a Load Balancer to expose the Kubernetes API server, so I set rke2_api_ip to its IP address.
• We use LVM and local-path-provisioner for the storage, maybe you prefer to do things differently.
• I like k9s to manage my clusters, so I install it by default on the control plane nodes.

Here are an inventory file, a custom manifest for cilium, and a playbook that will deploy a multi-node RKE2 cluster on freshly installed Ubuntu 24.04 servers.

# inventory.yml
[masters]
# The control plane machines
#
# Entry format:
# <env>-cp-<number> ansible_host=<IPv4 address> rke2_type=server
# <env> is the environment name, <number> is the machine number
# <IPv4 address> is the IP address of the machine
# <rke2_type> is either server or agent
env-cp-1 ansible_host=xxx.xxx.xxx.xxx rke2_type=server
env-cp-2 ansible_host=xxx.xxx.xxx.xxx rke2_type=server
env-cp-3 ansible_host=xxx.xxx.xxx.xxx rke2_type=server

[workers]
# The worker machines
#
# Entry format:
# <env>-wo-<number> ansible_host=<IPv4 address> rke2_type=agent
env-wo-1 ansible_host=xxx.xxx.xxx.xxx rke2_type=agent
env-wo-2 ansible_host=xxx.xxx.xxx.xxx rke2_type=agent
env-wo-3 ansible_host=xxx.xxx.xxx.xxx rke2_type=agent

[k8s_cluster:children]
masters
workers

[k8s_cluster:vars]
#A long random token (this is sensitive like a password)
#see https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-cluster-setup/rke2-for-rancher#1-install-kubernetes-and-set-up-the-rke2-server
rke2_token=RANDOMLY_GENERATED_TOKEN

# The version of RKE to use
rke2_version=vx.y.z

#IPv4 address of the Load Balancer, maybe you don't need this
rke2_api_ip=xxx.xxx.xxx.xxx

# The file where to store the kubernetes client configuration, in the format <env>.yaml
rke2_download_kubeconf_file_name=env.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    image:
      tag: "vx.y.z"
    kubeProxyReplacement: true
    k8sServiceHost: {{ rke2_api_ip }}
    k8sServicePort: {{ rke2_apiserver_dest_port }}
    cni:
      chainingMode: "none"
    hubble:
      enabled: true
      ui:
        enabled: true
      relay:
        enabled: true
        image:
          tag: "vx.y.z"

# playbook.yml
- name: Deploy RKE2
  hosts: all
  remote_user: root
  vars:
    rke2_custom_manifests:
      # Add your custom manifests here
      - "{{ playbook_dir }}/manifests/rke2-cilium-config.yaml"

    rke2_server_node_taints:
      - 'CriticalAddonsOnly=true:NoExecute'

    rke2_download_kubeconf: true
    rke2_download_kubeconf_path: "{{ playbook_dir }}/kubeconfs"

    rke2_ha_mode: true

    # We use cilium as CNI, so we disable kube-proxy
    disable_kube_proxy: true

    # Not required for us since we use a Load Balancer, you may need it
    rke2_ha_mode_keepalived: false

    rke2_disable:
      - rke2-canal
      - rke2-ingress-nginx

    rke2_cni:
      - cilium

  pre_tasks:

    - name: Check if ubuntu
      ansible.builtin.fail:
        msg: No ubuntu but {{ ansible_distribution }}
      when: ansible_distribution != 'Ubuntu'

    - name: Check if ubuntu 24
      ansible.builtin.fail:
        msg: No ubuntu 24 but {{ ansible_distribution_major_version }}
      when: ansible_distribution_major_version != '24'

    - name: Gather information about lvm
      stat:
        path: /dev/vg0/root
      register: vg0_root

    - name: Check for /dev/vg0/root
      ansible.builtin.fail:
        msg: No /dev/vg0/root found. LVM configuration seems not ok.
      when: rke2_type == 'agent' and (not vg0_root.stat.exists)

    - name: Check if the NetworkManager directory exists
      ansible.builtin.stat:
        path: "/usr/sbin/NetworkManager"
      register: nwm_dir

    - name: NetworkManager found
      ansible.builtin.fail:
        msg: Network Manager present
      when: nwm_dir.stat.exists

    - name: Disable SWAP
      shell: |
        swapoff -a

    - name: Disable SWAP in fstab
      replace:
        path: /etc/fstab
        regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
        replace: '# \1'

    - name: Set a hostname {{ inventory_hostname }}
      ansible.builtin.hostname:
        name: "{{ inventory_hostname }}"

    - name: Update Timezone to Etc/UTC
      copy: content="Etc/UTC\n" dest=/etc/timezone owner=root group=root mode=0644
      register: timezone

    - name: Reconfigure Timezone Data (if changed)
      shell: dpkg-reconfigure -f noninteractive tzdata
      when: timezone.changed

    #sysctl -w fs.inotify.max_user_instances=100000
    - name: "Set fs.inotify.max_user_instances"
      sysctl:
        name: fs.inotify.max_user_instances
        value: '100000'
        sysctl_set: yes
        state: present
        reload: yes

    #sysctl -w fs.inotify.max_user_watches=1003986
    - name: "Set fs.inotify.max_user_watches"
      sysctl:
        name: fs.inotify.max_user_watches
        value: '1003986'
        sysctl_set: yes
        state: present
        reload: yes

    #sysctl -w vm.max_map_count=262144
    - name: "Set vm.max_map_count"
      sysctl:
        name: vm.max_map_count
        value: '262144'
        sysctl_set: yes
        state: present
        reload: yes

    - name: Add or modify hard nofile limits for wildcard domain
      community.general.pam_limits:
        domain: '*'
        limit_type: hard
        limit_item: nofile
        value: 1000000

    - name: Add or modify soft nofile limits for wildcard domain
      community.general.pam_limits:
        domain: '*'
        limit_type: soft
        limit_item: nofile
        value: 1000000

    - name: Update apt repo and cache on all boxes
      apt: update_cache=yes force_apt_get=yes cache_valid_time=3600

    - name: Upgrade all packages on servers
      apt: upgrade=dist force_apt_get=yes

    - name: Install additional software
      apt:
        update_cache: true
        pkg:
        - linux-headers-generic
        - linux-headers-{{ ansible_kernel }}
        - curl
        - git
        - wget
        - vim
        - nano
        - htop
        - netcat-openbsd
        - net-tools
        - dnsutils
        - jq
        - lvm2

    # I like k9s, you may not
    - name: Install k9s on control plane nodes
      shell: |
        curl -L -O https://github.com/derailed/k9s/releases/latest/download/k9s_linux_amd64.deb && sudo dpkg -i k9s_linux_amd64.deb && rm k9s_linux_amd64.deb
      when: rke2_type == 'server'

    - name: Get lvm information
      shell: |
        ls -la /opt
        cat /etc/fstab
        lsblk -f
        vgdisplay || true
        lvdisplay || true
      register: result

    - name: Print lvm information
      ansible.builtin.debug:
        var: result

    - name: Check for /opt/local-path-provisioner
      stat:
        path: /opt/local-path-provisioner
      register: local_path_provisioner

    - name: Create logical volumes for local path provisioner on server nodes
      shell: |
        lvcreate -n k8s-local-path -L1500G vg0
        mkfs.ext4 /dev/vg0/k8s-local-path
        mkdir /opt/local-path-provisioner
        echo "/dev/vg0/k8s-local-path /opt/local-path-provisioner ext4 defaults,noatime 0 0" >> /etc/fstab
        mount -a
      register: result
      when: rke2_type == 'agent' and (not local_path_provisioner.stat.exists)

    - name: Print lvm create information
      ansible.builtin.debug:
        var: result

    - name: Check if a reboot is needed on all servers
      register: reboot_required_file
      stat: path=/var/run/reboot-required

    - name: Check if the rke2/config.yaml exists
      stat:
        path: /etc/rancher/rke2/config.yaml
      register: rkeconfig_result

    - name: Reboot the box on first install
      reboot:
        msg: "Reboot initiated by Ansible for first install"
        connect_timeout: 5
        reboot_timeout: 300
        pre_reboot_delay: 0
        post_reboot_delay: 30
        test_command: uptime
      #when: reboot_required_file.stat.exists and (not rkeconfig_result.stat.exists)
      when: not rkeconfig_result.stat.exists

    - name: Check SWAP disabled
      shell: |
        LINES=$(cat /proc/swaps | wc -l)
        if [ "$LINES" != "1" ]; then
          # 1 means only the headers and no swap
          echo "LINES is $LINES"
          exit 1
        fi

    - name: Create a networkd.conf.d for drop-ins
      ansible.builtin.file:
        path: /etc/systemd/networkd.conf.d/
        state: directory
        owner: root
        group: root
        mode: '0755'

    - name: Configure systemd Network RoutingPolicyRules
      ansible.builtin.copy:
        dest: /etc/systemd/networkd.conf.d/rke2-network.conf
        content: |
          [Network]
          ManageForeignRoutes=no
          ManageForeignRoutingPolicyRules=no
        owner: root
        group: root
        mode: '0644'
      register: systemdnetwork

    - name: Reload systemd networkd
      ansible.builtin.systemd:
        state: restarted
        daemon_reload: true
        name: systemd-networkd
      when: systemdnetwork.changed

    - name: Add k8s Binaries to PATH
      ansible.builtin.lineinfile:
        path: /root/.bashrc
        line: "{{ item }}"
      loop:
        - "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml"
        - "PATH=$PATH:/var/lib/rancher/rke2/bin"

  roles:
    - role: rke2

As you can see, most of the playbook is dedicated to the initial configuration of the servers (hostname, timezone, networking, LVM, etc.), while the deployment of RKE2 is completely handled by the role.

Feel free to get in touch if you have any questions or suggestions.
Till next time!