Ansible inventory management for AWS EC2 on a Small Scale

At TRI, we do everything the hard way… on a small scale. Many online services “best practices” and offerings are fine when you have a large operating budget and staff, and that leaves “smaller” shops with some notable problems to solve.

Until recently, we ran a bunch of AWS EC2 instances using Ansible, but Ansible Tower and various “dynamic inventory” schemes did not meet our realtively-small-scale needs. We don’t have hundreds or thousands of anything. What we wanted was a simply way to use a human-friendly naming convention for our hosts (ex: encoding-inspector-server-20200629-2) but still avoid having manually update rapidly-changing server IP addresses required for Ansible management.

Overview

We solved the problem like this:

  1. Create EC2 instances based on our local Ansible inventory file
  2. Query AWS for EC2 IP addresses and create an alternate SSH configuration file, mapping our local hostnames to pubic IP addresses
  3. Finally, configure Ansible to use the alternate SSH configuration file

Now we can use our inventory hostnames without having to update the IP addresses manually.

Implementation

We create new EC2 instances like this, using parameters from group_vars, hosts.yml, etc. This step ensures we’ve got the right number of instances running, and each is named with our internal inventory name.

---

- name: "Create ec2 instance(s) for {{ target_hosts | default('all') }}"
  hosts: "{{ target_hosts | default('all') }}"
  gather_facts: false
  tasks:

    - name: "Ensure ec2 instance for {{ inventory_hostname }}"
      ec2:
        key_name: "{{ aws_ec2_key_name }}"
        region: "{{ aws_ec2_region }}"
        image: "{{ aws_ec2_image }}"
        wait: "{{ aws_ec2_wait }}"
        vpc_subnet_id: "{{ aws_ec2_vpc_subnet_id }}"
        group: "{{ aws_ec2_group }}"
        instance_type: "{{ aws_ec2_instance_type }}"
        exact_count: 1
        count_tag:
          Name: "{{ inventory_hostname }}"
        instance_tags:
          Name: "{{ inventory_hostname }}"
        assign_public_ip: yes
        termination_protection: "{{ aws_ec2_termination_protection }}"
      delegate_to: 127.0.0.1
      register: ec2_results

This playbook ensures one AWS EC2 instance for each inventory hostname in our local inventory file. Key elements are:

  • instance_tags: / Name: "{{ inventory_hostname }} – Setting the Name tag to the local inventory name is the key to this solution.
  • exact_count: 1 – indicates that only one instance is desired that matches the “count_tag”
  • count_tag: / Name: "{{ inventory_hostname }}" – indicates the thing to count is the value associated with the Name: tag (which is, again, our local inventory name.)

After the EC2 instances are created (or terminated, for that matter,) we run the update_ec2_ssh_config.yml playbook to update the ~/.ssh/ec2_config file.

---
---

- name: Create ec2_config based on live ec2 instances
  hosts: all
  gather_facts: False

  tasks:

    - ec2_instance_info:
        aws_access_key: "{{ aws_access_key_id }}"
        aws_secret_key: "{{ aws_secret_access_key }}"
        region: "{{ item }}"
      register: ec2_instance_info
      delegate_to: localhost
      run_once: True
      with_items: "{{ aws_regions }}"

    - name: "Write ansible_ssh_config"
      template:
        src: templates/ec2_ssh_config.j2
        dest: "~/.ssh//ec2_config"
      vars:
        instances_by_region: "{{ ec2_instance_info.results }}"
        user: "{{ aws_ec2_user }}"
        identity_file: "{{ secure_keys_dir }}/{{ aws_ec2_key_name }}"
      delegate_to: localhost
      run_once: True
      changed_when: False # or else will imply all the instances are changed

Things to note:

  • ec2_instance_info module can only get info for one region at a time, so you must run it for each region where you have instances.
  • Vars for the template include the EC2 instance info, a user, and an identity file. All our instances use the same Ansible user (“ubuntu”) and the same ssh identity file to connect, so we do not provide separate values for each EC2 instance. You can see how this rolls out in the ssh config file template.

Here’s the template.

  • Host entries are arranged by region to match the EC2 info data we’re passing.
  • The “Host *” section applies the common user and identify_file value to all the hosts where it has not been set explicitly (i.e., all of them.)
  • Just for convenience, there’s a comment added for any inventory hostname where there’s no running EC2 instance.
  • The CheckHostIP no setting stops SSH from checking the known_hosts file against the key provided by the EC2 instance and screaming bloody murder every time we replace a machine (which has a new key.)
# {{ ansible_managed }}
# alternate ssh config file for ec2 instances

{% for region in instances_by_region %}
# Region {{ region.item }}

{% for instance in region.instances %}
{% if instance.tags.Name is defined %}
{% if instance.public_ip_address is defined %}
Host {{ instance.tags.Name }}
  Hostname  {{ instance.public_ip_address }}
{% else  %}
# Host {{ instance.tags.Name }}
#  Hostname  (no public ip address)
{% endif %}
{% endif %}

{% endfor %}
{% endfor %}

Host *
  User {{ user }}
  IdentityFile {{ identity_file }}
  CheckHostIP no

# end

With the new ec2_ssh_config file in place, all that’s left is to tell Ansible to use this config file instead of the default one for SSH.

...
#ssh_args = -C -o ControlMaster=auto  -o ControlPersist=300s
ssh_args = -F /Users/tomwilson/.ssh/ec2_config -o ControlMaster=auto -o ControlPersist=60s
...

The only option added is the “-F” option specifying the new SSH options file.

That’s it.

This solution lets us automatically create and manage AWS EC2 instances from a local inventory file, all without managing ever-changing IP addresses manually.

Bonus round

Now that there’s a new SSH options file, it’s possible to use this to connect to the EC2 instances from the command line, like this:

ssh my_inventory_hostname -F ~/.ssh/ec2_config

And for convenience I’ve added this executable file (note: on my Mac… this is certainly not how you would do this for Windows.)

#!/bin/bash
ssh $1 -F ~/.ssh/ec2_config "${@:2}"

And now all I have to remember (and type) is this:

sshec2 my_inventory_hostname

Discover the public IP address of an AWS Fargate container in a Docker entrypoint script

tl;dr: you must use the +tcp option with dig

We needed the public IP address to configure PASV_ADDRESS for a vsftpd service we’re running under AWS Fargate. Unfortunately, there’s no direct way for a Docker entrypoint script to get the current public IP address.

There are some annoyingly-complicated ways using AWS api’s, etc, but the simple solution is to use an external service. A quick Internet search revealed many references to this:

dig +short myip.opendns.com @resolver1.opendns.com

But if you have tried this from inside a Fargate-managed container, you have probably seen this:

root@ip-10-0-0-16:~# dig +short myip.opendns.com @resolver1.opendns.com
;; connection timed out; no servers could be reached

It turns out that DNS uses the User Datagram Protocol (UDP) and Fargate networking was blocking dig’s UDP request. Fortunately, dig has an option to make requests via TCP. Using the +tcp option, you should be able to get the public IP address like this:

root@ip-10-0-0-16:~# dig +tcp +short myip.opendns.com @resolver1.opendns.com
18.207.116.219

Black bean, corn and quinoa salad

Black bean, corn, and quinoa salad
photo: Kari Nelson

Tired of the cost of the WF version, and unsatisfied with online recipes I tried, I crafted up this one. The date on the original copy in my recipe book is 2016-07-14.

Yield is about 8 cups. This scales well, so long as you have a large enough mixing bowl.

Ingredients

  • 2 cups quinoa, cooked and cooled to room temperature

  • 1 can (15 oz) black beans, drained and rinsed

  • 2 cups frozen corn

  • 1 red bell pepper, small dice

  • 1 can (15 oz) roasted tomatoes, drained, rinsed, finely chopped

  • 1 or 2 jalapeƱo peppers, small dice

  • small bunch of cilantro, finely chopped

  • 1 1/2 tsp ground cumin

  • 3 oz olive oil

  • juice from 1 lime

  • 1/2 tsp salt

Directions

  • Cooking and cooling the quinoa takes most of the preparation time. I use an ice bath to drop the temp quickly.
  • Prep the rest of the ingredients while you’re waiting. Be sure to drain the beans and chopped tomatoes well to avoid water in the salad.
  • When the cooked quinoa gets down to room temp, mix everything in a large bowl. The frozen corn will cool it all enough to eat immediately, but it’s best to cool in the refrigerator a couple of hours.

TRI Resources T3 plugin for WordPress

TRI Resources International provides data, reference tools, and web services for the printer supplies industry. One of those services is “T3,” an API designed to serve data needed to support “ink and toner finders” like you see on virtually every printer supply retail site.

To avoid making each of our clients write an app to use the API, we’ve created a few javascript apps to generate some of the most common interfaces. We call these our “standard components.” Demo’ed here.

But inserting something like this into a WordPress site is a different task altogether. To simplify that process, we’ve created a WordPress plugin, “TRI T3 Shortcode.” Using the plugin, it’s easy to add T3 Standard Components into any WordPress page.

See T3 Standard Components, set up on this site using the TRI T3 Shortcode plugin.

Please contact TRI Resources International for more info about T3 or any of the related products or services.

Ansible, python and mysql… untangling the mess

Nearly every “how to” article or tutorial on the web describes one way of using Ansible, python, and connecting to MySQL as if that was the only solution.  Many don’t note code versions used, or even the pub date, and the Internet is rife with simply bad advice.  I finally gave up researching all this and ran a few quick tests  using VirtualBox/Vagrant to see what is really necessary to do a few things we need.  Our situation is:

  • Ubuntu 18.04, building instances using Ansible 2.7
  • We must manage remote mysql users
  • A target machine must be able to
    • run mysql commands from the command line (via a python script)
    • run mysql queries from inside a python script

Here’s what I found:

Python 2 vs python 3: just stop using python(2)

Unless you’re invested in a bunch of skanky old python(2) code, there is no reason to NOT use python3.  Python3 is the default version, and comes pre-installed on Ubuntu 18.04 (at least all of the images we are using) and it is accessible as “python3”.  Install it like this on your local machine too.  (Python 2 and python 3 are not compatible, so they are reasonably installed using different executable names. Don’t fight this.  The Homebrew folks added some confusion at first by installing python 3 as “python” in some cases, but that’s fixed now.)

To use python3 with Ansible, you must set the variable ansible_python_interpreter: “python3” , and then Ansible will just use python3 and you won’t need to mess around installing python(2) at all.  For anything.

What’s needed to use the Ansible mysql_user module: the PyMySQL pip package

Ansible runs on python (specifically, python3, if you are following along.)  If you try and run a mysql_user task without installing the necessary pip packages, you’ll get a surprisingly helpful error message: “The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required.”  So to make this work in Ansible, just install the PyMySQL pip package:

# this package is required to use the pip module
    - apt:
        name: python3-pip
        state: present

    # this is the required pip package
    - pip:
        name: PyMySQL

    # and finally, now you can use mysql_user module
    - mysql_user:
        login_host: 127.0.0.1
        login_user: fake_master_user
        login_password: fake_master_password
        login_port: 3306
        user: fake_user
        password: fake_password
        host: 127.0.0.1

What’s needed to run mysql from the command line: the mysql-client linux package

Finally with Ubuntu 18.04 and current versions of mysql packages, we have success by installing a single linux package.  This package makes the “mysql” command available from the command line for all users (and from python scripts.)

      - apt:
          name: mysql-client
          state: present

What’s needed access mysql databases from within a python script: just use the PyMySQL pip package

Our batch jobs do most of the mysql work using the mysql command line (using python’s subcommand with the shell=True option,) but there are times when we need to read some data in the script to determine what tasks to perform, etc.  The two most common pip packages are PyMySQL and mysql-connector-python.  After digging into the code examples, it turns out that there are only a few differences in the interface, and my conclusion is that for most purposes there is no real difference.

I recommend using PyMySQL, because it’s already required for the Ansible mysql_user module, and you can install it on the target host using the same Ansible tasks show above.

Note: WordPress’ unrelenting march to make editing easier totally wrecked this post, due to what I consider blatant disregard for backward compatibility with respect to existing posts. I’ve had to completely reassemble this. My apologies if it’s a bit out of order.