Jun 13, 2016

Expanding variables and shell expressions in parameters to Docker entrypoint

A known best practice when creating Docker images is when you need to run commands in runtime before starting the actual application/ daemon is to create an entrypoint script and pass the command as parameters in the CMD instruction. Another best practice is to exec the final command so it would be PID 1 and receive the signals passed to it. Let's create a small example. Here's the Dockerfile:

FROM alpine
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
ENV var value
CMD ["echo", "$var"]

And the entrypoint.sh script:

#!/bin/sh
set -eu
# Perform any needed tasks here.
exec $@

Now let's build and run this container:

$ docker build --tag entrypoint .
Sending build context to Docker daemon 28.67 kB
Step 0 : FROM alpine
 ---> 5f05d2ba9e65
Step 1 : COPY entrypoint.sh /entrypoint.sh
 ---> f59f4d7f3546
Removing intermediate container 27ca546c6b6c
Step 2 : ENTRYPOINT /entrypoint.sh
 ---> Running in 98c65b63948a
 ---> 1de45b33021b
Removing intermediate container 98c65b63948a
Step 3 : ENV var value
 ---> Running in 133a8781f0ac
 ---> bba451334fb2
Removing intermediate container 133a8781f0ac
Step 4 : CMD echo $var
 ---> Running in e8436c6c3202
 ---> a49d9b335b74
Removing intermediate container e8436c6c3202
Successfully built a49d9b335b74
$ docker run entrypoint
$var

As we can see the variable var wasn't expanded to it's content. After a bit of head scratching, The following simple change was made to the entrypoint script.

#!/bin/sh
set -eu
# Perform any needed tasks here.
eval "exec $@"

The change is to first evaluate the expression (expanding any variable and expression found), then exec it. The outcome is what you'd expect.

$ docker build --tag entrypoint .
Sending build context to Docker daemon 28.67 kB
Step 0 : FROM alpine
 ---> 5f05d2ba9e65
Step 1 : COPY entrypoint.sh /entrypoint.sh
 ---> b874d862999d
Removing intermediate container fb6483ff00e3
Step 2 : ENTRYPOINT /entrypoint.sh
 ---> Running in 82adf0b2c4c7
 ---> 6674f336c5e1
Removing intermediate container 82adf0b2c4c7
Step 3 : ENV var value
 ---> Running in 599f3f98c11d
 ---> 980f1e1e1ad5
Removing intermediate container 599f3f98c11d
Step 4 : CMD echo $var
 ---> Running in e29f1948480a
 ---> e27fd79143f8
Removing intermediate container e29f1948480a
Successfully built e27fd79143f8
$ docker run entrypoint
value

May 19, 2016

An example Ansible role

A few weeks ago I started a new job and a lot of time was spent on refactoring as well as adding to an existing Ansible automation code base. For me this was a chance to work more with Molecule for testing. Molecule is a infrastructure-as-code testing tool that is inspired by Test-kitchen and the tests can be written using Testinfra which in turn is using pytest. The reasons for me to choose this combination is that the tools are written in Python and that they're focused on Ansible. However I quickly grew tired of copying files from role to role or making the same changes to files again and again. So in that spirit I created a new Git repo with an empty Ansible role (no tasks, variables, handlers etc.) but has all of my changes and tweaks already applied and working tests out of the box.

Usage

To work on the role install VirtualBox and Vagrant (I use the versions in Debian's repos) and from PyPI Ansible, Molecule and Testinfra. Now, fork the repo. As you can see there are already README and LICENSE files. If you ever ran ansible-galaxy init or molecule init you'll notice that indeed the repo was created with those tools.

Dependencies

There's an example dependency present in meta/main.yml but instead of the declaring the dependencies in meta/main.yml and the sources of the dependencies in requirements.yml which leads to repeating yourself, the example shows how to declare the source of the dependent role directly in meta/main.yml (which I haven't seen mentioned clearly in the Ansible documentation. For repositories with playbooks I'd still add a requirements.yml file since there's no meta directory. Pulling the dependencies took some thought and what I came up with is:

ansible-galaxy install git+file://$(pwd),$(git rev-parse --abbrev-ref HEAD)

This is a workaround for installing the dependencies as it actually uses ansible-galaxy to install the git repo of the role and the dependencies as well.

Testing

First, I configured pre-commit hooks that check, among other things, the validity of the YAML files and the does a syntax check of the Ansible playbook.

As for Molecule, the configuration of the test environment is mainly under molecule.yml. That is were you'd go to change the Vagrant box to test. You can add multiple boxes and specify which box to test like so molecule test --platform <BOX_NAME>.

Also worth mentioning is the Ansible configuration in ansible.cfg. This is some what of a workaround as well because many of the options can be configured in molecule.yml which is used to generate its own ansible.cfg. However since Testinfra runs the tests over Ansible and Molecule doesn't pass the configuration along to it, the configuration isn't honored during testing. This caused me some grief as tests were constantly failing because Ansible would the host SSH key and fail as it was not known. The way I did is create an ansible.cfg at the root of the repo where Testinfra would look and passed that as the template to Molecule.

The playbook that is run in at tests/playbook.yml and the tests are under tests/ as well. There's an simple example test but the Testinfra documentation quite good. Just remember to that both the filename and function name should start with test_ and you won't have tests that aren't found.

A word on CI

Now you have all of the different pieces and workflow to run complete tests on roles the next obvious step is setting up a CI pipeline. In my tests and as I know the various CI services (I tried Travis-CI and CircleCI) disable the option to run any hypervisor. For me it's a deal breaker because I depend on VirtualBox (I need to test on different OSes, not just Linux). If LXC serves your needs than you should be able to run Vagrant with the LXC provider and therefore Molecule. For me it's a deal breaker.

A final word on boiler-plate

In a previous post I mentioned that I have several repositories that have the same boiler-plate and how I plan on dealing with that. Now, this is the first attempt at this. The idea is having a base repo that I clone, add another remote and voilĂ , a new project with the scaffolding already there. For bonus points, I can update the base repo and pull those changes in all projects. Here's how I do it:

git clone https://git.shore.co.il/ansible/ansible-role-example.git ansible-role-name
cd ansible-role-name
for file in $(git grep -l ansible-role-example); do sed -i 's/ansible-role-example/ansible-role-name/g' $file; done
git add .
git commit -m"- Renamed ansible-role-example to ansible-role-name."
git remote rename origin ansible-role-example
git remote add origin git@example.com/path/to/repo
git push -u origin master

And in case I update the ansible-role-example repo than I pull the updates by running git pull ansible-role-example master.

Mar 12, 2016

Pre-commit hooks

Pre-commit is a nice, simple tool to add Git hooks to your project. The primary goal is running fast checks on commits (before committing them), mainly linters and syntax checkers. Today I've 2 of my own, for Ansible playbooks and shell scripts. The Ansible playbooks hook is located at https://git.shore.co.il/ansible/ansible-pre-commit.git and the shell scripts hook is at https://git.shore.co.il/nimrod/shell-pre-commit.git. Both have a short README which describes installation and usage.

My view on testing

I find that Pre-commit suites my view on proportionate testing. The smaller the change, the faster the test (and as a result, more trivial). Personally, I prefer to structure my work as small commits that are easier to revert, these deserve fast (and more trivial) tests which Pre-commit provides. The bigger the change, the more rigorous (and thus longer) the test. In my opinion this helps in creating a good workflow which quickly finds small errors while developing and reduces the number of times one must ran the full test suite because he/she had a typo that failed the test. This is why I prefer to separate the test suite so that I can the ability to run the simpler and faster locally and get rid of simple error quickly.