My notes and ramblings

My experience with security researchers

2024-11-29T00:00:00+02:00

I've been sitting on this blog post for a long while. I have a history of working with (so called) security researchers that I would describe as poor. I don't want to besmirch the profession of security research. I enjoy reading security research write ups, I follow a lot of the security best practices, I subscribe to the security mailing lists of the OSes I use and overall have high regard for the professionals in the field.

On the other hand, over my career I have many different interactions with security companies and researchers working in those companies that have all been bad. I worked at a cyber security company along side security researchers from the IDF's 8200 unit. I received notices on security vulnerabilities on sites I or the companies I work at run (especially after publishing a security.txt policy). But best of all is my experience with the Israeli National Cyber Directorate. Let me get started.

Security audits and certifications

A few of the companies I worked at went through security audits to get a certification (SOC2 or HIPAA). As the person responsible for the infrastructure and our CI/CD pipelines I was a part of the audit from beginning to end and when the audit report was delivered, I addressed some of the findings. From the few audits I took part in, I can say that the worst was a company that ran a few automated scanners in the vein of SSL Test and the better ones ran something akin to Semgrep and maybe checking the OWASP top 10.

All of the audits I've been part of had not produced any worthwhile results. No actual vulnerabilities were ever found and most the time a few publicly available security scanners were used (the screenshot from the SSL Test is still vivid in my mind).

Working with security researchers

I worked at a cyber security company with an actual cyber security product. There we had a security research team with people from the IDF 8200 unit. From my dealings with them, they have poor knowledge of things you would expect (on the level of not knowing the difference between symmetrics and asymmetric encryption) and their research can boiled down to running Nmap and Metasploit.

When one of them learned that I run my own mail server, he claimed to be able to break in to my server. I said go for it, hoping to learn something new and fix whatever vulnerability my server may have. Looking over his shoulder, I saw that he was running Metasploit with a preset for mail servers. Having found nothing (not because I'm that good, I just install security updates and have sane settings) he turned quiet.

The Israeli National Cyber Directorate

I saved the best for last, the reason I felt the urge to write this post. Over the last 3 or 4 years I was contacted 3 times by the INCD to let me know of vulnerabilities they found in my personal sites and services.

The first time I was contacted by phone. I was a little surprised and took the matter seriously. I was told that my mail server had an RCE. Asking for details, I was told the CVE and the person on the other end explained to me that I need to update my mail server. I quickly checked the CVE and I found that Debian had backported the patch but the server version stayed the same (or maybe some suffix was added, I don't remember). I tried to explain that I had a patched server but it fell on deaf ears and they were adamant that the version I was using vulnerable and I had to update ASAP. I thanked them for letting me know and promised to look in to it.

The second time I was again contacted by phone. This time I was less surprised. I was told that my GitLab instance was misconfigured, although it required logging in, repositories were exposed through the /explore URL. I explained that it was deliberate, that I develop opensource software and that is were I store it and make it available for others (if you take a look, all of the repositories have an opensource license and my blog even links to them). Again, it didn't convince the person on the other side. I thanked them for letting me know and promised to look in to it.

The third time I was again contacted by phone. This time I was not a bit surprised. I was told that my SSH server is vulnerable and I have to update it. I explained that I am running OpenSSH on an OpenBSD machine and that the vulnerability in question only happens on Linux machines. The person on the other end didn't know what OpenBSD is (I tried explaining that the developers of OpenBSD also develop OpenSSH, they didn't seem to get it). Showing my age, I complained that this is a waste of the taxes I pay. The person on the other end didn't appreciate it and ended the call.

Closing thoughts

When I was growing up and the internet was becoming accessible to everyone a new phenomenon named script kiddies started. People scanning ports, open Windows shares and guessing SSH usernames and passwords. Then somebody got the bright idea of making a career out of it by selling people some scary stories and exaggerating their own capabilities and calling it security research. While true that there are unpatched and vulnerable machines on the internet, this is not security research and because I have sensible security practices I only encountered false positives due to rudimentary scanners flagging my servers as vulnerable without checking if they are indeed vulnerable.

I don't remember which company it was, but I remember one such company had an realtime map of the internet showing realtime attacks. Looking closely, each ping and each new connection to port 22 was an attack. The field is now filled with charlatans that instead of trying to break in to your servers now try to bill you for running Nmap or verifying your DMARC record. They've turned this in a very successful industry and the Israeli government seems to have fallen to this trap as well (as I'm pretty sure other goverments have as well).

Copy a drive on a running machine

2024-06-08T00:00:00+03:00

I was close to running out of disk space on the homelab machine I use for running services with personal data that I run in my home (I feel more comfortable having physical control). I was too lazy to connect the machine to a monitor and keyboard and reboot from a thumbdrive so I tried to copy the old drive to a new drive while the machine was running. Here's what I did:

Stop running processes that have open files. For me it was stopping Docker containers, some services (I have most things running in containers so most of the services are ones that come with a standard OS installation).

docker container ls | awk 'NR>1 {print $1}' | xargs docker stop
sudo systemctl stop fwupd.service systemd-timesyncd.service docker.service docker.socket containerd.service cron.service systemd-journald.service systemd-journald.socket systemd-journald-dev-log.socket

cd to a directory on a different drive to avoid keeping the path open (cd /tmp). Remount all partitions as read-only: sudo mount --all -o remount,ro -t vfat,btrfs,ext4. This may fail and you will see an error with a path that couldn't be remounted due to open files. Run lsof /path/that/has/open/files, find the processes that still have open files and either stop that service or kill the process (stopping the service is better as it won't get restarted). Repeat until the partitions are mounted read-only. Verify with the mount command. Run sudo sync and now you should be able to copy the drive as there won't writes to it while copying and the copy will be consistent.
Start a session on the host using tmux and copy dd if=/dev/sda of=/dev/nvme0n1. Now we wait.
Once the copy is complete, run sudo sync again and echo 1 | sudo tee /sys/block/nvme0n1/device/rescan. Now we need to resize the data partition, sudo parted /dev/nvme0n1 --script resizepart 3 100% (in my case the data partition was the 3rd partition after the EFI and root partitions). Open the encrypted partition sudo cryptsetup open /dev/nvme0n1p3 _dev_nvme0n1p2 and resize the encrypted partition sudo cryptsetup resize _dev_nvme0n1p3. Last resize is the filesystem in the encrypted partition, we mount it sudo mount /dev/mapper/_dev_nvme0n1p3 /mnt and resize it sudo btrfs filesystem resize max /mnt. Umount the data partition sudo umount /mnt.

Let's reinstall the boot loader.

sudo mount /dev/nvme0n1p2 /mnt
sudo mount /dev/nvme0n1p1 /mnt/boot/efi
for dir in dev proc sys; do sudo mount --bind "/${dir}" "/mnt/${dir}"; done
sudo chroot /mnt
grub-install /dev/nvme0n1
exit
for dir in dev proc sys; do sudo umount "/mnt/${dir}"; done
umount /mnt/boot/efi
umount /mnt

Because we copied the existing partitions, their UUIDs will remain the same so there's no need to update mounts. Stop the machine, remove the old drive and boot. With a little luck the UEFI BIOS will not need configuration to use the new drive and the machine will boot happily.

PEP 621 and Setuptools

2022-04-28T00:00:00+03:00

In the world of Python packaging, the trend is moving to using pyproject.toml to store all of the package's metadata as well as settings for different tools. Until recently, this trend meant adding another tool like Poetry, Flit or PDM instead of using the regular Setuptools (and maybe build). But with Setuptools 61.0.0, you can still use Setuptools and have a modern Python package.

Why?

Avoid having a plethora of Python package managers installed. This is more noticeable when some are wrappers or replacements of virtualenv and then you need to install them globally.
With Setuptools you keep close to the state of PEP 621 and your pyproject.toml generic

Why not?

You still need to manage virtual environments manually.
You need a very recent version of Setuptools. At the time of writing this post I didn't find a distribution that had a new enough version of Setuptools packaged.
You need a supported version of Python. Currently this means Python 3.7 or later.
You can get very close with an equivalent setup.cfg and get backwards compatibility.

How?

I created a sample Python project that has everything you need to get started. It's tested to make sure a valid and working package, that editable installation work, etc. You can skip directly to the pyproject.toml file.

A few notes. The only part that's Setuptools specific is specifying the attribute for getting the version dynamically. You can specify the version explicitly and be totally generic. The other settings that can be set dynamically need to be read from other files which isn't great. I would like to see the description be an attribute like version so I can point it to the docstring like Flit does. For editable installations you still need a setup.py file but it's a 2 line file that calls setup(). You don't need a setup.cfg file at all.

Looking forward

I'm pretty sure that once PEP 621 is approved, other package managers will adapt and tool specific configuration will be dropped. Also, support for using pyproject.toml in Setuptools will stop being experimental. I think that once that's done and the only difference will be the 2 lines specifying the build system we will see projects that want easier onboarding adopt Setuptools to avoid forcing more tools on users. Over time distributions will package newer versions of Setuptools and you will be able to have a working Python environment just with the distribution packages.

Appendix - PEP 582

Another PEP that's yet to be approved is PEP 582. This propsal is for a package directory that's local to a project (in the spirit of node_modules). The reason I mention it here is that this removes the need for virtual environments (and thus the other major selling point for the different package managers). Now, while it's not been approved and there's no version of Python that supports it out of the box, we can mimic it with a little creative shell scripting:

PYTHON_VERSION="$(python3 -c 'from sys import version_info as v; print(f"{v[0]}.{v[1]}")')"
export PYTHONPATH="$PWD/__pypackages__/$PYTHON_VERSION:${PYTHONPATH:-}"
export PATH="$PWD/__pypackages__/$PYTHON_VERSION/bin:$PATH"

With the above snippet we point Python to load packages from the __pypackages__ directory and add the bin directory to the PATH so executables are available. Lastly to install packages in that location we run python3 -m pip install -t "__pypackages__/$PYTHON_VERSION" foo. I use direnv quite a lot and I have the following snippet in a few projects:

PYTHON_VERSION="$(python3 -c 'from sys import version_info as v; print(f"{v[0]}.{v[1]}")')"
export PYTHONPATH="$PWD/__pypackages__/$PYTHON_VERSION:${PYTHONPATH:-}"
export PATH="$PWD/__pypackages__/$PYTHON_VERSION/bin:$PATH"
python3 -m pip install "__pypackages__/$PYTHON_VERSION" -e .

This performs an editable installation of the package and all it dependencies, so from that moment on I'm set and I don't any other package manager or even a virtual environment.

Setting up my ns4 instance

2022-02-26T00:00:00+02:00

This is one of the posts that is more of me writing what I did so I can do it again later and isn't geared toward the general public, you've been warned. A little bit of background. The instance in question is my bare-metal Dedibox instance. I use it for the following:

Secondary DNS instance
Docker registry
Some static web sites.
GitLab CI runner
Remote Workbench instance.

A bare-metal instance is very performant, allows me to run KVM and VirtualBox VMs and I feel a little better on the security front. Although I can due a remote KVM install of Debian, so far I've been using the online installation. This makes me wonder how stock is the installation and what I'll encounter if I do an in-place upgrade of Debian to a new release. The only data I care about are the images in the registry and they are backed up weekly to a machine in my house that I have off-site backups for. /var and /home are subvolumes in a btrfs volume in a LUKS encrypted partition. I enter the encryption key using KVM over IP. I have some private information but nothing I can't just git clone back to existence. I acknowledge that it's not as secure as can be but it's good enough for me and I judge the tradeoffs to be valid.

Everything listed above is deployed using GitLab CI pipelines but there's a chicken and egg situation here that some of the services depend on each other and at the very least I can't deploy the GitLab runner using GitLab pipelines. So here's what I did.

OS installation

OS installation takes ~15 minutes but afterwards KVM over IP is disabled for another hour. I work around this limitation by opening the KVM session first and then go ahead with the OS installation. In the online.net console, install Debian using the wizard. It's pretty standard stuff but the partitioning default to a RAID-1 setup which I don't want. Instead have /boot/efi at 512 MB, /boot at 2048 MB and / using the rest on the first drive (the order is important as changes don't boot for some reason I'm not interested enough to find). Because I must have a partition on each drive, keep /data on the second using all of the space.

Bootstrapping

Remove the existing SSH host key:

ssh-keygen -R ns4.shore.co.il
ssh-keygen -R "$(dig ns4.shore.co.il)"

Connect using KVM as root and setup sudo.

apt update
apt install -y sudo
echo 'nimrod ALL=(ALL) NOPASSWD: ALL' | tee /etc/sudoers.d/nimrod

Now SSH as normal and check sudo (sudo whoami). Now remove the passwords and the SSH public key from the root account.

sudo passwd --delete nimrod
sudo passwd --delete root
sudo rm -r /root/.ssh

Setting up the encrypted partition.

sudo umount /data
sudo rm --dir /data
sudo sed -i '#/data#d' /etc/fstab
sudo apt install -y btrfs-progs cryptsetup parted
sudo parted /dev/sdb --script mklabel gpt mkpart primary 0% 100%
sudo cryptsetup luksFormat /dev/sdb1
sudo cryptsetup luksOpen /dev/sdb1 data
sudo mkfs -t btrfs /dev/mapper/data
sudo mount /dev/mapper/data /mnt
sudo btrfs subvolume create /mnt/builds
sudo btrfs subvolume create /mnt/home
sudo btrfs subvolume create /mnt/var
sudo mkdir /builds
UUID="$(lsblk --output UUID --list --noheadings --nodeps /dev/sdb1)"
echo "data UUID=$UUID none luks,discard" | sudo tee -a /etc/crypttab
echo "/dev/mapper/data /builds btrfs defaults,compress=zstd,discard,relatime,subvol=builds" | sudo tee -a /etc/fstab
echo "/dev/mapper/data /home btrfs defaults,compress=zstd,discard,relatime,subvol=home" | sudo tee -a /etc/fstab
echo "/dev/mapper/data /var btrfs defaults,compress=zstd,discard,relatime,subvol=var" | sudo tee -a /etc/fstab
sudo cp --archive /home/* /mnt/home/
sudo cp --archive /var/* /mnt/var/
sudo umount /mnt
sudo cryptsetup luksClose data
sudo systemc reboot --now

The instance will now reboot, enter the encryption key in the KVM console. From the homelab repository:

cd ./Ansible
ansible-playbook bootstrap.yaml -l ns4
ansible-playbook debian_server.yaml -l ns4

Services

From the secondary-dns-docker repository:

eval $(docker remote ns4.shore.co.il)
docker-compose build
docker-compose pull
docker-compose up -d

Same as above, from the web-proxy-docker repository, branch ns4. Some secret environment variables are set in the GitLab pipeline, so for them to exist re-run the deploy job in GitLab afterwards.

From the homelab repository:

cd ./Ansible
ansible-playbook renew-certs.yaml -t ns4

Setup the Docker registry, but the cron container uses an image from the registry. Start just the registry container and afterwards run the GitLab deploy job. From the registry-docker repository:

eval $(docker remote ns4.shore.co.il)
docker-compose pull registry
docker-compose up -d registry
docker build -t registry.shore.co.il/registry-backup ./backup

Copying the registry backup to ns4 (it's going to take a long while).

scp -3Cr host01.shore.co.il:/var/backups/registry ns4.shore.co.il:/home/nimrod/

On ns4:

docker run --rm -u nobody -v /home/nimrod/registry:/registry:ro registry.shore.co.il/registry-backup restore /registry registry.shore.co.il
rm -r /home/nimrod/registry

From the gitlab-runner-docker repository (delete the old runner in the GitLab admin area):

eval $(docker remote ns4.shore.co.il)
docker-compose pull
./deploy ns4

From here on out it's just running GitLab jobs and pipelines for the static web sites. etc. Remember to run the deploy jobs for web-proxy-docker and registry-docker.

Remote workbench

Follow the instructions in the rcfiles repository and finish it off with wb -u.

Closing thoughts

Along with my previous post on my OpenBSD machine covers the more involved parts of my infrastructure. Overall, I think that I'm in a pretty good state here. My setup is cost conscious, relies very little on external services and is more pets than cattle. The coverage of infrastructure code is high and the parts that aren't covered are done rarely (like reinstalling OpenBSD which I do about once a year or reinstalling Debian which I do about every 2 years) and are either trivial or nicely documented.

Going forward, I'm in the process of merging the different *-docker repositories in to the homelab repository. Once done, I may copy the blog posts there and maintain them ongoing as changes are done.

Installing OpenBSD on Netgate's SG-2440

2021-09-19T00:00:00+03:00

This is documentation on how to install OpenBSD (in this case 6.9 but the procedure hasn't changed for as long as I can remember). Since the SG-2400 only has a serial connection (no monitor output), about half of the is over the serial console and the rest is over SSH. This post is for me to help me remember what did I do last time.

Setting up the serial console

On the laptop:

sudo apt install screen
sudo modprobe cp210x
dmesg --follow

Now connect the cable and watch the dmesg output to see the serial connection being added (should be at /dev/ttyUSB0) and then:

sudo screen /dev/ttyUSB0 115200

Insert the USB drive with the installer and reboot (shutdown -r now). To enable the serial connection in the installer, in the boot prompt run the following commands:

stty com1 115200
set tty com1
boot

The interactive installer prompts and answers:

Terminal type? [vt220]
System hostname? (short form, e.g. 'foo') ns1
Available network interfaces are: em0 em1 em2 em3 em4 em5 vlan0.
Which network interface do you wish to configure? (or 'done') [em0] em5
IPv4 address for em5? (or 'dhcp' or 'none') [dhcp] 192.168.3.1
Netmask for em5? [255.255.255.0]
IPv6 address for em5? (or 'autoconf' or 'none') [none]
Available network interfaces are: em0 em1 em2 em3 em4 em5 vlan0.
Which network interface do you wish to configure? (or 'done') [done]
Default IPv4 route? (IPv4 address or none)
A response is required.
Default IPv4 route? (IPv4 address or none) none
DNS domain name? (e.g. 'example.com') [my.domain] shore.co.il
DNS nameservers? (IP address list or 'none') [none] 9.9.9.9

Password for root account? (will not echo)
Password for root account? (again)
The root password must be set.
Password for root account? (will not echo)
Password for root account? (again)
Start sshd(8) by default? [yes]
Change the default console to com1? [yes]
Available speeds are: 9600 19200 38400 57600 115200.
Which speed should com1 use? (or 'done') [115200]
Setup a user? (enter a lower-case loginname, or 'no') [no] nimrod
Full name for user nimrod? [nimrod] Nimrod Adar
Password for user nimrod? (will not echo)
Password for user nimrod? (again)
WARNING: root is targeted by password guessing attacks, pubkeys are safer.
Allow root ssh login? (yes, no, prohibit-password) [no] yes

Available disks are: sd0 sd1 sd2.
Which disk is the root disk? ('?' for details) [sd0] ?
sd0: ATA, Micron_M600_MTFD, MU04 naa.500a0751122dae7a (119.2G)
sd1: SanDisk, Cruzer Blade, 1.26 serial.07815567071025103004 (3.7G)
sd2: Generic, Ultra HS-COMBO, 1.98 serial.04242240000000225001 (28.5G)
Available disks are: sd0 sd1 sd2.
Which disk is the root disk? ('?' for details) [sd0] sd2
Disk: sd2       Usable LBA: 64 to 59768768 [59768832 Sectors]
   #: type                                 [       start:         size ]
------------------------------------------------------------------------
   1: EFI Sys                              [          64:          960 ]
   3: OpenBSD                              [        1024:     59767745 ]
Use (W)hole disk MBR, whole disk (G)PT, (O)penBSD area or (E)dit? [OpenBSD] w
Setting OpenBSD MBR partition to whole sd2...done.
The auto-allocated layout for sd2 is:
#                size           offset  fstype [fsize bsize   cpg]
  a:          1024.0M               64  4.2BSD   2048 16384     1 # /
  b:          1919.9M          2097216    swap
  c:         29184.0M                0  unused
  d:          1591.9M          6029088  4.2BSD   2048 16384     1 # /tmp
  e:          2471.8M          9289248  4.2BSD   2048 16384     1 # /var
  f:          3339.8M         14351488  4.2BSD   2048 16384     1 # /usr
  g:           936.0M         21191488  4.2BSD   2048 16384     1 # /usr/X11R6
  h:          3783.8M         23108320  4.2BSD   2048 16384     1 # /usr/local
  i:          1668.0M         30857472  4.2BSD   2048 16384     1 # /usr/src
  j:          5855.9M         34273472  4.2BSD   2048 16384     1 # /usr/obj
  k:          6589.5M         46266432  4.2BSD   2048 16384     1 # /home
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] a
/dev/rsd2a: 1024.0MB in 2097152 sectors of 512 bytes
6 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2k: 6589.5MB in 13495360 sectors of 512 bytes
33 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2d: 1591.9MB in 3260160 sectors of 512 bytes
8 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2f: 3339.8MB in 6840000 sectors of 512 bytes
17 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2g: 936.0MB in 1916832 sectors of 512 bytes
5 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2h: 3783.8MB in 7749152 sectors of 512 bytes
19 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2j: 5855.9MB in 11992960 sectors of 512 bytes
29 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2i: 1668.0MB in 3416000 sectors of 512 bytes
9 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd2e: 2471.8MB in 5062240 sectors of 512 bytes
13 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
Available disks are: sd0 sd1.
Which disk do you wish to initialize? (or 'done') [done]
/dev/sd2a (46c9b63f83d3fd95.a) on /mnt type ffs (rw, asynchronous, local)
/dev/sd2k (46c9b63f83d3fd95.k) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd2d (46c9b63f83d3fd95.d) on /mnt/tmp type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd2f (46c9b63f83d3fd95.f) on /mnt/usr type ffs (rw, asynchronous, local, nodev)
/dev/sd2g (46c9b63f83d3fd95.g) on /mnt/usr/X11R6 type ffs (rw, asynchronous, local, nodev)
/dev/sd2h (46c9b63f83d3fd95.h) on /mnt/usr/local type ffs (rw, asynchronous, local, nodev)
/dev/sd2j (46c9b63f83d3fd95.j) on /mnt/usr/obj type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd2i (46c9b63f83d3fd95.i) on /mnt/usr/src type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd2e (46c9b63f83d3fd95.e) on /mnt/var type ffs (rw, asynchronous, local, nodev, nosuid)

Let's install the sets!
Location of sets? (disk http nfs or 'done') [http] disk
Is the disk partition already mounted? [yes] no
Available disks are: sd0 sd1 sd2.
Which disk contains the install media? (or 'done') [sd0] sd1
  a:          1358848             1024  4.2BSD   2048 16384 16142
  i:              960               64   MS-DOS
Available sd1 partitions are: a i.
Which sd1 partition has the install sets? (or 'done') [a]
Pathname to the sets? (or 'done') [6.9/amd64]

Select sets by entering a set name, a file name pattern or 'all'. De-select
sets by prepending a '-', e.g.: '-game*'. Selected sets are labelled '[X]'.
    [X] bsd           [X] base69.tgz    [X] game69.tgz    [X] xfont69.tgz
    [X] bsd.mp        [X] comp69.tgz    [X] xbase69.tgz   [X] xserv69.tgz
    [X] bsd.rd        [X] man69.tgz     [X] xshare69.tgz
Set name(s)? (or 'abort' or 'done') [done] -x*
    [X] bsd           [X] base69.tgz    [X] game69.tgz    [ ] xfont69.tgz
    [X] bsd.mp        [X] comp69.tgz    [ ] xbase69.tgz   [ ] xserv69.tgz
    [X] bsd.rd        [X] man69.tgz     [ ] xshare69.tgz
Set name(s)? (or 'abort' or 'done') [done] -game*
    [X] bsd           [X] base69.tgz    [ ] game69.tgz    [ ] xfont69.tgz
    [X] bsd.mp        [X] comp69.tgz    [ ] xbase69.tgz   [ ] xserv69.tgz
    [X] bsd.rd        [X] man69.tgz     [ ] xshare69.tgz
Set name(s)? (or 'abort' or 'done') [done]
Directory does not contain SHA256.sig. Continue without verification? [no] yes
Installing bsd          100% |**************************| 20423 KB    00:01
Installing bsd.mp       100% |**************************| 20515 KB    00:01
Installing bsd.rd       100% |**************************|  4107 KB    00:00
Installing base69.tgz   100% |**************************|   291 MB    00:54
Extracting etc.tgz      100% |**************************|   254 KB    00:00
Installing comp69.tgz   100% |**************************| 85958 KB    00:26
Installing man69.tgz    100% |**************************|  7560 KB    00:06
Location of sets? (disk http nfs or 'done') [done]

What timezone are you in? ('?' for list) [Canada/Mountain] Israel
Saving configuration files... done.
Making all device nodes... done.
Multiprocessor machine; using bsd.mp instead of bsd.
Exit to (S)hell, (H)alt or (R)eboot? [reboot]

With this part done, I should be able to login as the root account over SSH (or over the serial console). Now I should bootstrap the instance in the following ways: setting up doas for the regular user, setting up the internet connection and adding the SSH public keys for the regular user. Then I can run Ansible and setup everything else.

For the internet connection, I don't have any nice way of doing it, so I just copy the connection details from the router role in the homelab repository (get the password from the Keepass password database with ph show --field Password 'Web Sites/Bezeq International').

Add the public SSH keys:

ssh 192.168.3.1 'mkdir -p .ssh; chmod 700 .ssh; touch .ssh/authorized_keys; chmod 600 .ssh/authorized_keys'
{ ssh-keygen -yf ~/.ssh/shore_ecdsa; ssh-keygen -yf ~/.ssh/shore_ed25519; } | ssh 192.168.3.1 'tee .ssh/authorized_keys'

Bootstrap the instance (in the homelab repository):

ansible-playbook bootstrap.yaml -l ns1 -u root -k -e 'ansible_host=192.168.3.1'

Setup the router (still in the homelab repository):

ansible-playbook router.yaml -e 'ansible_host-192.168.3.1'
ansible-playbook update.yaml -l ns1

Boom! Done.

Parsing Apache logs

2021-09-18T00:00:00+03:00

Due to a clerical error at my ISP, my internet connection is down for the weekend. So what's an activity that doesn't require a working internet connection? Blogging. Earlier this year I was interviewing and on 3 different occasions I was presented with a technical challenge of parsing Apache access logs. I know from experience that the goal of the challenge is to gauge how well I can parse text files with the standard Unix command line tools (mainly awk but also grep, wc, sed and some plain shell scripting).

Now awk is really a great tool and its performance is better than any Hadoop cluster (as long as you can fit the data in a single machine) and usually the challenge is limited enough that it's doable with awk. However, it may be easier to read and debug to write the solution using Python and obviously with Python being a much more flexible tool, we can solve any problem (even a real-life one) that has to do with parsing Apache's access logs.

For this I'm going to use Parse. With Parse we can write specifications (sort of the reverse of f-strings) and with them we can parse log lines and get back nicely structured data. Also, for bonus points, I'm going to use generators (it should also improve performance a bit).

from parse import compile

def parse_log(fh):
    """Reads the file handler line by line and returns a dictionary of the
    log fields.
    """
    parser = compile(
        '''{ip} {logname} {user} [{date:th}] "{request}" {status:d} {bytes} "{referer}" "{user_agent}"'''
    )
    for line in fh:
        result = parser.parse(line)
        if result is not None:
            yield result

This function will work with any file opened with open or with sys.stdin. Let's grab an example log file from Elastic's examples repo and print the 10 most frequent client IP addresses.

import urllib.request

ipaddresses = {}
with urllib.request.urlopen(
    "https://github.com/elastic/examples/blob/master/Common%20Data%20Formats/apache_logs/apache_logs?raw=true",
) as fh:
    for record in parse_log(fh):
        ip = record["ip"]
        if ip in ipaddresses:
            ipaddresses[ip] = ipaddresses[ip] + 1
        else:
            ipaddresses[ip] = 1
sorted_addresses = sorted(
    ipaddresses.items(),
    key=lambda x: x[1],
    reverse=True,
)
for i in range(10):
    print(f"{sorted_addresses[i][0]}: {sorted_addresses[i][1]}")

Obviously this is a simple example, but this method is not limited in any way. There's no messing around with delimiters or worrying about long strings inside quotation marks nor checking the awk man page for functions you never used before. The resulting code is pretty clear and the performance is on-par with any shell script you can whip together.

Terraform project structure

2021-09-14T00:00:00+03:00

Recently I've been using Terragrunt and I have thoughts on what it offers and is it useful. My usage has been in an existing project that follows the Gruntworks guidelines closely and with the paid subscription to the Gruntworks library. These opinions are my own and they're based on my recent experience with Terragrunt as well as managing small and medium infrastructure with Terraform for the last few years both as a single developer and part of small team.

The main point of Terragrunt as I understand it is keeping from repeating yourself in code. I am not a fan of copying and pasting big blocks of code nor of having to change the same value in a few different places. So for me keeping code DRY is a worthwhile endeavor.

Keeping modules DRY

Terragrunt works by using modules. I like Terraform modules. Even the Terraform documentation suggests that you don't have a single top level module for your entire infrastructure. It makes development more difficult with more merge conflicts. It makes deploying for testing purposes more difficult because Terraform will keep trying to delete resources that aren't in your code (because someone else working in a different branch has made changes for some other reason). You can work around that by specifying the target you're interested in but that is error-prone and is tiresome after a while.

In a previous project I worked on we had a module for roughly each service. We had quite a lot of code that was copied from one module to another (like when creating a new RDS instance we also created the subnet group, the security group for the client, etc.). Over time we saw clearly what code was shared between the different modules, we created a library directory and started adding sub-modules there and after a while we had a nice library of reusable sub-modules and things were good.

Because we waited a bit before creating a new sub-module they were pretty stable. When we did have a change to the a sub-module that we wanted to deploy across the entire infrastructure, we would open a branch, work on all the needed changes there, test them in one of the testing environments and then open a PR that has all of the changes (the sub-module changes, the calling modules changes, any fallout from those changes).

This process fitted us nicely. The PR had the entire picture and we could really see if the change improved anything (like adding an output to a module to be used in a different module would be clear if you see it being used). We did on occasion had conflicting changes and we did had to use targeted plan and apply but as far as I can remember less than once a quarter.

Terragrunt recommends splitting the repository in 2, one for sub-modules and one for actually deployed modules. Then you create terragrunt.hcl files that list the sub-modules needed with the Git ref used. This allows you to use the RDS database sub-module from today but the auto-scaling group from last year. I see little point in this.

The change process goes as follows, 1 PR for the sub-modules repository and 1 for the live repository (or more, we haven't gotten around to discussing environments yet) Now I hear that the recommendation has changed. The new recommendation is that each sub-module will be in a separate repository. So more PRs for each change (that one change of adding an output and using it became less obvious but requires more work, I wouldn't call it a win). I wonder if there's any place that has 2 repositories, 1 for code 1 for the tests and you change the code, and when it's merged you go to the tests repo and update the tests there to use the new code to see if it passes?

Another outcome from this way of working I keep seeing is that because changes are not applied (or planned) before merging the changes to the sub-module, errors and issues are only found out later which triggers more PRs.

Environments, remote states and workspaces, oh my

Another way that Terragrunt keeps your code DRY is by generating the Terraform backend configuration, because you can't use variables there with Terrafrom. So you save less than 10 lines. Cool. Also, you won't have by accident (because you copied that code from another module) used the same location for 2 modules and have them delete each others resources. It happened to me more than once, but you see it clearly when you first run terraform plan so it's very easy to catch.

Now, the folks at Gruntworks suggest you create a directory for each environment. From what I can see, that means you copy your terragrunt.hcl file to each directory and you modify it slightly (I think you can see where I'm going with this). If your project has a different module for each environment, this is a win. no doubt about it. I've seen projects like that and it's really a pain to manage.

Before I ever heard about Terragrunt, I had this exact problem. I solved it using Terraform workspaces and a simple convention. Each environment would have its own workspace (let's say that the default workspace is the sandbox but that's up to you). Each module would have a bunch of tfvars files for each environment. The workflow for deploying to the dev environment would look like this:

terraform workspace select dev
terraform plan -tfvars dev.tfvars -out tfplan
# Review the changes.
terraform apply tfplan

For making life a little easier I also added the following snippet to each module:

locals {
  module = "${basename(path.module)}"
  env    = "${terraform.workspace == "default" ? "sandbox" : terraform.workspace}"
}

Yes, this is copied code and along with the backend configuration, over 10 lines of code mostly that is mostly duplicated. However, when I compare it to the terragrunt.hcl files, this is peanuts. I checked a few modules in the codebase I'm working on and we have terragrunt.hcl files that are 100s of lines long and share all but a few lines.

I found that this convention is easy to document, easy for new developers to pick up, uses existing tools so you can use your existing knowledge and all of the benefits of avoiding to use another tool in your workflow.

Workflow

Terragrunt builds a directory for each module (and each environment obviously), clones the Git repos you mentioned with refs you specified and then mucks about with the Terraform commands and plan files to stich everything togethere. Even on paper this doesn't look like a good idea and it isn't one in practice, making debugging issues difficult.

It also suggests that you can have different versions of the sub-modules in use across different environments, putting emphais on having the main branch match exactly what is each environment instead of putting emphasis of avoiding drift between the different environments.

Conclusions

This post is a critique of the Gruntworks recommended setup and workflow and I think that if you read it all you would see that I think that there are better and easier ways. You can compare Terragrunt to a badly managed Terraform project and find that it helps you. But when you compare to it one that uses the suggested convention, it makes things more difficult, doesn't deliver on the promise of keeping your code DRY and promotes bad habits.

I didn't plan on reviewing Terragrunt until I used it. Terragrunt makes life less enjoyable. It has a convoluted workflow locally (with those bloody git clones), it makes debugging issues difficult and the upside is just not there. I would recommend to anyone who thinks about adopting Terrgrunt to first read the workspaces documentation before going with Terragrunt and think hard on the code review, the testing and development workflows.

Misc. projects and updates

2021-05-11T00:00:00+03:00

I have a few smaller projects that I never blogged about and now might be a good time.

Presentation toolkit

I like Markdown and Vim, I don't like PowerPoint. On enough occasions I was asked to present and I built a workflow to make my life easier. For that I have a container image with all of the tools that I require. There's an example presentation in that repo and there's also a GitLab CI template for those using GitLab and wish to integrate it in their pipeline. For me, creating presentations in Markdown and DOT is much faster and I find the results (especially when dealing with code) much nicer.

CloudWatch logs container image

When I started working with AWS ECS I had trouble debugging deployment issues. The logs in the ECS console were not complete and the container logs didn't hold the answers either. After a few times I had to login to EC2 instances and tail the ECS agent logs, I decided that sending those to CloudWatch logs is needed to better deal with such issues. I created another container image that has the CloudWatch plugin for the AWS CLI. With that you can send the ECS agent logs (or any other log file) to CloudWatch Logs. Deploying it as an ECS service (set the scheduling to daemon) and having it run on all of the ECS instances solved that issue for me.

Cron container image

Several services that I run need containerized Cron. However, regular Cron daemons aren't suited for running in a container (they remove environment variables before starting jobs, they log to syslog, etc.). Aptible created a Cron daemon suitable for containers named Supersonic, however there's ready container image so I created one. You can see it at in my GitLab instance.

The image can be used in 2 ways. You can it in a multi-stage build and copy the Supersonic binary from it.

FROM registry.shore.co.il/cron as supersonic

FROM alpine:latest
COPY --from=supersonic /usr/local/bin/supersonic /usr/local/bin/

Or you can build on top of it (ONBUILD instructions copy the crontab file and validate it).

FROM registry.shore.co.il/cron
COPY script /usr/local/bin/
USER root
RUN apk add --update --no-cache aws-cli
USER nobody

Lastly, updates

Subscribers to my blog might have noticed that I've been blogging rather profusely this last week. The reason is that I've finished the contract I had with my last employer. I've taken a short break and now I'm looking for work again. Right now I'm available for freelance work if something interesting comes up. Otherwise, I'm looking for long-term work. If anyone you know (or you are yourself) is looking for DevOps/ SRE/ infrastructure engineer with a preference for simple and effective tools and a strong tendency for open source software, drop me a message at nimrod@shore.co.il.

The reason for the excessive blogging (apart from the fact that I now have free time and I hope that someone somewhere finds my work interesting) is that I hope save time. I see that although almost all of what I wrote is publicly available in my GitLab instance, people apparently don't look too closely go just by my résumé. I hope that these last few blog posts make it more apparent who I am, what it is that I do and my approach to things. Thus saving both the interviewers and me some time.

Deploying a Hugo blog with gitreceive

2021-05-10T00:00:00+03:00

This blog post is actually one I meant to write a long time ago, but always put off. I'm going to describe how to use gitreceive to deploy a blog that's built using Hugo. I find the Heroku-style of deploying using git push nice in some cases and this is a nice example of using gitreceive as any. Obviously other static blogging tools can be used.

First, setup the server

We need to create a user on the server, grant it access to deploy the blog and add the SSH public keys.

sudo useradd -m hugo
sudo install -d -m 755 -o hugo -g hugo /var/www/html/blog
sudo install -d -m 700 -o hugo -g hugo ~hugo/.ssh
sudo install -m 600 -o hugo -g hugo /dev/null ~hugo/.ssh/authorized_keys
echo 'command="GITUSER=hugo gitreceive run %s %s" ssh-ed25519 AAAAC3....' | sudo tee ~hugo/.ssh/authorized_keys

Now we'll install git, gitreceive and Hugo.

sudo apt-get install -y git
cd /usr/local/bin
curl -L https://github.com/gohugoio/hugo/releases/download/v0.83.1/hugo_0.83.1_Linux-64bit.tar.gz | sudo tar -xz
sudo curl -L https://raw.github.com/progrium/gitreceive/master/gitreceive -O gitreceive
sudo chmod +x gitreceive

Create the receiver script with the following content at /home/hugo/receiver and mark it as executable:

#!/bin/sh
set -eu

repo="$1"
mkdir -p "/var/tmp/gitreceive/$repo"
echo '----> Unpacking'
tar -x
echo '----> Building blog'
hugo --cleanDestinationDir
echo '----> Deploying blog'
rm -rf /var/www/html/blog/*
cp -r public/* /var/www/html/blog/
echo '----> Done'

Second, the client side

Actually, this is quite simple, just adding a git remote.

git remote add deploy hugo@example.com:blog

And now let's give it a test.

$ git push deploy
Counting objects: 39, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (31/31), done.
Writing objects: 100% (39/39), 6.55 KiB | 2.18 MiB/s, done.
Total 39 (delta 9), reused 0 (delta 0)
----> Unpacking
----> Building blog
Start building sites …

                   | EN
-------------------+-----
  Pages            |  3
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  1
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Total in 10 ms
----> Deploying blog
----> Done
To blog:blog
 * [new branch]      master -> master

That's it, the blog is deployed. Obviously having the output from the script is useful. We can change the receiver script to do a lot of other things. Running docker-compose build && docker-compose pull && docker-compose up -d can produce a simple and straightforward dev environment. We can make it more general by running a script inside the repo and build a makeshift CI tool. Lastly, I have an Ansible role to do all of the server configuration for you in my GitLab instance.

Spam me - Update #2

2021-05-10T00:00:00+03:00

Previously I wrote about the spam me page and later an update on how things were going.

Things were stable and I didn't get any spam. But something bothered me. I would only get notifications on my laptop and only when it was on. Notifications would be lost if the laptop was offline. Also, Patchbay had downtime a few months back and I didn't get notifications at all. Lastly, I'm keen on self-hosting.

Nextcloud Notifier

Nextcloud has a notification mechanism that is used internally. I get notifications on pending updates on my cellphone and laptop. Pushing notifications over HTTP requires authentication, but using the occ CLI doesn't require authentication.

I wrote a simple web service to post messages through the Nextcloud notification mechanism. The service runs the occ command using docker exec so usage is anonymous. Notifications are received on both cellphone and laptop and I'm content. Now you can send me messages and I'll always get it.

Future uses

Now that I have a notification service, I saw more uses than I originally envisioned. I send notification on failed Cron jobs, for example:

backup || wget --spider https://notify.shore.co.il/send?message=Backup%20failed.

I have CI jobs that run on a schedule (rebuilding container images) so now I have a GitLab CI template that sends me a message if the CI pipeline failed.

Collecting metrics

2021-05-08T00:00:00+03:00

A few startups I worked at had a similar story. When they got started they didn't have any metric collection (maybe some system metric from their cloud provider, but nothing more). After a few times where they had to debug an issue where metrics were needed they decided to start collecting metrics from the application. Since they were a small team with little experience in setting up the needed infrastructure or the man power to handle such a task they decided to use a SaaS product (NewRelic and Datadog are both good choices here). Then as the company grew and the number of users, processes, components and instances grew so did the bill from the that SaaS. This is usually the time where they decide that they need an DevOps person on the team (not just because of the metrics issue, but as the company matures, the customer base grows, uptime requirements increase, scaling is an issue, etc.). What follows is my advice on such undertakings.

First of all, use StatsD. Not specifically the StatsD daemon but the protocol. It's mature, flexible enough for most tasks, supported in all languages and in all metrics collection software. You can even use netcat to push metrics from shell scripts.

You don't have to setup storage for the metrics you collect, CloudWatch, Librato or similar services can be used and are cheaper than the more integrated SaaS offering. If you have Elasticsearch already for log collection, you can use that for storing metrics as well (great for a few dozen instances). InfluxDB and Gnocchi support the StatsD protocol. As you can see, it's a flexible solution.

I think that most people associate the StatsD protocol with the Graphite protocol and software, but there is a key difference: StatsD uses UDP. It's faster and there's less overhead. You can default to sending the metrics to localhost and if nothing collects them that's still fine (great for local development and CI jobs where you don't want to run metrics collection software or where it doesn't make sense). I know that some would say that TCP is more reliable than UDP and you can lose metrics using it. To that I would say that the lower overhead of UDP, the lack of connection is actually an advantage. No connection closed errors, that single packet is more likely to reach its destination in case the system is under heavy load than opening a new TCP connection. You can read this GitHub blog post to see how they dealt with their metric loss.

I know that Prometheus is the new hotness and very popular when running Kubernetes and it's not a bad choice. But there are a few things that you need to keep in mind when considering it. It's best when you have service discovery available for it (if you're already using Kubernetes or running in a cloud provider that's not an issue but that's not everybody). Also, for short lived tasks (like Cron or processes that pull from a job queue) or for processes that can't open a listening socket (or if you have many of the same processes running on the same host), you need to setup a push gateway. The process pushes metrics to the gateway and Prometheus collects them afterwards (sounds an awful like the StatsD daemon, doesn't it). Prometheus has a StatsD exporter so you can still use StatsD along with Prometheus.

If you want some ad-hoc or more lightweight metrics collection, statsd-vis is a good solution. It holds the data in memory for a configurable time and has a builtin web UI to watch the graphs. I have a a very small Docker image for that.

Lastly, for Python applications I recommend the Markus library. It's easy to use, reliable and you add more metrics as needed. 2 thumbs up.

Database resource handling

2021-05-08T00:00:00+03:00

A few years ago I was working at a company where part of the product was an online encyclopedia. The content was curated by a team in the company using an homegrown CMS. Deployments of it were done by dumping the content of a few database tables, zipping them up, uploading to S3, and in the production database, dropping those tables and restoring from content of the zip file in S3.

This was a Laravel app and deployments were done using Ansible. We had Ansible code for that, but the database dump and uploading to S3 was manual and the configuration was in the repo where the Ansible roles were maintained, not the repo were the app was developed.

I know most people would jump up and say that this causes downtime that can be avoided, but this was the way it was done when I joined. It avoided the need to care about database migrations for those tables. And given that this company was based in Israel and we work on Sundays, nobody cared that the entire product (not just the encyclopedia) was down for ~10 minutes on Sundays. As a side note, you'd be surprised at how many Israeli startups do weekly deployments on Sundays.

This caused a few issues for the company. First of all, developers had individual instances on Azure that were initially set up by the Ops team. At the time we had Azure credits but those were going to run out in a few months and we wanted to move developers to use their local machines entirely.

Another thing was that the process of updating the resources was involved and required somebody from the Ops team. It meant that once a breaking change was merged to the master branch of the application, tests would break for all developers until the database resources were updated.

Yet another downside was automated testing. Previous attempts at having a CI pipeline required sharing the database between all of the jobs. It needed to be maintained by the Ops team. And it caused to jobs to fail if a breaking change was deployed.

The solution I came up with was extending the Laravel app to do most of the work. First I added a new config array for the version of the resources we currently want. IIRC we decided on the format of year and week number. Then I wrote a new Artisan command to download the the zip file, drop the tables and restore from the zip file. Laravel offers a way to save what it calls system variables in the database, so the currently deployed version was stored in the database and I could skip this process entirely if the currently version and wanted version match making the process usually faster and idempotent.

With the command in place, it was added to to all of the relevant places. Developers had a short script that they maintained which ran all of the tasks that were needed for updating the app (like database migrations, SCSS builds, Gulp builds) so the command was added to the script. It replaced a few Ansible tasks with a single task to run that command.

The developers maintaining the CMS added the functionality to upload a database dump of the those tables and exposed it in the admin UI and the loop was complete.

The benefit of this approach was the developer workflow. Instead of working with the Ops team to update the resources, they would just update the config to point to the new version. It allowed downgrades in cases we wanted to revert to an older version. It made the PRs much clearer to everybody. The Dev team was self-sufficient, the Ops team had one thing less to worry about. We were one step closer to having developers develop using their local machines (using Docker and Docker Compose) instead of those instances in Azure. We were using Bitbucket and they just came out with the open beta for Bitbucket Pipelines and I was able to set up automated testing that was completely independent for each run. Overall, a lot upsides and time saved for about 200 lines of PHP code (it took me around 2 days since it was the first time I wrote more than 2 lines of PHP and the code review had a lot of issues for me to address).

Saving sent messages with Exim

2021-05-08T00:00:00+03:00

In shore.co.il I use the Exim mail transfer agent (SMTP server) along with the Dovecot IMAP server. Messages received in Exim are sent to Dovecot over LMTP. So far, pretty standard stuff. All of the configuration and setup can be seen in my GitLab instance.

I wanted to save sent messages to a folder (like GMail does) but didn't find a way to do that over LMTP documented anywhere so here's what I did. I created a router in Exim that if the sender is one of the local domains (domains Exim receives messages for), then they're redirected to the LMTP transport but with +Sent suffix attached so they're saved to the Sent folder. You can see the configuration here.

Accessing the GitLab remote Terraform state from Ansible

2021-05-08T00:00:00+03:00

Note to self: How to get the outputs from a GitLab remote Terrafrom state from Ansible.

---
- name: Get Terraform outputs from the GitLab remote state
  hosts:
    - localhost
  connection: local
  become: false
  gather_facts: false
  vars:
    project_name: amilive
    tf_workspace: "{{ lookup('env', 'TF_WORKSPACE')|default('default', true) }}"
    gitlab_token: "{{ lookup('env', 'GITLAB_TOKEN') }}"
    gitlab_base_url: "{{ lookup('env', 'GITLAB_BASE_URL') }}"

  tasks:
    - name: Projects
      ansible.builtin.uri:
        headers:
          PRIVATE-TOKEN: "{{ gitlab_token }}"
        method: GET
        url: "{{ gitlab_base_url }}/projects"
      register: projects

    - name: Request
      ansible.builtin.uri:
        headers:
          PRIVATE-TOKEN: "{{ gitlab_token }}"
        method: GET
        return_content: true
        status_code: [200]
        url: "{{ gitlab_base_url }}/projects/{{ project_id }}/terraform/state/{{ tf_workspace }}"
      vars:
        project_id: "{{ (projects.json|selectattr('path', 'equalto', project_name))[0].id }}"
      register: tf_state

    - name: Env output
      debug:
        var: (tf_state.content|from_json)["outputs"]["env"]["value"]

shore.co.il infrastructure - May 2021 edition

2021-05-08T00:00:00+03:00

Hardware

The hardware I'm using consists of:

Netgate SG-2440 running OpenBSD.
Linksys EA6350 running OpenWrt.
ASrock N3150-NUC running Debian.
An online.net (now Scaleway) Dedibox running Debian.
An ADSL modem provided by my ISP.
A purpose built PC in the living room running Debian.
APC UPS (I don't remember the exact model I don't feel like getting and checking).

The OpenBSD box is ns1.shore.co.il. It's the router for the home network, the primary DNS server for the shore.co.il zone, DNS resolver for the network, DHCP server and running HAProxy. The local Debian box has a 0.5TB drive and is the LDAP server, mail server, Nextcloud, GitLab. I store all of my private information on it encrypted. I have off-site backups I take about every 2 weeks that I store in my mom's house (also encrypted). The living room PC runs Kodi, Transmission and my podcast downloader. It has a magnetic 8TB drive that holds music, movies and other such media. The OpenWrt box is the wireless access point. Lastly, the Dedibox is a bare metal instance which obviously has a faster internet connection than I have locally. It runs this blog, a few other web sites, the container registry, the secondary DNS server for shore.co.il and most GitLab CI jobs run on it. Also, it runs my workbench (a container that has all of the tools I need and use) that benefits from the faster internet connection, faster drives, abundant memory and beefy CPU. The data drive is also encrypted, but not backed up (everything on it can be recreated in less than a day).

Deployments

Initial setup is done using Ansible. Services on the different Debian boxes run in Docker containers and deployed using GitLab. The other OSes are maintained using just Ansible. There's no redundancy since it would take more money than I would like to spend. There's no infrastructure-as-code (no Terraform) since the only thing I could code is the single Dedibox instance (and it's not supported by the Scaleway provider last time I checked). All of the Debian instances run a GitLab Runner in a Docker container and have access to the dockerd socket so they can create containers, run jobs in them and build and push images. I know this is a security risk, but since only I use this GitLab instance I'm worried about it. The deployments are pretty consistent, projects have a docker-compose.yaml file, the GitLab runner runs docker-compose build, docker-compose pull and docker-compose up to deploy services. All of the code is in the Shore group in my GitLab instance. The templates for the CI pipelines that I use are also in my GitLab instance.

Security

Most services are only available over SSL (apart from services that don't support it like DNS, DHCP, etc.). I'm using Let's Encrypt to issue globally valid certificates. In my home network this has presented a problem. I wanted to have multiple hosts using SSL with the single IP address and not rely on the router to decrypt the traffic. I wanted the traffic to remain encrypted until it reaches the host and that the certificate to globally valid. For that I run HAProxy which uses SNI to identify the requested host and forwards the traffic accordingly.

The SSH servers all have rate limits and only allow public key authentication. I rely on SSH keys to authenticate and login instead of LDAP, I prefer it since if the LDAP server is down I'm not locked out entirely. Further more the only services that use the LDAP server are on the same host and connect to it via a Unix socket, further securing the access. The LDAP server is not available on the network.

For regular tasks like renewing the SSL certificates or updating the hosts I have written Ansible playbooks. I routinely rotate the SSL keys and also the DH parameters. I run them manually from my laptop, I don't want them to be updated automatically by the hosts themselves. Also, rebooting the NUC and Dedibox requires a manual step to unlock the encrypted drives. I can update all of the hosts, rebuild all of the container images and deploy in about 2 hours and with very little interaction and I do so every few weeks.

Changes from the previous iteration

There are a few changes in the infrastructure sine the last versions. First, instead of the Dedibox I used an EC2 instance. The Dedibox costs more, but the time saving from the beefier instance, from the CI improvements and being able to run VMs on it are worth it.

I used to use Ansible for everything, including deploying services which I do now with Docker Compose and GitLab runners. The development workflow with Docker is easier and faster than using Ansible along with Vagrant and Molecule. I can honestly say that I'm surprised more people don't this more often. It's really easy, simple, secure and reliable. I find that this approach is useful for simple setups like mine (or dev or QA environments), especially since the same Docker Compose setup can be used for local development.

The addition of the Nextcloud and GitLab services have made me entirely self-hosted. I use online.net and a DNS registrar but apart from that I don't rely on any external service and I hold all of my private data (except that most of the emails I send end up in Google's servers anyway). My sites are indexed by Google, Bing and Yandex but I'm not sure what can I do about that.

Future improvements

The UPS isn't supported by NUT or anything else available on Linux or OpenBSD. I will replace it sometime in the future with one that does so that I can trigger a clean shutdown when there's a power outage and the battery is running low. Also, I would like to replace the NUC with a newer and faster one.

I have external monitoring on services but I've yet to setup internal log aggregation or metrics collection. I plan on setting up an EFK stack (I have some POC code laying around but I need to update it and bring it in line with the rest of the infrastructure). I also want to investigate Sensu for running checks locally (a Nagios replacement), I have my eye on Testinfra for the host checks.

I'm using Z-Push along with Nextcloud for Activesync but it doesn't work with my phone so I want to evaluate SOGo as a replacement. I want to try the Dropbear-initramfs integration so I can unlock encrypted drives remotely over SSH. I want to replace the workbench using Docker with the toolbox project.

I avoided using a VPN for now and I don't want to go down that route. But I've been in very closed networks so I want to setup a Websocket proxy to my SSH server on the Dedibox so I can connect over port 443 and tunnel out from such networks.

Lastly, I see XMPP, Matrix or Mastodon (or maybe 2 of them) in the future for secure and self-hosted chatting with friends.

LDAP authentication for web services

2021-05-08T00:00:00+03:00

Some web services I run don't offer integration with LDAP for authentication. One possible way to have authentication is to use the Vouch proxy. I used it along with Nextcloud (which has integration with LDAP) providing OAuth. But I encountered a limitation to this approach. Some clients only support basic authentication and don't support the newer JWT tokens and OAuth flows (clients for the Transmission torrent clients are an example for that). I didn't want to deal with secret management or with .htaccess files. I wanted users to be able to authenticate using their LDAP password.

First attempt was using the LDAP authnz module for Apache. But either I didn't set it up correctly or that connecting to the LDAP server over a Unix socket doesn't work as expected. Anyway, authentication always succeeded when using the Unix socket and I didn't want to change the LDAP setup I have (I prefer using the Unix socket with containers as I can easily limit which containers have access to the LDAP server by cross-mounting the socket only to containers I want to have access).

I ended up creating a small service in Python with Flask and Flask-SimpleLDAP. The service exposes just a single endpoint /validate which returns a 200 code when basic authentication succeeds or a 401 code when it fails. Authentication uses the LDAP server over the Unix socket as I wanted. It can be easily integrated with Nginx using the auth_request directive. An example can be seen here. The entire service is available on my GitLab instance. There's even a Docker image you can use in my container registry.

For now I'm using a fork of Flask-SimpleLDAP (until my PR for adding support for accessing the LDAP server over a Unix socket is merged).

Monitoring shore.co.il

2021-05-08T00:00:00+03:00

Recently, I had some time to work on a project I had on my to-do list for a long time, monitoring services in shore.co.il. The project is now done and is available in my GitLab instance.

Requirements

When I write monitoring, I mean periodic checks on services and alerts if they fail. I had a specific requirement set in mind with this project. I wanted the monitoring to be reliable, meaning that if anything and everything in my infrastructure failed, I would still get alerts. This was critical for me since I run a lot of my infrastructure at home and a prolonged internet or power outage would bring down many services. Cheap and easy would also be nice.

Architecture

I decided on using Lambda functions along with SMS notifications from SNS on AWS. Lambda functions can be reliably triggered using CloudWatch Events on a schedule (every x minutes) and failures can be published to a SNS topic that has a target that sends SMS messages to my cellphone. So far, very reliable, no dependency on anything in my infrastructure. For added reliability, I added CloudWatch alerts in case a function failed to be invoked recently or if the invocation failed. Said alerts would also send me an SMS message. SMS messages cost a little (hopefully there would little of those), I don't have enough Lambda function invocation or runtime to go over the free tier and the price for the code in S3 isn't great either. For me, it was easier, cheaper and more reliable than setting up Nagios, Sensu or similar.

Solution

I wrote a few Python functions to test the different services I run (DNS, SMTP, IMAP, SSH, different web services). To deploy them I wrote a Terraform module that does everything from creating the SNS topic, upload the Python code and hook up the Lambda functions. Everything is ran inside a GitLab CI pipeline and uses the GitLab remote Terraform state (I recently had reason to try it out and I was impressed).

Conclusions

I don't think I would set up this specific solution for a company. A company would most likely have an on-call schedule. Maybe using a SaaS product would be easier and better in some aspects (like running checks from multiple locations). But for my small infrastructure and considerations it was a success. The project can be adapted to use a service like PagerDuty to have an on-call schedule and it can be deployed to multiple regions to run checks from multiple regions. Lastly, Nagios and Sensu have a library of ready checks in Ruby or Perl so you don't have to write them yourself. This project has been live for more than a week now and has been reliable. The AWS Cost Explorer predicts that the cost for this month would be a few dollars. I call it a success.

Human and structured logs

2021-05-08T00:00:00+03:00

As anyone who's heard of the 12 factor app knows, logs are best outputted to the standard output of the process and have the environment handle the logs accordingly. A while back I worked at a company where we collected logs with the EFK stack. We had quite a bit of logs and ingested daily and tracking down issues in Kibana using regular expressions became unwieldy. Structured logging (read json lines, where every line is a json object) was the answer. It would allow us to more easily find all of the logs for a single transaction or account and our EFK stack could parse out-of-the-box.

There was a previous attempt at structured logs before I joined the team that failed to be merged. It was too invasive, the diff too big, it never worked correctly and it required too much work to carry the diff forward and eventually the branch was dropped.

Another complaint I heard from the developers about that previous attempt was that it made reading the logs when developing locally (reading the logs from the Docker container) or for failing CI builds more difficult. Indeed, it was not human-readable.

Others at the company (with experience log4j and enterprise software) wanted to output all of the logs to 2 files with different formats. I saw this as a mistake. It would make handling logs inside of containers more difficult. It would require dealing with log rotation.

Another goal for me was to be able gradually convert components to the new log handling so we won't end up with a huge diff that's never merged again.

I went with the structlog package and after delving in to it, I created a module that only requires importing to use it. The module configures the log format (human or json) and level according to environment variables but it doesn't require any change to the logging calls. The structlog package also comes with a great looking console renderer where ever control character are valid. Here's how it looks:

And here's the module:

import logging
import logging.config
import structlog
import structlog.stdlib
import structlog.processors
import structlog.dev
import os

emptyLogRecord = logging.makeLogRecord({})


def add_extra_params(logger, log_method, event_dict):
    record = event_dict['_record']
    for (k, v) in list(record.__dict__.items()):
        if k != 'stack_info' and k not in emptyLogRecord.__dict__ and k not in event_dict:  # noqa: E501
            event_dict[k] = v
    return event_dict


pre_chain = [
    structlog.processors.StackInfoRenderer(),
    structlog.stdlib.PositionalArgumentsFormatter(),
    structlog.stdlib.add_log_level,
    structlog.stdlib.add_logger_name,
    structlog.processors.TimeStamper(fmt='%Y-%m-%d %H:%M.%S'),
    structlog.processors.format_exc_info,
    add_extra_params,
]

processor = {
    'human': structlog.dev.ConsoleRenderer(),
    'json': structlog.processors.JSONRenderer(),
}

logFormat = os.environ.get('LOG_FORMAT', 'human')
assert logFormat in processor

try:
    logLevel = getattr(logging, os.environ.get('LOG_LEVEL', 'WARN').upper())
except AttributeError:
    logLevel = logging.WARN  # The default log level.

config = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'json': {
            '()': structlog.stdlib.ProcessorFormatter,
            'processor': processor['json'],
            'foreign_pre_chain': pre_chain,
            'format': '%(message)s',
        },
        'human': {
            '()': structlog.stdlib.ProcessorFormatter,
            'processor': processor['human'],
            'foreign_pre_chain': pre_chain,
        },
    },
    'handlers': {
        'default': {
            'formatter': logFormat,
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        '': {
            'handlers': ['default'],
            'level': logLevel,
            'propegate': True,
        },
    },
}

logging.config.dictConfig(config)

structlog.configure(
    processors=[processor[logFormat]],
    context_class=dict,
    logger_factory=structlog.stdlib.LoggerFactory(), )

I called this module logging_config and if a module had imported it, then logs would be human readable by default and if the LOG_FORMAT environment variable was set to json (it would be in the staging and production environment where we had an EFK stack running and collecting logs) the output was in json format.

This was the first step and it could be merged without breaking existing components. However, most logging calls still had formatted strings which didn't expose all of the data in a structured format. We had to go module by module and change to have a single, static log message and pass all of the variables in the extra parameter. It would mean changing calls that looked like this:

logger.error(f"Failed to send message {message} to user {uid}.")

to something like this:

logger.error("Failed to send message.", extra={"message": message, "uid": uid})

We could add fields to the output easily, deal with missing parameters more easily, output the Flask request context more easily, etc. We could do this piecemeal, going through one module at a time and not have to keep a growing diff maintained. It would keep the human readable logs where needed and have easily searchable logs in Kibana where available. We kept to the 12 factor app methodology and the benefits of it. It required little change to our setup and deployment. It had noticeable benefits out of gate which made the Dev team pitch in with converting modules to the new logging calls convention.

Spam me - Update

2020-11-20T00:00:00+02:00

In last month' post I wrote about the new direct messaging I setup and the concern about abuse from that (spam and privacy concerns). While I can't speak on the privacy part (I don't know if anybody is listening to that channel besides me), I can update on the spam part. Since going live I received exactly 0 unsolicited messages. Nada, zilch, bupkis, גורנישט. I'm a little disappointed with that since it shows that my blog is not read by millions of people (a shock, I know). But it also seems that automated scanners, scanning repos in Github don't act on this kind of information. With that, I plan on keeping everything running for the foreseeable future.

Spam me

2020-10-24T00:00:00+03:00

A while back I saw an interesting project, Patchbay. At first I wanted to use it when I run long tasks on remote machines (as the example shows). I would obviously script the desktop part, commit it to my rcfiles repo and have it run on startup. As a security/ privacy concern, I planned on keeping the full URL private. So I shelved it until I would have a proper secret management system in place for such things.

A few months went by and I remembered that project and started to play around with receiving such messages but sending them from a webpage. The outcome is shore.co.il/spam. I'm announcing this on my blog as I'm actually interested to see if I get any spam this way. The desktop side of things is in this rcfiles commit and the source for web page is in my blog commit, both are quite public.

There isn't something technically interesting here (apart from Patchbay). But the experiment aspect is interesting to me. I would like to see who reads my blog and will send me messages (hopefully interesting ones). I'm not going to advertise this in any other way. And I would like to see if I get any spam as a result of this blog entry or from having the URL public in my Git repos. I'll post an update in a few weeks with initial results.

Docker socket over SSH

2018-01-09T00:00:00+02:00

Yesterday I described how to connect to a remote dockerd over TCP. I didn't touch security considerations at all (firewall, TLS certificate). This because, for my use, I prefer a different method, forwarding the Unix socket over SSH. Here's how.

First, you need OpenSSH version 6.7 or later (both client and server). Also, the login user on the remote instance must have permissions to access the Docker socket (in other words, be a member of the docker group).

Here's how to forward the remote socket:

ssh -fNTo ExitOnForwardFailure=yes -o ServerAliveInterval=30 -L $HOME/.ssh/docker.sock:/var/run/docker.sock host
export DOCKER_HOST=$HOME/.ssh/docker.sock

And to close the connection and return to the local dockerd kill the ssh process that's running in the background, rm the docker socket under $HOME/.ssh and unset DOCKER_HOST.

The reason I prefer this method is that it's easier to setup for ad-hoc tasks and arguably more secure since you not only authenticate the user and host with SSH, but you limit access to only those that are part of the docker group.

Bind dockerd to a TCP port

2018-01-08T00:00:00+02:00

On a modern system (one running Systemd) when installing Docker, the dockerd daemon is run using Systemd' socket activation. By default the socket is /var/run/docker.sock. If you want to connect to a remote machine over TCP, the obvious thing to do is to create /etc/docker/daemon.json and set the hosts list there. But that will conflict with the command line flags for socket activation. The correct way is to override Systemd' socket activation config. Here's how (all command are as root):

mkdir -p /etc/systemd/system/docker.socket.d
echo '[Socket]' > /etc/systemd/system/docker.socket.d/tcp.conf
echo 'ListenStream=2375' >> /etc/systemd/system/docker.socket.d/tcp.conf
systemctl daemon-reload
systemctl restart docker

Bundling a binary file into a shell script

2017-12-06T00:00:00+02:00

When creating an auto-scaling group in EC2 I often try to package the deployment script into the user data. Installing some packaged software is easy to do but bundling configuration files that are needed is less straightforward. If the files are not confidential in any way, I either clone a Git repository or download a tarball from our static assets domain. But this leads to a dependency on external services and a slightly more complex deployment procedure. A few days ago I was faced with the same options again but it didn't sit right with me to do all this for a couple of files that are a few K's in size totally. I remembered that some software have installation scripts that bundle the binary blob inside the script.

First version

I searched and found an article in the Linux Journal that seemed to show what I wanted to (and seems to be copied everywhere). You could download a single file that was a shell script with the binary blob inside. Your usage will be close to this

wget http://hostname.tld/bundle
sh bundle

or this

wget http://hostname.tld/bundle
chmod +x bundle
./bundle

Which is fine. However the code was a bit longer than it should have been and I felt it could be done better. A little more research and I found an answer in Stack Overflow that mentioned uuencode and uudecode. Reading the man page I saw it was closer to what I wanted. The code I wrote is available on my GitLab instance.

The implementation works as follows. The bundle has the script at the start of the file with the encoded binary at the end. The shell executes the script part (which ends with exit as to not continue any further, causing errors) and uudecode only starts processing after it sees the relevant header. The script feeds itself to uudecode (uudecode "$0") which decodes the binary and outputs it to disk which the script can then use. The code has both the build instruction in the Makefile and usage example in the bats tests.

Second version

However something kept nagging me. I wanted a simple invocation method like so:

curl http://hostname.tld/bundle | sh

And in the case of the user data in EC2, I could simply use the bundle. Otherwise I would need to host it somewhere and in the user data I would download and run the bundle. Which means that if the bundle was unavailable the instance would fail to provision.

Everything I found assumed that the file was present in the file system for uudecode to decode. If it was piped there was no file that uudecode could then decode. I kept mauling over it and a came up with a short, clean solution to this problem, which is available here, again with build instruction and test examples.

This time I used AWK to replace a single line in the script with the file, encoded using uuencode but this time in base64 (to keep the script valid without any characters with special meanings). That is piped to uudecode which decodes and saves it to disk. The script can then continue with the binary blob present.

This method is less space efficient and the build procedure is less obvious. But the ability to use resulting script as the user data (or piping the output from curl to sh) is worth it in my opinion.

Building inside a Docker container with the correct user

2017-11-26T00:00:00+02:00

Lately I've been using Docker container as clean, easily portable and easily removable build environments. In those cases the image contains the needed build tools and the project is mounted to a volume inside the container. The artifacts are then built inside the container but are placed inside the volume. However a small problem arises, the artifacts (and whatever other files are created, like cache) are owned by the default user, root, making editing or removing said files less straightforward.

The trivial solution

The trivial solution is to run the container with the correct user id, like so

uid="$(id -u)"
gid="$(id -g)"
docker run -v "$PWD:/volume" --user "$uid:$gid" buildimage make

I personally find it a tiresome after the 3rd time I had to sudo chown the project because I forgot to specify the uid and gid and it's a (low) barrier of entry for new users.

A better solution

The solution I've come up with is this small script that sets the uid and gid values to those of the owner and group for the volume and then execute the commands.

#!/bin/sh
set -eu
[ "$(id -u)" = "0" ] || { echo "Not running as root, continuing as the current user."; eval exec "$@"; }
command -v stat > /dev/null || { echo "Can't find stat, exiting."; exit 1; }
command -v gosu > /dev/null || { echo "Can't find gosu, exiting."; exit 1; }
uid="$(stat . -c '%u')"
gid="$(stat . -c '%g')"
eval exec gosu "$uid:$gid" "$@"

The script is also available for download. The only dependency is gosu. You can download and check it to your VCS and incorporate it into your Dockerfile, or download it via the ADD directive, like so:

FROM buildpack-deps
RUN curl -fsSL https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64 -o gosu-amd64 && \
    install -o root -g root -m 755 gosu-amd64 /usr/local/bin/gosu && \
    rm gosu-amd64 && \
    curl -fsSL https://www.shore.co.il/blog/static/runas -o runas && \
    install -o root -g root -m 755 runas /entrypoint && \
    rm runas
ENTRYPOINT [ "/entrypoint" ]
VOLUME /volume
WORKDIR /volume
ENV HOME /volume

Setting the home directory to the mounted volume will result in some files (like package managers cache) to be created there, which you may or may not want. And then finally, to build run

docker run -v "$PWD:/volume" buildimage make

How to fix the PocketCHIP for Vim

2017-04-22T00:00:00+03:00

I ordered me a PocketCHIp to have a cheap, portable Linux computer with a physical keyboard with the added benefit if it running Debian out of the box. I quickly discovered that in Vim the dash or minus key is not what you'd expect. A quick search turned up that the key is mapped the numpad minus via xmopmap. To change the mapping to what is for me a better setting run sed -i 's/KP_Subtract/minus/g' ~/.Xmodmap and to apply the setting afterward run xmodmap ~/.Xmodmap.

VirtualBox extensions

2017-01-17T00:00:00+02:00

I happened to run into a Vagrant image today that required the USB2 controller extension to VirtualBox. The procedure to install the extension pack was not immediately clear to me.

Installing through the VirtualBox GUI

When trying to go through the VirtualBox GUI, the application would close when I clicked on 'add new package' in the preferences window. Guessing it had something to do with root privileges I tried to run the GUI with sudo but that failed because I'm using Wayland and it could not find display :0. Using Gnome3, gksudo is no longer available and the replacement is pkexec which uses PolicyKit. running pkexec VirtualBox opened the nice Gnome3 authentication prompt but resulted in the same error. Giving up on starting the VirtualBox GUI as root, I moved my efforts to the VBoxManage CLI.

Installing with the VBoxManage CLI

Here are the steps I took to successfully install the extension pack:

Find the version of VirtualBox you have installed by running VBoxManage --version.
Download the matching version of the extension pack from https://www.virtualbox.org/wiki/Downloads if you're using the latest version or from https://www.virtualbox.org/wiki/Download_Old_Builds if you're using an older version.
Assuming you're using version 5.1.10, install the extension pack by running VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.1.10-112026.vbox-extpack. For me, this opened the same nice Gnome3 authentication prompt.

Testing DNS with a clean cache

2016-11-01T00:00:00+02:00

Every so often I make changes to a DNS record, test it, find out it's wrong, fix it and still get the old response because of caching somewhere along the line. After it happened to me and a colleague during a launch of a new version of a website, I decided to address the issue. I wanted a way to test DNS quickly and easily (preferably locally on command line), for it to be lightweight, doesn't require changes to my existing setup and doesn't require learning new tools. I decided to create a Docker image that has its own DNS resolver and each new container from that image has a clean cache and doesn't depend on other DNS servers or is affected from their caching.

Usage

To create a new container:

docker run -it registry.shore.coil/resolver

Inside the container you have access to nslookup, dig and mail for testing purposes. If you need to test new changes, exit the container and create a new one with no cache.

If you want to run just a single command (like getting the MX record for shore.co.il):

docker run registry.shore.co.il/resolver dig +short shore.co.il mx

How does it work

On launch, the container runs and uses its own DNS resolver (in this case NSD). This way the OS caching or upstream caching don't interferes with querying and every new container starts with an empty cache.

Self service AWS IAM policy

2016-09-01T00:00:00+03:00

A common practice for me when a new member joins the team or when someone forgets his/ her AWS account password is to change the account password myself, send the new password over an insecure channel (email, Slack) but force the account to change the password on first login. Also, I prefer to have users manage their own keys to AWS themselves. But without the correct IAM policy users aren't able to perform either action. Here's an IAM to allow both:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetAccountPasswordPolicy",
                "iam:ListAccount*",
                "iam:GetAccountSummary",
                "iam:GetAccountPasswordPolicy",
                "iam:ListUsers"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:ChangePassword",
                "iam:*LoginProfile",
                "iam:*AccessKey*",
                "iam:*SSHPublicKey*"
            ],
            "Resource": "arn:aws:iam:::user/${aws:username}"
        }
    ]
}

If you want a little script with the AWS CLI, here's one for you:

tempfile=$(mktemp)
accountid="$(aws ec2 describe-security-groups \
    --group-names 'Default' \
    --query 'SecurityGroups[0].OwnerId' \
    --output text)"
curl https://www.shore.co.il/blog/static/policy.json | sed "s//$accountid/" > $tempfile
aws iam create-policy \
    --policy-name change-own-password \
    --policy-document file://$tempfile
rm $tempfile

Ad-hoc serving of git repositories

2016-08-16T00:00:00+03:00

On some occasion you want to serve your git repo from your local copy (perhaps your git repository is quite large and your internet connection is slow or your build process would benefit from pulling from an intermediary without authentication). Here are 2 ways to serve your git repository without any configuration or software installation. Both ways serve a single repository without authentication or encryption but read-only (no push).

Using the git protocol

The git executable is itself a git server using the native git protocol. Inside the root of the repository run the following command

git daemon --reuseaddr --verbose  --base-path=. --export-all ./.git

And on the client you can clone by running

git clone git://servername/ reponame

Using the HTTP protocol

This way serves the repo over HTTP using Python 2's SimpleHTTPServer. Run the following in the rot of the git repo

git update-server-info
cd .git
python -m SimpleHTTPServer

And on the client clone by running

git clone http://servername:8000/ reponame

Final words

I've added both ways as git aliases in my rcfiles repo.

SSH security

2016-07-05T00:00:00+03:00

Over the years I'd heard many people share their best practices regarding securing SSH access and have been asked by friends and colleagues how to secure their servers. So here are my opinions and practices regarding SSH security. The main point I try to get across is balance. Balance between security and functionality.

Practices I avoid

First, changing the listening port. The upside is that a high random port is scanned less often and the various script kiddies sometimes fails to notice it, thus reducing the noise in the logs. This however is no real security measure as any capable attacker will quickly spot the daemon listening on a different and all benefits will be lost. The downside is that by not using the default port you need to configure all clients accordingly. So, no substantial wins and minor loss. I pass on this idea.

The second most common is allowing access only from the office IP or a few select IP addresses. The security benefit is high but the risk is also high. I view SSH access to servers as critical and limiting access puts you in risk of locking yourself out in case of trouble/ emergency. Not having management access to your servers is more dangerous to your business than allowing whomever to try (not succeed) to access your servers. Therefore I prefer to limit the access in different ways.

Practices I employ

Most importantly, allow only key-based authentication. This works without relying on 3rd party services and is extremely secure when done properly. The most important issue keeping your private key private. Personally I keep my home directory on a separate partition that is encrypted with LUKS so the keys are encrypted when in rest. If you don't have encrypted storage for your keys, password encrypt the keys (consult the ssh-keygen manual for instructions).

Another measure of security is limiting the number of authentication attempts any single IP can perform in a given time. This is achieved by 2 actions. By ensuring that MaxAuthTries in your sshd config is not set too high (the default is 6 which is damn reasonable) and by limiting the number of TCP connections to port 22 any IP can initiate. With UFW on Linux this is done by running ufw limit ssh and for OpenBSD or FreeBSD I'd refer you to Peter Hansteen's great PF tutorial.

These 2 steps will create a barrier to entry that no brute-force attack will be able to overcome in anyone's lifetime. This is what I do on all of my servers and it has served me well. However, although the commonly used OpenSSH is extensively used, researched and tested, bugs still happen. Keeping current with updates is vital.

Also, some other good practices are disabling root login, SSH protocol version 1 is deprecated, insecure and must be turned off (is off by default, but I felt it's worth mentioning). To further simplify things, here is a short Ansible playbook that covers the actions mentioned above for Debian based systems.

---
- hosts: all
  handlers:
  - name: Restart SSH
    service:
      name: ssh
      state: restarted

  tasks:
  - name: APT install and update
    with_items:
    - openssh-server
    - ufw
    apt:
      name: '{{ item }}'
      state: latest
      update_cache: yes
      cache_valid_time: 3600

  - name: Configure SSHd
    with_dict:
      MaxAuthTries: 10
      PasswordAuthentication: no
      PermitRootLogin: no
      Protocol: 2
    lineinfile:
      dest: /etc/ssh/sshd_config
      regexp: '{{ item.key }}'
      line: '{{ item.key }} {{ item.value }}'
      state: present
    notify:
    - Restart SSH

  - name: Enable UFW
    ufw:
      state: enabled

  - name: Rate limit SSH
    ufw:
      rule: limit
      port: ssh
      protocol: tcp

Expanding variables and shell expressions in parameters to Docker entrypoint

2016-06-13T00:00:00+03:00

A known best practice when creating Docker images is when you need to run commands in runtime before starting the actual application/ daemon is to create an entrypoint script and pass the command as parameters in the CMD instruction. Another best practice is to exec the final command so it would be PID 1 and receive the signals passed to it. Let's create a small example. Here's the Dockerfile:

FROM alpine
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
ENV var value
CMD ["echo", "$var"]

And the entrypoint.sh script:

#!/bin/sh
set -eu
# Perform any needed tasks here.
exec $@

Now let's build and run this container:

$ docker build --tag entrypoint .
Sending build context to Docker daemon 28.67 kB
Step 0 : FROM alpine
 ---> 5f05d2ba9e65
Step 1 : COPY entrypoint.sh /entrypoint.sh
 ---> f59f4d7f3546
Removing intermediate container 27ca546c6b6c
Step 2 : ENTRYPOINT /entrypoint.sh
 ---> Running in 98c65b63948a
 ---> 1de45b33021b
Removing intermediate container 98c65b63948a
Step 3 : ENV var value
 ---> Running in 133a8781f0ac
 ---> bba451334fb2
Removing intermediate container 133a8781f0ac
Step 4 : CMD echo $var
 ---> Running in e8436c6c3202
 ---> a49d9b335b74
Removing intermediate container e8436c6c3202
Successfully built a49d9b335b74
$ docker run entrypoint
$var

As we can see the variable var wasn't expanded to it's content. After a bit of head scratching, The following simple change was made to the entrypoint script.

#!/bin/sh
set -eu
# Perform any needed tasks here.
eval "exec $@"

The change is to first evaluate the expression (expanding any variable and expression found), then exec it. The outcome is what you'd expect.

$ docker build --tag entrypoint .
Sending build context to Docker daemon 28.67 kB
Step 0 : FROM alpine
 ---> 5f05d2ba9e65
Step 1 : COPY entrypoint.sh /entrypoint.sh
 ---> b874d862999d
Removing intermediate container fb6483ff00e3
Step 2 : ENTRYPOINT /entrypoint.sh
 ---> Running in 82adf0b2c4c7
 ---> 6674f336c5e1
Removing intermediate container 82adf0b2c4c7
Step 3 : ENV var value
 ---> Running in 599f3f98c11d
 ---> 980f1e1e1ad5
Removing intermediate container 599f3f98c11d
Step 4 : CMD echo $var
 ---> Running in e29f1948480a
 ---> e27fd79143f8
Removing intermediate container e29f1948480a
Successfully built e27fd79143f8
$ docker run entrypoint
value

An example Ansible role

2016-05-19T00:00:00+03:00

A few weeks ago I started a new job and a lot of time was spent on refactoring as well as adding to an existing Ansible automation code base. For me this was a chance to work more with Molecule for testing. Molecule is a infrastructure-as-code testing tool that is inspired by Test-kitchen and the tests can be written using Testinfra which in turn is using pytest. The reasons for me to choose this combination is that the tools are written in Python and that they're focused on Ansible. However I quickly grew tired of copying files from role to role or making the same changes to files again and again. So in that spirit I created a new Git repo with an empty Ansible role (no tasks, variables, handlers etc.) but has all of my changes and tweaks already applied and working tests out of the box.

Usage

To work on the role install VirtualBox and Vagrant (I use the versions in Debian's repos) and from PyPI Ansible, Molecule and Testinfra. Now, fork the repo. As you can see there are already README and LICENSE files. If you ever ran ansible-galaxy init or molecule init you'll notice that indeed the repo was created with those tools.

Dependencies

There's an example dependency present in meta/main.yml but instead of the declaring the dependencies in meta/main.yml and the sources of the dependencies in requirements.yml which leads to repeating yourself, the example shows how to declare the source of the dependent role directly in meta/main.yml (which I haven't seen mentioned clearly in the Ansible documentation. For repositories with playbooks I'd still add a requirements.yml file since there's no meta directory. Pulling the dependencies took some thought and what I came up with is:

ansible-galaxy install git+file://$(pwd),$(git rev-parse --abbrev-ref HEAD)

This is a workaround for installing the dependencies as it actually uses ansible-galaxy to install the git repo of the role and the dependencies as well.

Testing

First, I configured pre-commit hooks that check, among other things, the validity of the YAML files and the does a syntax check of the Ansible playbook.

As for Molecule, the configuration of the test environment is mainly under molecule.yml. That is were you'd go to change the Vagrant box to test. You can add multiple boxes and specify which box to test like so molecule test --platform .

Also worth mentioning is the Ansible configuration in ansible.cfg. This is some what of a workaround as well because many of the options can be configured in molecule.yml which is used to generate its own ansible.cfg. However since Testinfra runs the tests over Ansible and Molecule doesn't pass the configuration along to it, the configuration isn't honored during testing. This caused me some grief as tests were constantly failing because Ansible would the host SSH key and fail as it was not known. The way I did is create an ansible.cfg at the root of the repo where Testinfra would look and passed that as the template to Molecule.

The playbook that is run in at tests/playbook.yml and the tests are under tests/ as well. There's an simple example test but the Testinfra documentation quite good. Just remember to that both the filename and function name should start with test_ and you won't have tests that aren't found.

A word on CI

Now you have all of the different pieces and workflow to run complete tests on roles the next obvious step is setting up a CI pipeline. In my tests and as I know the various CI services (I tried Travis-CI and CircleCI) disable the option to run any hypervisor. For me it's a deal breaker because I depend on VirtualBox (I need to test on different OSes, not just Linux). If LXC serves your needs than you should be able to run Vagrant with the LXC provider and therefore Molecule. For me it's a deal breaker.

A final word on boiler-plate

In a previous post I mentioned that I have several repositories that have the same boiler-plate and how I plan on dealing with that. Now, this is the first attempt at this. The idea is having a base repo that I clone, add another remote and voilà, a new project with the scaffolding already there. For bonus points, I can update the base repo and pull those changes in all projects. Here's how I do it:

git clone https://git.shore.co.il/ansible/ansible-role-example.git ansible-role-name
cd ansible-role-name
for file in $(git grep -l ansible-role-example); do sed -i 's/ansible-role-example/ansible-role-name/g' $file; done
git add .
git commit -m"- Renamed ansible-role-example to ansible-role-name."
git remote rename origin ansible-role-example
git remote add origin git@example.com/path/to/repo
git push -u origin master

And in case I update the ansible-role-example repo than I pull the updates by running git pull ansible-role-example master.

Pre-commit hooks

2016-03-12T00:00:00+02:00

Pre-commit is a nice, simple tool to add Git hooks to your project. The primary goal is running fast checks on commits (before committing them), mainly linters and syntax checkers. Today I've 2 of my own, for Ansible playbooks and shell scripts. The Ansible playbooks hook is located at https://git.shore.co.il/ansible/ansible-pre-commit.git and the shell scripts hook is at https://git.shore.co.il/nimrod/shell-pre-commit.git. Both have a short README which describes installation and usage.

My view on testing

I find that Pre-commit suites my view on proportionate testing. The smaller the change, the faster the test (and as a result, more trivial). Personally, I prefer to structure my work as small commits that are easier to revert, these deserve fast (and more trivial) tests which Pre-commit provides. The bigger the change, the more rigorous (and thus longer) the test. In my opinion this helps in creating a good workflow which quickly finds small errors while developing and reduces the number of times one must ran the full test suite because he/she had a typo that failed the test. This is why I prefer to separate the test suite so that I can the ability to run the simpler and faster locally and get rid of simple error quickly.

bundle certs

2016-03-02T00:00:00+02:00

Like I said in a previous blog post, I rarely blog but I run git init project-name pretty regularly. So here's a new such repo, bundle_certs. A simple shell script for bundling (in the correct order) SSL certificates.

How I start new projects

This little tool, along with ssl-ca and ssh-ca have some commonality in how I use them and this seems like a good opportunity to share. I keep my rc files (like .vimrc) in the rcfiles repo >_. However I don't install them as mentioned in the documentation. Instead I add them as Git sub modules and now I can be reasonably sure that when I clone the rcfiles repository, the aliases and sourced files mentioned in .bashrc are present. Here's how:

ssh cgit 'git init --bare /srv/git/REPONAME'
git submodule add -b master -f https://www.shore.co.il/cgit/REPONAME

First I create the remote repository (most of you would probably use Github but I prefer self hosting). Then I add it as a Git submodule.

Repository boiler-plate

Truth be told, there are more line of tests, documentation, license, etc. than there is actual code in these repositories. It happened to a few times that I added something nice to a repository that I wanted to have in all (or most) of my other repositories and in new repositories going forward.

One solution I thought of is creating a base template repository that all others are forked from. The upside is if I change something in the base repository I can fetch it in all other repositories. The downside is not all repositories are the same (different license, programming language, pre-commit and git hooks).

Another option I know of are tools that manage a specific aspect of the repo, for example the license, or .gitignore.

A third option is using a project management tool like Cargo for Rust or Leiningen for Clojure. But not all aspects or languages have such tools.

The fourth option I'm thinking of is using a scaffolding tool, mainly Yeoman as it seems to the most popular one but its focus is on JS and webapps.

As of now, my plan is to try and maintain a base repo for certain project types and see how it goes (Yeoman would just take more time to get started with).

Finding is a script sourced or not?

2016-02-29T00:00:00+02:00

I've recently written a shell script that contained several functions and I wanted to support 2 usage methods. The first is quite regular, marking it as an executable and running it. The second is to source the script and gain the functions declared. The problem is not actually performing any tasks (or outputting anything) if the script is being sourced. It took a bit of fiddling but I found a short one-liner to add at the top of the script that solves this in a POSIX-compliant way (at least on my test machines, Debian with Bash and Dash and KSH on OpenBSD). Here is an example usage:

#!/bin/sh -e
# Check if the script is being sourced or not.
[ "$_" != "$0" ] && expr "$-" : ".*i.*" > /dev/null && sourced=1

if [ "$sourced" ]
then
    echo Sourced
else
    echo Run
fi

The solution is using 2 heuristics. If the last argument ($_) if different from the command name (must be first command run, otherwise the last argument will be overwritten). The second is if the option flags ($-) contain i for interactive. This works when both marking the script as executable and passing the name as a parameter to the shell.

Sharing Ansible modules

2015-11-15T00:00:00+02:00

With Ansible you're expected to share roles with the Ansible Galaxy tool (either through the Ansible Galaxy hub or just using straight git repositories). This works well enough (and personally I am using ansible-galaxy init to start each new role, even those that I'm not going to share with the community). However, for sharing modules there is no such easy solution, or is it?

Sharing with git submodule

I'd like to start by saying that git submodule is the poor man's package manager and it's lack of popularity is (somewhat) justified. However, this is a nice demonstration of a case where there is no package manager available and of using git submodule instead. Also, I've only been able to use this technique for modules written in Python, which is nice considering the lack of boiler-plate that Ansible provides and that Python is my personal preference.

The whole story is really quite simple, create a separate git repository with the modules in it. You can put them in sub-directories and as a far as I know, there's no restriction on the hierarchy depth. In your playbook directory create a library directory (the Ansible default, so you can change this in ansible.cfg) and create an empty __init__.py file inside that directory. Add a git submodule inside that directory and you're done. Let's see an example

git init ansible-modules
cd ansible-modules
# Write great module
git commit -a
git push
cd /path/to/your/ansible/playbook/repository
mkdir library
touch library/__init__.py
git submodule add host:/path/to/ansible-modules library/my_modules
git add .gitmodules
git commit
git push

Really, not that complicated. The only magic (undocumented) bit is creating a __init__.py file inside the library directory, which is a shame that the Ansible documentation doesn't cover that. If you want to see a real-life example, checkout my ansible-playbooks and ansible-modules git repos.

Why I don't write on my blog often?

2015-11-10T00:00:00+02:00

I often criticize myself on not blogging more often. The process goes like this: I'm doing something mildly interesting and I say to myself 'This is mildly interesting, maybe someone else will find this mildly interesting.'. But 9 out of 10 times, what ever I'm doing has some code (when I say code I usually mean an Ansible playbook, a shell script or something similar) accompanying. Instead of a lengthy blog post, I publish a git repo. The repo has a README file, the code is documented, there's a Makefile or fabfile, you can clone and fork the repo. It's almost always better than a blog post.

However now I have many repositories and just a few blog posts. What I'm going to do from now on is I'll publish the git repo, but add a short post announcing the repo.

ssl-ca

I'm announcing ssl-ca, a tool to generate a certificate authority, keys and signed certificates. The main use case is an internal network (like a development or staging environment, but not just) where you control all nodes. For that goal, it's as close to a real CA as needed and somewhat secure. There's no OCSP or CRL, the certs serial is random, but the default hash, bit length and algorithms are modern and secure. You can get it at: https://git.shore.co.il/nimrod/ssl-ca/.

Using Ansible as a Python module

2015-01-01T00:00:00+02:00

At my current employer we have several servers in production with various providers, some of them with multiple IP addresses. When configuring the firewall to allow traffic from other servers I reached for Ansible. The obvious solution was to use a nested loop, something like this:

- name: Allow other servers
  ufw:
    rule: allow
    from_ip: '{{ item[1] }}'
  with_nested:
    - all_hosts
    - '{{ item.ansible_all_ipv4_addresses }}'

However, this syntax is invalid (and other variations I tried). Using include with with_items is deprecated and I didn't manage to get it to work with registering variables as well. What I had left was programmatically generating a playbook, but investigating further I found that Ansible can be imported as a Python module.

Incorporating Ansible in Python

To retrieve all of the ip addresses I'd ran the setup module to gather the information

from ansible.runner import Runner
struct = Runner (module_name='setup', pattern='all_hosts').run()

Now we have a complex data structure that is the output of Ansible's fact gathering module. Running it in the interpreter and examining the structure is not hard at all and that is how I managed to write the following code to extract a list of all of our server's ip addresses.

ipaddresses = []
for host in struct['contacted']:
    for ip in struct['contacted'][host]['ansible_facts']['ansible_all_ipv4_addresses']:
        ipaddresses.append (ip)

Putting that information to good use

Now that we have a list of the ip addresses, we can start running Ansible commands right from with Python (just like we did) or build a playbook by outputting a YAML file. I chose the latter.

from yaml import safe_dump
doc = {'all_ipv4': ipaddresses}
print (safe_dump (doc), file='vars.yml')

This will create a vars.yml file with the all_ipv4 variable already defined there to be imported to any playbook and run. For example:

---
- hosts: all_hosts
  vars_files:
    - vars.yml
  tasks:
    - name: Allow other servers
      with_items: all_ipv4
      ufw:
        rule: allow
        from_ip: '{{ item }}'

With this much little code we were able to query all of our hosts, extract the needed information and output it back to Ansible for further use. I see this as a product of the good decisions the Ansible developers choose early on (YAML, Python, SSH). As always, for any feedback you may have, email me.

SSL/TLS ciphers

2014-07-12T00:00:00+03:00

The problem at hand

You have a website and you want to encrypt the traffic going in and out of your webserver. Since you heard about the attacks currently known at SSL and TLS, you want to configure your server to not be vulnerable to any. In a perfect world (or if you control your clients) all you have to do is allow TLS 1.2 and AES-GCM with elliptic-curve Diffie-Hellman key exchange only (AESGCM+ECDH when using openssl) and you're set. This combination is secure, fast, offers perfect forward secrecy and at the time of writing there are no known attacks that make it crackable in a reasonable time. So what's the problem? With a public website you don't control the web browser the visitor uses. If he or she is using IE on Windows XP or Android 2.x the browser doesn't support TLS 1.2 or AES-GCM and the visitor can't access the website. How do you keep your website secure yet reasonably accessible?

Known attacks on SSL and TLS

First, SSL 2.0 is insecure (it's even disabled by default in IE7) so we'll not be using it. Version roll back attacks allow a man in the middle to change the response from the client to force a lower grade (read the lowest grade possible) cipher suite. The BEAST attack exploits a weakness in CBC ciphers in TLS 1.0. But fixes all major browsers have been released for quite some time, so we're going to assume that the client is secure and CBC ciphers are safe to use (reasonable assumption, but still an assumption). CRIME and BREACH exploit a weakness in compression and RC4 is considered to be weak although not broken like DES or MD5.

IE in Windows XP

All version of IE that are available on Windows XP offer RC4 and 3DES as the best ciphers available. Unfortunately Chrome uses the Windows scrypt library so it has the same limitation. For a user this means that if you're on Windows XP you should be using an up-to-date version of Firefox to have the best experience until you can move from Windows XP (or Windows in general). For the website manager it leaves you with 2 options, either add support for either 3DES or RC4 ciphers with SHA1 hashes (for openssl, add RC4-SHA or 3DES-SHA at the end of the cipher list) or ask users to use Firefox if they're still on XP. I chose the latter rather then the former, but I have that luxury.

What are we left with?

Since modern browser browsers that support SSL 3.0 support TLS 1.0, we'll be using TLS 1.0 or newer. Any AES cipher (AES-GCM preferred) with ECDH key exchange (preferred) or DH key exchange and SHA2 (preferred) or SHA1 hashes and disable compression. On my server (OpenBSD firewall/ load-balancer/ SSL terminator and reverse-proxy) with the included OpenSSL and Nginx the configuration is as followed

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers !kRSA:!3DES:!RC4:!DES:!MD5:!aNULL:!NULL:AESGCM+ECDH:AES256+ECDH:AES128:+SHA1;
ssl_prefer_server_ciphers on;

Take note that I first disable what I don't want, then allow what I do want in the order I prefer. I've also disabled DH key exchange with AES-GCM since all browsers that support AES-GCM support ECDH so I've opted for that (the reasoning being that ECDH is faster than DH so it's preferable).

Final words

This is not enough to call your site secure. I haven't mentioned secure cookies, HSTS, input sanitation, cross-site scripting, OCSP, certificate strength, implementation vulnerabilities (such as OpenSSL's heartbleed) or any of the other security considerations. For testing purposes I used sslscan and Calomel's SSL validation add-on for Firefox. You can also SSLLabs' SSL test.

Blogging with Pelican

2014-04-19T00:00:00+03:00

What is Pelican?

Pelican is a static site generator. It's written in Python, focusing on blogs, using reStructuredText, Jinja2 and Fabric (but you can use Markdown and makefiles and has provisions for normal web pages as well). It's a pythonic tool that's easy to use and was a breeze to setup.

Installing Pelican

As Pelican is a static blog/ website generator, all we're doing is in your workstation. All you need to have server-wise is a bog-standard web server (like Apache or Nginx). Everything else is done on your local machine. I installed Pelican from Debian (it's currently available in testing)

apt-get install python-pelican fabric

Alternatively, you can use pip

pip install pelican fabric

Creating a blog

Create a blog directory and an empty blog

$ mkdir blog
$ cd blog
$ pelican-quickstart
Welcome to pelican-quickstart v3.4.0.

This script will help you create a new Pelican-based website.
Please answer the following questions so this script can generate the files
needed by Pelican.


> Where do you want to create your new web site? [.]
> What will be the title of this web site? My Blog
> Who will be the author of this web site? 
> What will be the default language of this web site? [en]
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n)
> What is your URL prefix? (see above example; no trailing slash) 
> Do you want to enable article pagination? (Y/n)
> How many articles per page do you want? [10]
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n)
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n)
> Do you want to upload your website using FTP? (y/N)
> Do you want to upload your website using SSH? (y/N) y
> What is the hostname of your SSH server? [localhost] 
> What is the port of your SSH server? [22]
> What is your username on that server? [root] 
> Where do you want to put your web site on that server? [/var/www] 
> Do you want to upload your website using Dropbox? (y/N)
> Do you want to upload your website using S3? (y/N)
> Do you want to upload your website using Rackspace Cloud Files? (y/N)
> Do you want to upload your website using GitHub Pages? (y/N)
Done. Your new project is available at blog

Since Pelican uses OpenSSH, you can use servers defined in your SSH preferences. Now, lets configure the blog to our liking.

Configuration

In the blog directory there are the 2 configuration files: pelicanconf.py for configuring Pelican and publishconf.py for configuration that are only for publishing using Make or Fabric. Pelican also creates standard Makefile and fabfile.py for you. I've made the following modifications to pelicanconf.py:

TIMEZONE = 'Asia/Jerusalem'
PATH = "content"
DIRECT_TEMPLATES = ('index', 'archives')
DISPLAY_CATEGORIES_ON_MENU = False
DISPLAY_PAGES_ON_MENU = True
TAGS_SAVE_AS = ''
TAG_SAVE_AS = ''
STATIC_PATH = ['static']

And to publishconf.py:

CATEGORY_FEED_ATOM = None

I've set the timezone to mine (so that the time of published articles is correct), add everything under contents/static as static contents to be uploaded to the server, disabled showing of categories of articles and creating feeds for them, disabled saving of articles by tags and set pages (which are simple web pages unlike articles which are blog entries) to show on the menu. Next, themes.

Themes

Pelican comes with a default theme (the same as used by Pelican's website) but I wanted something more understated so I took at look at https://github.com/getpelican/pelican-themes and chose pelican-mockingbird. Either clone it or add it as a git submodule (depends on if you're using Git to version control your blog or not)

git clone https://github.com/wrl/pelican-mockingbird.git #If you're not using Git.
git submodule add https://github.com/wrl/pelican-mockingbird.git #If you're using Git.

and set the theme to that by adding the following to pelicanconf.py:

THEME = "./pelican-mockingbird"

I've also edited base.html and article.html inside of pelican-mockingbird/templates to suite my liking. Next, let us add a new entry.

Adding an entry

Create a ReStructuredText file inside of contents. The filename is for personal use and not critical. The heading is the article name and you can add the following for Pelican to use:

:date: 2014-04-19
:slug:  this-will-the-filename
:author: 
:summary:

After we added the content we want to upload it to our web server (I use fabric)

fab publish

If you don't have keys set for the server it will ask you for your password to the server. Last thing, you can create pages, create a pages directory inside contents and save the files there. Their format is the same as articles but they'll have a somewhat template applied and they will be shown in the menu. A good example will an 'About Me' page.

That's it, you now have Pelican installed, configured and published to your web site. If you want to see a real life example, clone my blog.