Automating tasks with Makefiles

Almost 20 years ago, one of the first posts on this blog (hosted elsewhere at the time) was about documentation.

Since then, I’ve written about documentation and checklists and the like spradically. The problem is that although I know documentation and checklists are a good thing, I don’t use them enough.

It is more fun to write code.

At the same time, I have a hidden perfectionist in me (trust me, he’s there), so if I write code to perform some process, I can spend a lot of time making sure it works just right.

So, (part of) the cure for my lack of documentation, is to write code that performs a task and let the code be the documentation. (I’ve even used this as an excuse to practice literate programming because then I can write code and readable documentation at the same time in Emacs.)

Anyway, back to code as documentation.

With my background in setting up systems, I know all too well the pain of having to repeat something over and over. At the same time, because I’m so old, I don’t want to learn any new tool when the tools I have are already. So, while friends of mine have used Ansible and similar tools to set up complete MediaWiki systems, I’m too opinionated about how I do things that, try as I might, I couldn’t just use their system.

Which brings us to Make. GNU Make in particular. I coudl get into the byzantine differences between makes, but I tend to be on Linux and, hey, GNU make is available on the other systems.

For the past year or so, I’ve been working on deploying MediaWiki with Make. I just used it to stage a major upgrade at a client of mine. Today, I have a small project I need to deploy, so I decided to try and use my Makefile method. Over the next few days, I’ll document this.

Get the makefile skeleton

Obviously the first thing to do is get my makefile skeleton set up. I’ve learned that I only need a stub of a file to do this and I’ve been adapting it over the years. Here’s what I have so far:

include makeutil/baseConfig.mk
baseConfigGitRepo=https://phabricator.nichework.com/source/makefile-skeleton

.git:
    echo This is obviously not a git repository! Run 'git init .'
    exit 2

#
makeutil/baseConfig.mk: /usr/bin/git .git
    test -f $@                                                                                                                              ||      \
        git submodule add ${baseConfigGitRepo} makeutil

With that in place as my Makefile, I just run make and the magic happens:

$ make
echo This is obviously not a git repository!
This is obviously not a git repository!
exit 2
Makefile:1: makeutil/baseConfig.mk: No such file or directory
make: *** [Makefile:6: .git] Error 2

Ok, well, I run git init && make and the magic happens:

$ git init && make
Initialized empty Git repository in /home/mah/client/client/.git/
test -f makeutil/baseConfig.mk                                                                                     ||       \
    git submodule add https://phabricator.nichework.com/source/makefile-skeleton makeutil
Cloning into '/home/mah/client/client/makeutil'...
remote: Enumerating objects: 106, done.
remote: Counting objects: 100% (106/106), done.
remote: Compressing objects: 100% (106/106), done.
remote: Total 106 (delta 52), reused 0 (delta 0)
Receiving objects: 100% (106/106), 36.38 KiB | 18.19 MiB/s, done.
Resolving deltas: 100% (52/52), done.

  Usage:

    make <target> [flags...]

  Targets:

    composer   Download composer and verify binary
    help       Show this help prompt
    morehelp   Show more targets and flags

  Flags: (current value in parenthesis)

    NOSSL      Turn off SSL checks -- !!INSECURE!! ()
    VERBOSE    Print out every command ()

Better.

Set up DNS

I want to put the client domain on its own IP with its own DNS record. I don’t have “spin up a VM” anywhere close to automated, but I have been using my bind and nsupdate to update my files, so I’ve automated that.

# DNS server to update
dnsserver ?= 

# List of all DNS servers
allDNSServers ?=

# Keyfile to use
keyfile ?= K${domain}.private

# DNS name to update
name ?=

# IP address to use
ip ?=

# Time to live
ttl ?= 604800

# Domain being updated
domain = $(shell echo ${name} | sed 's,.*\(\.\([^.]\+\.[^.]\+\)\)\.*$,\2,')

NSUPDATE=/usr/bin/nsupdate
DIG=/usr/bin/dig

#
verifyName:
    test -n "${name}"                                                                                                       ||      (       \
        echo Please set name!                                                                                           &&      \
        exit 1                                                                                                                          )

#
verifyIP:
    test -n "${ip}"                                                                                                         ||      (       \
        echo Please set ip!                                                                                                     &&      \
        exit 1                                                                                                                          )

#
verifyDomain:
    test -n "${domain}"                                                                                                     ||      (       \
        echo Could not determine domain. Please set domain!                                     &&      \
        exit 1                                                                                                                          )
    test "${domain}" != "${name}"                                                                            ||      (       \
        echo Problem parsing domain from name. Please set domain!                       &&      \
        exit 1                                                                                                                          )

#
verifyKeyfile:
    test -n "${keyfile}"                                                                                            ||      (       \
        echo No keyfile. Please set keyfile!                                                            &&      \
        exit 1                                                                                                                          )
    test -f "${keyfile}"                                                                                            ||      (       \
        echo "Keyfile (${keyfile}) does not exist!"                                                     &&      \
        exit 1                                                                                                                          )

# Add host with IP
addHost: verifyName verifyIP verifyDomain verifyKeyfile ${NSUPDATE}
    printf "server %s\nupdate add %s %d in A %s\nsend\n" "${dnsserver}"                     \
        "${name}" "${ttl}" "${ip}" | ${NSUPDATE} -k ${keyfile}
    ${make} checkDNSUpdate ip=${ip} name=${name}

# Check a record across all servers
checkDNSUpdate: verifyName verifyIP
    for server in ${allDNSServers}; do                                                                              \
        ${make} checkAddr ip=${ip} name=${name} dnsserver=$server              ||      \
            exit 10                                                                                                         ;       \
    done

# Check host has IP
checkAddr: verifyName verifyIP ${DIG}
    echo -n ${indent}Checking $server for A record of ${name} on ${dnsserver}...
    ${DIG} ${name} @${dnsserver} A | grep -q ^${name}.*IN.*A.*${ip}         ||      (       \
        echo " FAIL!"                                                                                                           &&      \
        echo ${name} is not set to ${ip} on ${dnsserver}!                                       &&      \
        exit 1                                                                                                                          )
    echo " OK"

Now, I’ll just add the IP that I got for the virtual machine to the DNS:

$ make addHost name=example.winkyfrown.com. ip=999.999.999.999
> > Checking for A record of example.winkyfrown.com. on web.nichework.com... OK
> > Checking for A record of example.winkyfrown.com. on ns1.worldwidedns.net... OK
> > Checking for A record of example.winkyfrown.com. on ns2.worldwidedns.net... OK
> > Checking for A record of example.winkyfrown.com. on ns3.worldwidedns.net... OK
> > Checking for A record of example.winkyfrown.com. on 1.1.1.1... OK

(Of note: this goes back to the checklist bit. When I first tested this, I found that my nsupdate wasn’t propagating to one of my secondaries. It prompted me to check who was allowed to do zone transfers from the host and fix the problem.)

Basic server setup

I believe in versioning (wherever it is easy). So the first thing we’ll do is install etckeeper.

#
verifyHost:
    test -n "${REMOTE_HOST}"                                                                                        ||      (       \
        echo Please set REMOTE_HOST!                                                                            &&      \
        exit 10                                                                                                                         )

#
verifyCmd:
    test -n "${cmd}"                                                                                                        ||      (       \
        echo Please set cmd!                                                                                            &&      \
        exit 10                                                                                                                         )

doRemote: verifyHost verifyCmd
    echo ${indent}running '"'${cmd}'"' on ${REMOTE_HOST}
    ssh ${REMOTE_HOST} "${cmd}"


# Set up etckeeper on host
initEtckeeper:
    ${make} doRemote cmd="sh -c 'test -d /etc/.git || sudo apt install -y etckeeper'"

Initial installation of Apache+PHP on the server

Finally, let’s set up a webserver!

# Install the basic LAMP stack
initLamp:
    ${make} doRemote cmd="sh -c 'test -d /etc/apache2 || sudo apt install -y        \
        php-mysql php-curl php-gd php-intl php-mbstring php-xml php-zip                 \
        libapache2-mod-php'"
    ${make} doRemote cmd="sh -c 'test -d /var/lib/mysql || sudo apt install -y mariadb-server'"
    ${make} doRemote cmd="sh -c 'sudo systemctl enable apache2'"
    ${make} doRemote cmd="sh -c 'sudo systemctl enable mariadb'"
    ${make} doRemote cmd="sh -c 'sudo systemctl start apache2'"
    ${make} doRemote cmd="sh -c 'sudo systemctl start mariadb'"

    curl -s -I ${REMOTE_HOST} | grep -q ^.*200.OK                                           ||      (       \
        echo Did not get "'200 OK'" from ${REMOTE_HOST}                                         &&      \
        exit 1                                                                                                                          )
    touch $@

And the basic website:

setupSite: initLamp verifyRemotePath
    ${make} doRemote cmd="sh -c 'test -x /usr/bin/tee || sudo apt install -y        \
            coreutils'"
    (                                                                                                                                                       \
        echo "<VirtualHost *:80>"                                                                 &&      \
        echo "  ServerName ${REMOTE_HOST}"                                                &&      \
        echo "  DocumentRoot ${REMOTE_PATH}/html"                                 &&      \
        echo "  ErrorLog ${REMOTE_PATH}/logs/error.log"                   &&      \
        echo "  CustomLog ${REMOTE_PATH}/logs/access.log combined"                      &&      \
        echo "  <Directory ${REMOTE_PATH}/html>"                                                        &&      \
        echo "          Options FollowSymlinks Indexes"                                                 &&      \
        echo "          Require all granted"                                                                    &&      \
        echo "          AllowOverride All"                                                                              &&      \
        echo "  </Directory>"                                                                                           &&      \
        echo "</VirtualHost>"                                                                                                   \
    ) | ${make} doRemote                                                                                                            \
        cmd="sh -c 'test -f /etc/apache2/sites-available/${REMOTE_HOST}.conf || \
                sudo tee /etc/apache2/sites-available/${REMOTE_HOST}.conf'"
    ${make} doRemote                                                                                                                        \
        cmd="sh -c 'test -L /etc/apache2/sites-enabled/${REMOTE_HOST}.conf      ||      \
            sudo a2ensite ${REMOTE_HOST}'"
    ${make} doRemote                                                                                                                        \
        cmd="sh -c 'test ! -L /etc/apache2/sites-enabled/${REMOTE_HOST}.conf || \
            sudo systemctl reload apache2                                                                   ||      \
            ( sudo systemctl status apache2 && false )'"
        touch $@

Finally, let’s deploy MediaWiki!

How I chased down a PHP mode indention bug in Emacs

(When I posted this to reddit, someone pointed out that I could have gotten the same information from c-show-syntactic-information by hitting C-c C-s. Awesome!)

I was getting irritated by emacs refusing to indent PHP code properly when I was working on a MediaWiki extension.  I’ve run into this before, but, today, in order to procrastinate a little, I ran into it again with the following code:

try {
    $object->method( ARG );
}

It kept trying to put $object the same level as try, so it would end up with:

try {
$object->method( ARG );
}

So I chased it down. I used C-h k to find the function being called. After a bit of edebug, I found that the code in the function c-indent-line being called was essentially:

(c-get-syntactic-indentation (c-guess-basic-syntax))

In fact, doing M-; (c-get-syntactic-indentation (c-guess-basic-syntax)) RET when point was sitting on $ gave the result 4 when it tried to indent and 0 when it didn’t.

(Ok, the code had two more levels of indention than the above, so it was giving 12 and 8, but let’s not get carried away with details.)

Now, running M-x php-mode RET (i.e. none of the added configuration in .dir-locals.el) gave the proper indention. In my .dir-locals.el, though, I had set up a c-offsets-alist that mostly worked with bits and pieces copied from all over.

Running just M-; (c-guess-basic-syntax) RET returned ((statement-block-intro 1379)) so I figured I needed to add (statement-block-intro . +) to my c-offsets-list.

I added that, and it worked. And now I know how to chase down indention bugs.

(Header image by Ivo Kruusamägi [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons.)

Making git and Emacs’ eshell work together

tl;dr

I often end up in eshell.

Sometimes, because I’m running Emacs on Windows (where shells work, but it’s a pain), and sometimes just because.  The problem, until today, was that anytime I would invoke a git command that wanted to call the pager (say, diff), I would see the following annoying message from less:

WARNING: terminal is not fully functional
-  (press RETURN)

This happens because eshell sets $TERM to “dumb”.  It doesn’t try to fool anyone.  It’s dumb.

But, since I’m stubborn and lazy, I just put up with it the stupidity of eshell and the annoyance of git’s invocation of less.  Till today.

Some would say the answer is “Don’t use git in emacs — use magit!” And they’d be right. I do use magit, but commands should work, too.

So, after a spree of productivity yesterday, I woke up today and hit that annoying message again.  I decided to track it down.

I came across this StackOverflow thread. There is a hint there — I didn’t know less could be told to only page in certain cases — but not anything that says “only sometimes use less”.

So I managed to hack something together:

git config --global core.pager '`test "$TERM" = "dumb" && echo cat || echo less`'

I haven’t tried this under Windows, yet, but I’m hoping it works.

Image CC-by-SA: Richard Bartz, Munich Makro Freak

Emacs for MediaWiki

Tyler Romeo wrote:

If I had the time, I would definitely put together some sort of .dir-locals.el for MediaWiki, that way we could make better use of Emacs, since it has a bunch of IDE-like functionality, even if it’s not IDEA-level powerful.

A client wanted to me to help train someone to take over the work for maintaining their MediaWiki installation. As part of that work, they asked for an IDE and, knowing that other MW devs used PHPStorm, I recommended it and they bought a copy for me and the person I was to train.

PHPStorm has “emacs keybindings” but these are just replacements for the CUA keybindings. Somethings that I expected the keybindings to invoke, didn’t. (It’s been a while since I’ve used PHPStorm, so I’ve forgotten the details.)

In any case, I’ve found that a lot of what I wanted from PHPStorm could be implemented in Emacs using the following .dir-locals.el (which I put above my core and extensions checkouts):

((nil . ((flycheck-phpcs-standard .
        "…/mediawiki/codesniffer/MediaWiki")
     (flycheck-phpmd-rulesets .
        ("…/mediawiki/messdetector/phpmd-ruleset.xml"))
     (mode . flycheck)
     (magit-gerrit-ssh-creds . "mah@gerrit.wikimedia.org"))))

The above is in addition to the code-sniffing I already had set up to put Emacs’ php-mode into the MW style.

The one thing that PHPStorm lacked (and where Emacs’ magit excels) is dealing with git submodules. Since I make extensive use of submodules for my MediaWiki work, this set up makes Emacs a much better tool for working with MediaWiki.

Naturally, I won’t claim that what works for me will work for anyone else. I’ve spent 15 years in Emacs every day. I was first exposed to Emacs in the late 80s(!!) so the virus has had a long time to work its way into my psyche and, by now, I’m incurable.

The writer’s text editor

Let’s just use Emacs presents a compelling case for Emacs as a writer’s tool — not a geek’s tool — a writer’s tool.

In the process he takes us through his history and frustration with Emacs to the modern day where tools like Org-Mode, elements of a modern UI, and darkroom-mode, plus (in my experience and in his) a more active development community, have made Emacs into a great writing environment.

Oops, I did it again!

Working on free software projects isn’t easy. Just because you’re giving away your work for anyone to use doesn’t mean that anyone is going to take it, no questions asked. Take my MediaWiki work as an example. I am being paid for the work, but it is freely licensed and I’m learning about the standards of quality that the community has formed around the code. Frankly, before becoming involved in such a serious PHP-based project, I didn’t have a very high opinion of PHP. Even Rasmus (creator of PHP) doesn’t seem to live in a pure php world and, as a result, thinks of systems where PHP is merely the web frontend instead of almost the entire system. So working with others who have been neck-deep in PHP for years, building one of the top-10 sites on the net entirely in PHP, and gaining intimate familiarity with the quirks of PHP, has been a wonderful experience. But MediaWiki isn’t the only free software project I’m involved in. I also contribute to Emacs occasionally. (For those not so familar with Emacs vs Vi, let’s just say this is like the social situation between Republicans and the Democrats or the Roman Catholics and Southern Baptists: You live next door to them, but you know they’re going to hell.) And it is my most recent commits to Emacs that have gained me noteriety. Yesterday, I was catching up on some blog reading (Planet Emacs, thankyouverymuch) and came across a nifty use of loccur.el. But it used defadvice instead of a hook (and hooks are better — no this is different than emacs vs vi, I swear). I looked at the code and thought, “Hey, I can make a tiny little contribution to Emacs here!” So I made a couple of small changes. Little did I know what a problem that was going to be. Óscar Fuentes used my commit message as an example of how not to write a commit message. This was not the first time I’ve been so honored. Three weeks ago, I made a mistake committing to the bzr repository for emacs and was again used as an example for the Emacs-devel community of how not to make a commit. There are two reasons I’m such a stellar example for the other Emacs developers. First, I’ve been using bzr for a couple of years while working on the iHRIS Suite. This experience (2 years more than most Emacs developers) naturally made me think I had things under control. So I didn’t bother reading over Bzr for Emacs Devs. Second, Emacs recently switched its source-control system (after much debate and some effort on speed the bzr side) from the ancient, worn, CVS to bzr. So people are still adapting their work flow. I just happened to make some commits that were particularly egregious and ended up being great examples of what people should avoid. So, yes, Free Software is a great thing, but that doesn’t mean the developers don’t take it seriously. And being reprimanded in public isn’t the most pleasent experience. But at least I can blog about it!

Quick Emacs Tip: Scale fonts in a buffer

Emacs23 includes default keybindings for scaling fonts up or down:

C-x C-+ — scale the current buffer’s face/font up C-x C-+ — scale the current buffer’s face/font down

This is good as far as it goes, but I’d really like something a little easier. So I’ve added the following to my .emacs file:

(global-set-key [(control mouse-4)] (lambda () (interactive)                                        (text-scale-increase 1)))  (global-set-key [(control mouse-5)] (lambda () (interactive)                                        (text-scale-decrease 1)))

Now, if I hold down the control key and scroll the mouse wheel, the font will get larger or smaller. Some other applications (e.g. Firefox) use “Control +” and “Control -” to do something similar along with “Control 0” to return to the default size. You could set these keybindings in Emacs without losing too much (unless you’re used to C-- and C-0 as prefixes, in which case, you still have C-M-- and C-M-0):

(global-set-key [(control ?+)] (lambda () (interactive)                                        (text-scale-increase 1)))  (global-set-key [(control ?-)] (lambda () (interactive)                                        (text-scale-decrease 1)))  (global-set-key [(control ?0)] (lambda () (interactive)                                        (text-scale-increase 0)))

None of this will permanently affect the size of the font, so for the next buffer you open or the next time you start Emacs, you’ll have the same size font you started with. If you want to change the default font size, use M-x customize-face RET default RET instead.

Emacs Hack: mediawiki.el

I’ve gotten some feedback on the mediawiki mode I’ve been working on.  So I’m releasing a new version.  Some highlights:

  • Now works with HTTP Auth (I’ll write a seperate post on how Emacs handles credentials for HTTP Authentication).
  • Introduced tab-completion of sites.  If you have multiple MW sites set up that you work on, this makes switching between sites super-easy.
  • Started working on making it more XEmacs compatible.  As far as I can tell, XEmacs lacks the Unicode support that GNU Emacs has.  The released version of XEmacs also lacks the POSIX character classes for regular expressions and ships with a very out-dated version of url.el.  All these combine to make it very difficult.  But do-able.  I’m surprised there are still XEmacs users, but if it doesn’t cause me too much pain, I’ll help them out.
  • Misc other clean up (including making the url.el wrappers much better).

notify.el — pop-up notifications from emacs using dbus

Since emacs on Linux can use DBus calls now, I wrote a short notify.el as a demonstration to myself and to save myself from forking notify-send (because, uh, forking is bad) whenever someone mentions my nick in IRC. There is a delay built into it so that your “friends” on IRC can’t DDOS you with notifications. Oh, the really nifty thing about this is the function keywords-to-properties. elisp keywords (:example) allow you to fake named parameters in function calls. This is done a couple of different places in Emacs, the most visible being defcustom. But there isn’t any universal way that I could find to parse the keyword-value pairs into something usable. So I cribbed from defcustom and wrote something that I hope will be useful to others.

mediawiki.el – Emacs mode for editting MediaWiki pages

I seem to be spending an inordinate amount of time editting MediaWiki wikis, so I’ve had some time to put together a better MediaWiki mode for emacs. In the process, I wrote some code that many people (myself included) seem to think Emacs needs. HTTP POSTing in native elisp is too hard right now, so a http-post-simple.el was written. The original mediawiki.el required this library, but it wasn’t included. I refactored the dependency away and now I have some form parsing functions in elisp that I can contribute back to the Emacs core. Anyway, if you use Emacs and edit MediaWiki pages, check out mediawiki.el and let me know what you think. I’m especially interested in bug reports from anyone.