Making git and Emacs’ eshell work together


I often end up in eshell.

Sometimes, because I’m running Emacs on Windows (where shells work, but it’s a pain), and sometimes just because.  The problem, until today, was that anytime I would invoke a git command that wanted to call the pager (say, diff), I would see the following annoying message from less:

WARNING: terminal is not fully functional
-  (press RETURN)

This happens because eshell sets $TERM to “dumb”.  It doesn’t try to fool anyone.  It’s dumb.

But, since I’m stubborn and lazy, I just put up with it the stupidity of eshell and the annoyance of git’s invocation of less.  Till today.

Some would say the answer is “Don’t use git in emacs — use magit!” And they’d be right. I do use magit, but commands should work, too.

So, after a spree of productivity yesterday, I woke up today and hit that annoying message again.  I decided to track it down.

I came across this StackOverflow thread. There is a hint there — I didn’t know less could be told to only page in certain cases — but not anything that says “only sometimes use less”.

So I managed to hack something together:

git config --global core.pager '`test "$TERM" = "dumb" && echo cat || echo less`'

I haven’t tried this under Windows, yet, but I’m hoping it works.

Image CC-by-SA: Richard Bartz, Munich Makro Freak

Keeping MediaWiki sites up to date

I just completed a survey of about 2100 wiki’s running MediaWiki. The initial list came from S23-Wiki’s WikiStats, but I plan to add to it. Google, for instance, says there are “About 3,070 results” for the search string “inurl:Special:Version mediawiki "Magnus Manske, Brion Vibber"“.

My purpose in doing this is to find out what version of MediaWiki these sites are running. To do that, the easiest thing is to use the API. This way, you can just ask a site to tell you about itself and get back useful information.

Of course, sometimes the version of MediaWiki installed is too old (before 1.8) or has the api disabled and, in these cases, I had to get a bit more creative.

Usually, I could find the information on the Special:Version page.

And still, there were about 200 wikis that I tried to check that were no longer active, had database problems or some other issue.

Of those that I did get a version from, 692 (36%) were running a version marked “1.20wmf1” indicating they are run by Wikimedia; 51 (2.7%) were running a version older than 1.10; 1043 (54%) were running a version older than 1.17.3 or 1.18.2 both released over a month ago on March 22 (a more recent version of each was released last week).

I would be tempted to think that those 51 wikis running an especially old version of MediaWiki were just unmaintained or abandoned. A spot check, though, seems to show that this isn’t the case.

For example, I found one that was running a seven year old(!) version of MediaWiki on an Ubuntu server whose packages had been updated in the past month and whose wiki pages had been modified in the past couple of days. According to some traffic and search ranking sites, it gets thousands of visitors a day.

When a site owner is good about keeping his packages up-to-date and his pages spam-free (as this one was), it doesn’t seem right to not also provide a way to keep his MediaWiki site up-to-date. But even today, the instructions for installing MediaWiki on Ubuntu, push the user pretty hard to use the tarball installation instead of the Ubuntu packages.

We’ve got to do better.

A freetard on Steve Jobs at his passing

[photocommons file=”Outside Palo Alto apple store following Steve Job’s death.jpg” width=”300″ float=”right”]
I heard an NPR bit on him yesterday. I didn’t know his father was Syrian.

The story, though, was irritating for its fanboi-ism (even though a woman reported it).

As far as I can tell Steve Jobs was no more a “computer genius” than Bill Gates. He certainly didn’t invent the smart phone, nor was he the first person to release a cellphone with a camera. The story attributed both of these things to him, incorrectly.

He was a marketing genius. But he was also a micro-manager who would look at, say, versions of the Mac or iPhone that his engineers brought him and make design changes on the spot. He expected long hours from his workers.

He also was good at controlling the Apple brand. When Jobs left Apple, his replacements wanted to imitate IBM’s success with the PC, so they licensed out the specifications for the Mac so that people could start developing clones of the Mac. The Mac really suffered during this period. When Jobs returned to Apple, one of the first things he did was kill off the clones.

I respect a lot of what he has done as a business man — he knew what Apple needed as far as business and marketing. I respect his taste for design. He was able to market the iPhone and iPod as the devices to have.

But over the years, I’ve become more attracted to the sort of people Fake Steve Jobs called “freetards”. And, truly, the parody does seem to really have insight into how Steve Jobs actually thought.

I’m not happy Steve Jobs is dead. And he is a significant person beyond just the tech world. He did a lot to popularize computers during my life. He’ll be missed by many.

I’m just more interested, now, in making computing resources (and knowledge) available to everyone. I respect his focus on usability — we freetards use a lot of the ideas that Apple engineers developed and popularized — but while I do want to sustain my current standard of living, I’m interested in making resources available for everyone, not selling more shiny widgets.

HipHop packaging

[photocommons file=”20020730083218 – Debian.jpg” width=”300″]Of scripting languages used for web applications, PHP is pretty lightweight and fast. It is built to execute quickly and with little overhead, making it easy to scale.

Still, there are places where it could be better. Engineers at Facebook took this challenge and created HipHop. We at the Wikimedia Foundation would love to have HipHop packaged so that we can deploy it on our cluster. And, ultimately, while we realize we might need to do the initial work of packaging HipHop, we don’t want to be the ones responsible for keeping the package up-to-date.

Since I have some experience with packaging PHP applications and am a sometimes-active member of Debian’s PHP maintainers team, I felt this was a natural place for me to jump in.

And, because I’m lazy, the first thing I did was look for any other work on packaging HipHop that had been done. I found James DuPont’s work and built on it. Now it comes time to move this forward.

I’ve already taken his ITP for HipHop.

You’ll note that it depends on two other bugs to get the HipHop patches for curl and libevent included. These were essentially dismissed by both maintainers.

The curl maintainer pointed out that it that patch was un-needed or, at least, needed better a better defense. I can’t really provide that. I’m not sure that I can adapt the HipHop code to the suggested APIs, but I’m willing to try if the HipHop developers aren’t.

The libevent developer pointed out (in response to the Debian bug), that the patch was against an old, unmaintained version of libevent. Newer versions evidently make half of the patch un-needed and the other half needs to be adapted to the newer version of libevent.

I really want to make this happen. I really want Debian (and Debian derivatives like Ubuntu) to have HipHop packages. But I’m not sure how much time I can give to this right now since we’re in the middle of pushing out a new release of MediaWiki. If you can help solve any of the packaging problems mentioned above, please dive in!

WMF is hiring

[photocommons file=”Wikimedia logo family complete.svg” width=”300″ align=”left”]Our Deputy Director, Erik Möller, has posted a list of the open Engineering positions at the WMF that we’re hiring for right now.

If you’ve ever wanted your work to matter, to mean something more than a paycheck and the 9-5 grind, and you have the skill and aptitude for software engineering, product management, or QA, then I don’t know of a place that you can work that a bigger impact than the only non-profit with a website listed in the top 10 websites — an organisation that is dedicated to a bigger vision that we are really trying to achieve instead of something we tell you to get more eyeballs for our ad banners.

Check out our job openings — we even have some non-engineering positions. Unlike the other top web properties, Wikimedia is never going to make you rich with stock options, or a signing bonus, but we’re still small enough where your own work can make a big difference.

The writer’s text editor

Let’s just use Emacs presents a compelling case for Emacs as a writer’s tool — not a geek’s tool — a writer’s tool.

In the process he takes us through his history and frustration with Emacs to the modern day where tools like Org-Mode, elements of a modern UI, and darkroom-mode, plus (in my experience and in his) a more active development community, have made Emacs into a great writing environment.

(Starting) A philosophy of bugs on an Open project

Being a bugmeister is a new thing for me.  As someone pointed out, the title seems to be made up and almost exclusive to Open Source projects.  There is a paucity of information on How to bugmeister.  What is involved? What are the metrics for measuring my success or failure? What power do I have, besides irritating developers and administrators with bugs?

As it turns out, the answer to the last question is probably close to “none.”  But that power can be used to great effect.


To help me understand how much to bother developers with bugs, I’m using the priority field.  Developer’s should know that bugs assigned to them that are higher priority should be fixed sooner.  From a developer’s point of view, this seems like a good way to find out what bug should be fixed next.  (Note that fixing bugs is only part of a developer’s work.)

This means of course that putting a high priority on bugs that haven’t been assigned to anyone (or that have only been auto-assigned by Bugzilla) doesn’t make much sense.

This conflicts with the bug reporters’ intuitive feel about “their” bug.  They may be extremely frustrated by the buggy behavior.  Why should the priority of the bug be low when (obviously) it mattered enough to them to bother reporting it on Bugzilla?  If they are especially irritated by the bug, then they’ll see it as a very high priority bug.

From the perspective of the Foundation, though, where there are millions of users for Wikipedia and only a relatively small number of paid developers, a different understanding of priority is needed.  If only one or two editors is affected, that bug has a lower priority than one that affects hundreds of thousands of readers.

Added to the problem is an already existing database of bugs from the past nine years.

Priority and Getting Your Bug Fixed

Today, someone suggested to me that the priority of a bug should increase the longer it has been open rather than decrease.  This particular bug had been open for 18 months with relatively little discussion and no duplicates.

From the few months I’ve been on the job, a bug only a few people has noticed in over a year is, almost by definition, a low priority.

This is not to say that their irritation with the issue is meaningless, so it is here that the political savvy of the bugmeister is tested.

I have to figure out how to communicate how the people concern with a bug can get their bug noticed when it has been ignored for so long.

If users are irritated enough that they think their issue should be a higher priority, then they have two ways of getting the problem resolved.

The first is to show that the bug affects a substantial number of people and that the irritation is more than what is shown by the people who have registered an interest in having this bug be fixed.

The second is to find a developer familiar with the MediaWiki code and get them to fix the code.  The wonderful thing about Wikipedia is that this person doesn’t even have to be inside the WMF.  Plenty of developers outside the WMF have commit access.  Someone without commit access can even supply a fix by attaching a patch to the bug. When this happens, I review and apply the patch.  Then other people have the chance to review the patch and, if it fixed the problem, the fix would be deployed on Wikimedia properties.

Note, though, that bumping the priority on an old issue does not get it solved.  That is only really effective for irritating the bugmeister.

Ubuntu, for Humans

Amber, a non-technical mother, tries Ubuntu. This sounds like Alexis‘s use of Ubuntu. I’m a geek (like her husband) and my wife wants to learn how to use Linux. The amazing and amusing thing (to me) is that she when I installed Ubuntu on our kids laptops, Alexis was the one who began talking to them about the philosophy of Free Software and the obligations of the GPL. I wish Amber the best and hope that she can join the ranks of other “normal” people I know who use Ubuntu: my mother, my friend, Jim Bonewald, my cousin, Jeremy Stein (and the rest of his family), and of course, Alexis and my kids. Linux users may not yet be measurable, but we’re growing. And a lot of credit goes to Canonical and Ubuntu.

Remote Shell Game

So I was playing with screen and, after Debian put /usr/bin/screen in my /etc/shells, tried it as a login shell. It works great! I can log in and, automatically, I’m re-connected right where I left off.

One problem: I use tramp and scp. Tramp and scp just don’t work when screen is the default shell. After struggling mightly with the documentation, I switched back to bash as my login shell and implemented the following in my .bash_profile.

Tramp sets $TERM to tramp-terminal-type. By default this is “dumb” (which is also what $TERM is set to when using scp), and that’s good, because screen can’t handle “dumb”. It says something like “Clear screen capability required.” At the end of my .bash_profile, I added:

  if [ "$TERM" != "dumb" ]; then    screen -x -RR    exit  fi  

So you won’t have to look at the man page, “-x” means “attach to an already attached screen session”. You could actually have two different xterm windows with the same session running in each. If you prefer that only one window get attached at a time (say, you leave from work and forget to log out and you want to force that session to close when you log in from home), then change the “-x” to “-D”. “-RR” just means “Do what ever is necessary to re-attach.”

One nice thing about this (versus using /usr/bin/screen as your shell) is that you have a fallback in case something strange happens with screen and you can’t reconnect with it. In that case:

TERM=dumb ssh remote-box

and you’re back on the system and able to debug your screen problem.

XFS in kernel!

I took a risk earlier this year and went with XFS for the filesystem on I chose it because it was very mature, portable, and supported quotas in the metadata. Other filesystems use special files to support journaling or quotas, but this seemed too fragile to me.

There was a risk, though, since XFS hadn’t been merged with the main kernel tree. The risk was that it would later be unsupported. The latest summary of changes for 2.5 indicates that XFS has been merged with the main tree! I was afraid it wouldn’t make it.