Entries in Life

I’m very happy to be working with the Open Culture non-profit, the Wikimedia Foundation.  Still, despite our almost-rabid openness, there is still some ways that we could improve. One of those ways is fixing bugs in Wikipedia and MediaWiki. We’ve been fairly good about fixing the most onerous problems, but until recently there was no bugmeister in charge of the bug database.  As a result, some things slipped and, when they did, the person who reported the issue felt ignored. Since becoming bugmeister, Rob Lanphier has been helping me figure out how to manage the bugs and get the community involved.  I put his most recent suggestion into play with an Open Call for Parser Bugs, inviting the community to help set the agenda for our next weekly bug triage.  The results have been great.  Almost 20 bugs (in addition to my initial 10) were tagged in the first day. Since we already have a “parser” keyword, some people have asked why I chose this theme.  It is true that developers who want to fix parser bugs (a vanishingly small number of developers) could just look at bugs already tagged “parser”.  But if we just focused on the “parser” keyword, we wouldn’t find bugs like #468: Preceding text and single apostrophes are not included in links and we might lose steam before getting to the bugs that really matter to people. This last bit, what I like to call the cheerleader aspect of the bugmeister’s job, is fairly important.  An open source software (OSS) developer’s attention is a precious commodity.  If the developer is only scratching her own itch, she isn’t going to do a lot of things.  Paid OSS developers, like those employed by the WMF, are the first step beyond a self-interested development focus, but even that is not enough.  A developer is hired to work on a particular project, within a particular group, but significant parts of Wikimedia’s software are not written by anyone at the foundation. Without a bugmeister, a liaison between the community using the software and the developers working on the software, might crop up in parts of the code that are not maintained by any active (whether paid or not) developers.  And no one would notice. So the bugmeister acts as a proxy for the community.  What does the community want that people working on the software haven’t focused on yet?  To be a good proxy, I need to involve the community in my triage meetings. This is the first step of this great journey. (Note: I do not think that developers of free software are only self-interested.  But I do think we do tend towards self-interestedness in our actions without a motivation to raise our awareness of others.  Thanks to Jeroen De Dauw for pointing this interpretation of what I wrote out to me.)

We really don’t want to encourage people to run their MediaWiki site off of subversion checkouts. Still, if we don’t release often enough, some people will see a tasty feature in HEAD or an extension that only works against the TRUNK and feel compelled to skip the stable releases to use it. In 2010, we only released one version of MediaWiki: 1.16. We’re currently working on the 1.17 release. This involves branching the code, slogging through the code review of the as-yet-unreviewed revisions, and getting all the 1.17 blocker bugs fixed. As our release schedule has slowed, some admins don’t bother to look for upgrades or, the more adventurous ones, run off of HEAD. Both of these are less than ideal and can expose a MediaWiki site to security risks. One of the things I would like to do as Bugmeister is help get MediaWiki back on a regular schedule. We used to have quarterly releases of MediaWiki, so maybe we can shoot for two (or three???) releases this year. I’ve talked to a couple of people and, assuming we can manage to get 1.17 out the door in the next few weeks, it should be possible to get 1.18 out by the middle of the year. Now, don’t think of this as an official announcement, I’m just expressing my personal hopes and goals. And we do need to get 1.17 out soon. But I hope to get 1.18 out the door by July.

After a year working on MediaWiki projects at Wikimedia, I’m filling the position of Wikimedia Bugmeister. I’ve had a lot of fun in the MediaWiki community over the past year and grown to really appreciate the skills of the MediaWiki developers and, in this new role, I hope to work with the the developer community and the users of MediaWiki and Wikipedia to make great things happen. “Make great things happen” sounds kind of cliched, but I really believe that Wikimedia is a nexus of a lot that is happening in Free Culture, so there is great potential in everything that happens here. In my new role as Bugmeister, I’ll have some influence on the tack that Wikimedia takes and the problems that are addressed. I’m excited about the responsibility and have great hopes for what happens next.

$14m might sound like enough money to run a popular website, but for comparison, I work as a sysadmin at a tiny, tiny publishing company with more money and staff just in our department than that to do *almost nothing* compared to what WMF achieves. WMF is an INCREDIBLY efficient organisation.

Heard on WikiTech-l in a thread about better wiki editors. Just something to keep in mind when you’re getting annoyed with the fundraiser. (UPDATE: More interesting discussion on Non-Profits (especially free culture ones like the WMF and FSF) and compensation.)

This past week, my status updates on IdenTwiBook were filled with references to Gdansk and WikiMania. This was my first year to attend the conference and my mother asked me: “Are you going to blog a bit for us lay folks about your trip to Poland and what you did there? (You know, plain language, no computer talk.)”

Since I was a newbie this year, perhaps I can offer a newbie’s perspective that non-technical people (like my mom) will appreciate. Here’s a try.

WikiMania is a conference for anyone involved in Wikipedia. Anyone includes the casual reader, the comma-splice-fixer, the extension author, Administrators, Bureaucrats, and, of course, MediaWiki developers like myself. Ultimately, though, WikiMania is a celebration of Free Culture, from the freely available knowledge and the freely licensed videos, photographs and music that Wikipedia holds to the freely available software that it runs on.

Gdansk, the birthplace of solidarity and the city for this year’s conference, led to the theme “Freedom of Knowledge in the City of Freedom” — an especially apropos title for those who take such freedom seriously.

For three days, we gathered for meetings, talks and socializing at Gdansk’s Polish Baltic Philharmonic. The venue (aside from the lack of air conditioning for this Baltic Sea-side city) was incredible.

Aside from the meetings, there were a couple of cultural events. The first was an impressive concert celebrating Władysław Szpilman (author of “The Pianist”) by the Baltic Philharmonic. This was particularly fascinating for me because, while laptops peppered the crowd at every other event, the concert was free of them. Listening to the music and watching the musicians just required too much of a person’s attention.

The second cultural event was a premiere screening of “Truth in Numbers?” While I personally thought it was a pretty decent documentary, many people thought that the makers gave far too much screen time to Wikipedia’s critics. It was yet another example of the sort of conflict we see almost every day as authoritative sources like newspapers or cultural gatekeepers like the MPAA and RIAA come head to head with amateurs who are using the Internet to freely and widly diseminate their efforts and collaborate with others.

So that’s it. WikiMania is a gathering of free culture fanatics. Their mania, especially while there, is infectious.

Over the past couple of weeks, I’ve been working on MediaWiki’s search functionality. First, I fixed a bug I found with Unicode normalization in the lucene-search extension used by Wikipedia itself. Later, I adapted the normalization that Chinese and Japanese language MediaWiki’s have been doing for full-width characters to all languages. If you’re like me when I started this, you don’t really have a clue what I mean when I say “Fullwidth Latin Characters” even if you know what Unicode is. If you’re familiar with Unicode, you know that the first section is compatible with ASCII. The alphabetical characters contained here are called “halfwidth”. They don’t mix easily with “fullwidth” Japanese and Chinese characters. In order to accomodate the mixture of latinate languages (like English) with these ideogram-based languages there are fullwidth forms available. These fullwidth forms (ＡＢＣ vs ABC) occupy a different area of the Unicode character set and, if your software doesn’t normalize them, then searching for “Mark” won’t match “Ｍａｒｋ”. (And, if you look over the Wikipedia page titled “Latin characters in Unicode”, you’ll find several copies of the alphabet in Unicode space.)

(This is copied from the first part of last week’s weekly report. I’ve copied it here since more people may be interested in what it feels like to work in a highly visible open source project.)

I’m halfway through a 3 month contract (a relationship with the WikiMedia Foundation that I hope to continue) and it seems like a good place to write up some of my lessons learned. Too many times I haven’t spent enough time going over my code. Or, when I’m refactoring someone else’s code, I don’t look over it enough. Sometimes this is due to my own inexperience with the MW code base: If I had more experience, I would have a better idea from just looking at the code that dieUsage() was being used incorrectly. Still, experience can be created, and when it is experience you lack, the effort to create experience must be made. This is where testing comes in. More than once while refactoring the UploadChunks api, Tim has just looked at the code and told me I wasn’t testing it properly. The same could be said for more trivial things like whitespace. Mixing whitespace commits with more substantive changes as well as just the way Emacs formats the code by default have irritated other developers. From this experience, I think I need to borrow an idea from Atul_Gawande’s Checklist Manifesto and make — gasp — a checklist. The checklist is the first step, but the second would be automating as much of the checklist as possible and creating a pre-commit hook that I could use to check my own code. From my experience, here are some ideas for a checklist:

• Whitespace use.
  • Tabs, not spaces
  • avoiding spaces for indention and lining up code
  • spaces around parens
  • etc.
• Code correctness.
  • Ensure each exit point has a test. (Since I prefer to write unit tests for my code, noting a test for each exit point in a code comment would take care of this. An automated check would just verify that the exit point has a notation.)
  • Make sure no E_STRICT warnings are thrown during tests

Note that any automation here is going to fail to really enforce anything: it is impossible to verify code-correctness completely without actually running the code. Instead the idea is not to verify that the code is bug-free, but just to make sure that I’ve caught the things that more experienced MW developers aren’t going to roll their eyes at. That said, the number of developers actively working on MW code as well as the CodeReview extension on mediawiki.org make developing MW code an overwhelmingly positive experience. These things mean that — combined with the depth of PHP and MediaWiki knowledge and care that people take — problems see the light of day really quickly. When your code is likely to end up on one of the top-10 sites on the Internet, this amount of care is crucial.

Last night, I signed up for a three month contract with the Wikimedia Foundation (WMF), set to begin in January. I’ll be working offsite with Tim Starling as my mentor. My primary duties will be to “Support MediaWiki Code Review and Release Management process”, something I feel especially well suited for. The WMF has been working to formalize the process (see, for example Special:Code/MediaWiki) and I’ll be helping with take it further. I have plenty of ideas, but my first task will be integrating myself into the environment. I’m pretty excited to have a full-time position on such a high profile piece of Free Software. Last night, I started reading “Shop Class as Soulcraft” and, while my work is not anywhere as tactile as the work Matthew Crawford talks about in his book, I’ve found that working on Open Source software provides me with a similar sense of accomplishment. It gives me something I can point to and say “I did that” or “I fixed that”. And, with MediaWiki, the thing I can point to becomes something that much more recognisable to others. “You know Wikipedia?” I’ll ask, “That’s what I work on.” (Oh, and if you want, you can throw a few in the pot to help pay the rent for the WMF.)