test-sometime-before-production

There is, for those of you not geeky enough to know, a new fad sweeping the world of software development. It’s called “Extreme Programming” or “XP” for short. One of the core concepts of XP is testing, specifically test first design or test driven development (TDD). So, all this is well and good for new projects, but what about me? I’m stuck with thousands of lines of code written by the previous developer. Not a test in site. Most of it desperately needs to be refactored. Now, your typical XP fanatic would say “So, write the tests before you refactor” and I’m sure that’s the “right” thing to do. But I don’t. Instead, I just dive in, refactor away and then think “Hey, this is the big money maker. I need to test!” And then I remember (in addition to all the TDD hype I’ve been hearing) that you should never do something twice when you can write a program. So, today I wrote a bunch of tests. And found a bunch of bugs. And saved myself a bunch of grief. And, quite possibly, save the firm some money. I should do this more often!

YAPC: Socializing and Learning

Sunday, I arrived in Toronto for YAPC. Getting into Canada was actually more difficult than getting out. This is mostly because the gate agent for US Air Express didn’t realize that people traveling by air between the two countries don’t have to have a passport until 2006. If YAPC is in Canada again next year, I will definitely have a passport ready. But after a delayed flight on a small plane, I did finally arrive. Instead of using the $300/night hotels close by, I stayed in the hotel/dorm that YAPC was held in. Which means that there were some summer students around. Like the freshman I overheard in the elevator: “There must be a Star Trek convention or something going on this week.” (“Yeah, there are a lot of Dorks at YAPC” I did not say.) Since I arrived late, no one was around, so I asked about a close theater and headed off to see “Batman Begins” (which is far superior to every other comic-book superhero movie I’ve seen). The next day YAPC began. One of the things I was hoping to find was some information on project planning. This is a weak area for me. Before Larry even gave the keynote, I met a consultant who recommended “Planning Extreme Programming”. Luckily, this is also a small book — about 200 pages — so based on the reviews I read and the consultant’s recommendation, I expect it to be packed with information rather than filler. I attended a few talks on testing and learned a lot here that will be useful as I implement the testing infrastructure where I work. Specifically “one test is better than none, and two is better than one” provides motivation to start getting at least one test written for each module, each program I write — just to verify that it loads properly. After that, I’ll add some simple functionality tests. I also learned about or was reminded of some Perl modules that are available on CPAN that will be useful to me as I begin testing more. Tuesday, I heard a talk by Steve Purkis of MultiMAP about making a MySQL database “Highly Available”. Their solution is highly Perl-specific, and I found from his talk some really good resources like DRDB. He also confirmed my suspicion that our current MySQL configuration has to go. Unfortunately, it looks like I’ll have to do some home-brew synchronization work. Why, oh why, doesn’t this JUST WORK?!? Oh, that’s right, I’m using MySQL instead of PostgreSQL (which has working clustering, instead of just the promise of it. ) Tuesday night there was a dinner cruise on the lake. And I met Ricardo Signes (rjbs), the guy who wrote the code behind de.lirio.us, — a site that is a clone of the O’Reilly-funded de.li.cio.us — de.lirio.us made a really big splash when it went public because people liked de.li.cio.us but the code for de.lirio.us is open source. That and the fact that whoever is running de.li.cio.us made a big stink. (To be clear, rjbs isn’t running the de.lirio.us site.) And I’m finally beginning to feel like Pennsylvania is a pretty decent place to live. While Lancaster County isn’t a hotbed of technology, we have people like rjbs and mjd in the Philly area and big Perl shops like pobox.com here as well. Wednesday, I got the most out of the lightening talks. Two projects were announced there: JSAN and AnnoCPAN. JSAN hopes to be for JavaScript what CPAN is for Perl. You can even load Javascript “modules” directly from JSAN. Hopefully people will start to use this more and we’ll all get some nice tools to use for AJAX. I’m jealous of AnnoCPAN. I was going to start putting POD pages up on perlwiki.org so that people could add to them or fix them as needed. Oh well, AnnoCPAN does look pretty nice. And annotation of documentation is probably better than rewriting it. Jim Keenan’s talk on Phalanx encouraged me to get with the Harrisburg.pm people at the conference about starting in. I met Jim two weeks ago at the book release party, so I had an idea of what he was up to, but his talk really pushed me over the edge. Hopefully we’ll be able to get some great social hacking happening. Finally, here’s a list of CPAN modules I learned about (or was reminded of) while at YAPC. Each of these is worth a look.

  • Template::Test — Verify that the logic in my templates (which I strive to keep simple) is working.
  • Devel::Cover — Produces HTML output that can provide incentive to write more tests. The HTML output gives “red-light, green-light” visualization for what is happening with tests. If I incorporate this with the testing infrastructure that I’m putting together, other people (i.e. my boss) should be able to see my progress over time as far as writing test cases that cover the functionality we have.
  • Class::DBI::DataMigration — Dan Friedman’s effort to distill his experience on multiple migration projects to something that is easy to set up and use. Since I’ve been doing a lot of this by hand up till now, this tool will come in handy.
  • DBD::Mock — I learned how useful this tool could be yesterday for testing the sort of functionality you want to use in cases of, say, database failure. Since the site doesn’t degrade very will in case of a failure, I should be able to use this tool to come up with and test solutions.
  • SQL::Translator — Translating between Database’s different dialects. And it also gives you nice tables of the schema.

Wonderful world of Free Software

Did I ever tell you that I love Free Software? I do. I love the fact that, in a meeting today about spam and the prevention of it, there was talk of appliances. The NT admins pointed out that these appliances use the same software we are using, but just put it inside a locked box and give you an Web-based interface. But, since the software is freely available, we already use it. And we’ve already adapted it to our needs in ways that the appliance couldn’t be used. But, that’s just part of the story. Did I ever tell you that I love Emacs? I do. I love the way I can run the same operating enviornment on Windows, Linux, or Mac OS. I love the way I can adapt Emacs to my every need. For example, at work I have to sort through mail and look for spam or phrases within spam that can be blocked. I wrote up a nifty bit of ELisp so that whenever I hit a spam, it grabs the most easily blockable parts (URLs, email addresses) and helps me find other pieces. It has made my job that much easier. And, since Emacs is free software, it has grown and evolved with the input and contributions of all its users. Emacs is about 20 years old, but it is one of the most powerful enviornments that you can use for day-to-day computing. I use it to read email, edit code, take notes, prepare documentation … I’m using it right now to type this entry. Better, with Emacs and Free Software in general, if I run into a problem with the software I can fix it. Sure, I’ll send email to the author, but sometimes he’s busy. I can fix it. That’s what I’ve done with the RSS reader in Emacs. I’ve run into problems with the XML parsing code with RSS, but I was able to fix it. And, get this, since Emacs is free software, I was able to contribute my code back, safe in the knowledge that other’s wouldn’t have the same problem. But it’s not just Emacs. Its a whole slew of software. I use Mail::Audit::List for sorting my mailing lists into separate folders. Today, I found a bug in it, fixed it, and submitted the patch. This power, the ability to fix your own problems, or, lacking the skill to fix them, appeal to the community of fellow user for a fix, is awesome. Proprietary software doesn’t give you that power. At the most, you can appeal to your fellow users for a work-around. Rarely will you find a fix. You’re gonna have to wait for the vendor to come out with a band-aid. And then, there is a chance that they won’t understand the problem (and so won’t give a complete fix) or they’ll fail to fix the problem at all. Does Free Software have its problems? Sure. But you can fix them.

Dave talks about the Python and leads into programming using an outliner. This is very interesting to me. I love Perl (the un-Python) and it is interesting that he talks about the advantages of programming using an outliner because this is what led me to contribute the outline-mode enhancements to emacscperl-mode.el.

Emacs is actually a pretty nifty folding editor. Here is how to use it to edit Perl code in this fashion.

I hate sleep — it means that work takes up more of my life than I want it to. Sleep is good for my kids (so I get a few hours to myself), but, overall, I hate it.

I perused the archives for the blogger list today and came up with the specs for a couple more calls so I’ll be able to make it possible for you to go through posted messages on your blog with emacs and edit them.

Then I had a great idea for integrating htmlize.el in the blogger buffer so that you can add faces and whatever to your messages to make them pretty and stuff.

Hey, with Emacs21, you could drag-n-drop pictures into your emacs buffer and have them automatically posted to your site. Coolness!

That’s the real trouble with Emacs — too many navel-gazing programmers intent on programming. Not enough navel-gazing programmers intent on blogging.

Thoughts about Package Formats

“Never do anything twice” That should be the mantra of all system administrators. Whatever you do, automate it. If it is installing a piece of software or the steps that you have to go through to add a user, automate it. On Unix, various tools exist to help. When it comes to managing the configuration of multiple machines, CFengine stands out. Its meta-language allows you to actions to take on various classes of machines. You can create classes (e.g. web-servers) and have packages installed on those machines and various configuration tweaks made. It really saves time because it helps you document all that you’ve done to set up a package or service. For compiling software and installing binaries, there are different methods available depending on your operating system. Solaris has a package system, but it installs software in funny places (usually something like /opt/packagename) and there is no obvious, easy way to make your own packages. RedHat Linux uses RPMs which at least learned the fallacy of putting every package in its own special hierarchy, but they didn’t make producing RPMs very easy. They use some sort of meta scripting language that is yet another thing to learn. At least the packaging system is free, though. The various free BSDs (FreeBSD, NetBSD, and OpenBSD) have what they call a ports system. This is based on Makefile, so you don’t have to learn anything if you already know about Makefiles, but it can be difficult to get everything right. For example, after you’ve gotten the whole thing to compile and install correctly, you may have to go through a few iterations of the install to produce a binary package so that you can ensure that you have all the files included and all the extra steps taken care of. The best system I’ve seen yet is Debian’s. They use Makefiles plus some scripts to simplify things. The scripts can take care of 90% of the work in most cases, and in those few that they don’t, they greatly simplify things. When it comes to installation, they have fakeroot which ensures that they’ve captured all the files that are installed. Another good thing about Debian is their Apt protocol. Apt will take a debian package, grab all dependencies from the Internet (optionally compiling them) and install everything. It can be done totally non-interactively — a huge benefit. The driving force behind good system administration, as with good programming, is laziness. As the SysAdmin, you have great power available to you to automate a lot of what you do. Use it.