Entries in Life

Last night, I was able to get hierarchical administration going. That means that we’ll be able to let people admin their own domains for email. They’ll be able to add mailboxes as needed.

One worry about this, though. Quotas! We have to enforce quotas per domain. Right now the only way I can think of doing this is cron jobs since, to my knowledge, Cyrus IMAPd doesn’t allow you to set quotas for a group of users.

I also spent a little time looking at the current non-functionality of PHP but decided to let it go for now since we aren’t using it.

I did try chatting with Sheila, but she and Mert were busy chatting it up.

It was 3:00am when I finally finished up last night. Alexis is being really helpful, and when I woke up this morning my clothes were ironed!

BTW, if you are the sysadmin for more than one Solaris box, you would benefit from learning to create installation packages. For a step-by-step recipe, see sunfreeware.com’s pkgadd instructions.

I personally, prefer to use Debian‘s package building utilities and then convert the resulting .deb into a SVR4 installation package using alien. I’ll write more on exactly how to do that later.

Hmmm…. Today, while putting together a Solaris PKG file for NetBackup, I noticed that their installation puts a few files world-writable under /usr/openv/java. Also, it most, if not all, of the java package owned by uid 1055. Yuck! Hey, Veritas! Fix your installation!

One of the great things about being a sysadmin is that when a friend needs technical help, you have the resources available to help them out. For example, my church, St Basil’s just got a website. It isn’t much now, but since I started hosting it here, we have about 50 times as much power available.

I’ll be re-designing soon, of course.

So, what needs to be done to get everyhost up and running? Jeff has a lot of the front-end stuff going, so we don’t need to worry about that, but some of what he wants to do is dependent on the backend — the stuff of sysadmin work.

• Install 2.4.13 kernel with reiserfs quota patches. I’ve got reiserfs going on the system with 2.2.19 kernel, but there doesn’t seem to be quota patches availible for the 2.2 kernels. That, and I read over at Linux Weekly News that they finally got the Memory Management working properly under 2.4.12 .
• Complete Cyrus authentication off of SASL/LDAP. The default debian package doesn’t have the latest Cyrus IMAPd, nor does it have SASL.
• Complete Exim/LDAP mail routing. I’ve started on this and the basics seem to work. A secondary thing to do is get mailman working with virtual domains. This may be as easy as changing MAILMAN_HOME for every domain.
• Copy over the mod_perl installation from our production server.
• If I have time: install the updated acmemail. This is supposed to have a bunch improvements (like addressbooks) that we want.

I’ll try to add more as I think of it, but for now you can see what I’ve actually accomplished over here.

Jeff and I are trying to make a go of everyhost.com. There seem to be a ton of hosting services, but I think low prices, good service, and word-of-mouth will do wonders. That, and the fact that Jeff hasn’t been able to find work convienent to where he lives.

But, I’ve also got a lot of work I need to get done on the house before Christmas, when my parents are coming down. So I’ve told Alexis that I’ll work on everyhost.com until Thanksgiving and then I’ll work on the house.

Here is a facinating story about falling off a cliff and surviving.

Documentation is very important. I started a new SysAdmin gig a couple of months ago and the people here did a good job of documentation. A lot is documented about the systems themselves and what sort of maintenance contracts we have and that sort of thing. All this is good stuff.

But: What is not documented is the relationships and dependencies between the various sites at this company (at least on the Unix side of the house). They are spread out all over the place: Canada, India, Texas, Louisiana, D.C.

The problem comes in because the administration for DNS and Sendmail was done without documentation.

Then, the time came to upgrade DNS. Management got wind of this problem and decided that this was a problem of some urgency. Nevermind that their main DNS and mailserver was running an un-patched copy of Solaris with the RPC portmapper open to the world — this problem needed to be fixed now.

The first time through, I discovered that they were depending on internal MX records in DNS to do mail routing. Uh… wrong! So, I prepared to take out the internal MX records. However, this meant that I had to change the sendmail configuration. Since they were running an old, unpatched copy of that, I decided to upgrade sendmail as well. I set up a mailertable and tried to get all the internal MX records into it. In the process, I discovered some relatively unknown machines running SMTP. You’d think they’d want to get rid of them if no one knew about them, eh? But no, the political climate (and some special people) guaranteed that they would stay.

I was able to clean up DNS a bit as a result of this upgrade. I had to; the new bind was far more sensitive about configuration problems than the older bind.

After extensive testing, I put the changes in place. It took longer than expected — things always do — but it got done.

Oops! There was no checklist of things to make sure that everything was done right (and this was a rush project, so there was no time to create one), so 6000 users lost their mail for about 12 hours.

Of course, a bigger deal was made of it than was necessary. It was a big deal, but really, no one believed the specter of lost sales of a nuclear power plant because email was down.

Finally, though, all the problems were fixed. What were the lessons I learned?

• Document everything. For your sake and the sake of the person who comes after you. Especially document dependencies. People shouldn’t be able to claim grief if you had no way of knowing about it. If it isn’t documented, it doesn’t exist.
• Make sure you have management’s support. You’ll need these guys saying I gave him the go ahead if something goes wrong.
• Try to get as much information about the changes as you can. Test the information you have. Test it again.
• Get someone else to review what you are doing if you can. You might miss something.

“Never do anything twice” That should be the mantra of all system administrators. Whatever you do, automate it. If it is installing a piece of software or the steps that you have to go through to add a user, automate it. On Unix, various tools exist to help. When it comes to managing the configuration of multiple machines, CFengine stands out. Its meta-language allows you to actions to take on various classes of machines. You can create classes (e.g. web-servers) and have packages installed on those machines and various configuration tweaks made. It really saves time because it helps you document all that you’ve done to set up a package or service. For compiling software and installing binaries, there are different methods available depending on your operating system. Solaris has a package system, but it installs software in funny places (usually something like /opt/packagename) and there is no obvious, easy way to make your own packages. RedHat Linux uses RPMs which at least learned the fallacy of putting every package in its own special hierarchy, but they didn’t make producing RPMs very easy. They use some sort of meta scripting language that is yet another thing to learn. At least the packaging system is free, though. The various free BSDs (FreeBSD, NetBSD, and OpenBSD) have what they call a ports system. This is based on Makefile, so you don’t have to learn anything if you already know about Makefiles, but it can be difficult to get everything right. For example, after you’ve gotten the whole thing to compile and install correctly, you may have to go through a few iterations of the install to produce a binary package so that you can ensure that you have all the files included and all the extra steps taken care of. The best system I’ve seen yet is Debian’s. They use Makefiles plus some scripts to simplify things. The scripts can take care of 90% of the work in most cases, and in those few that they don’t, they greatly simplify things. When it comes to installation, they have fakeroot which ensures that they’ve captured all the files that are installed. Another good thing about Debian is their Apt protocol. Apt will take a debian package, grab all dependencies from the Internet (optionally compiling them) and install everything. It can be done totally non-interactively — a huge benefit. The driving force behind good system administration, as with good programming, is laziness. As the SysAdmin, you have great power available to you to automate a lot of what you do. Use it.

Training seems to be a big thing. Everybody knows this, so we’ve got some problems with scammers. So, if you are going to try to start your career in IT, how do you know that you aren’t being taken advantage of? The first way, of course, is reputation. Does the school or training center have a good reputation? Most acredited colleges and universities now offer courses and certification training and, while these may be more expensive than other options, they would be my first choice.

It is still important to keep in mind what the return is for any training or education that you pursue. Don’t think that a four year degree guarentee’s you a job. It doesn’t. Just the other day, I shared the bus with a recent Electrical Engineering graduate who was working in the mail room of a local company because he couldn’t find a job. If you plan to go to college, take advantage of any internships you can. If you go the certification and training route, don’t expect that your training or cert alone is going to get you more than a helpdesk job.

Still, it is hard to tell. We still have a labor shortage in IT, so some companies may be willing to do OTJ for someone who only has a piece of paper.