I told you sysadmin work happened when it was least convenient.
It was four o’clock in the afternoon and in the process of diagnosing another email problem, I noticed that we were rejecting email because we were at the limit for child processes. Sendmail creates children to handle email. If it creates too many child proceses, it can overload the machine, so we put a limit on how many children it can create.
Well… it was creating (“forking” is the word we use, actually) children left and right. Why? It seemed to be some sort of attack.
After some troubleshooting, taking out the spam blocks, removing the cap on the number of processes, I decided that there must be some sort of outside cause. I called the security guy here and we started looking at it. Finally, we noticed an overwhelming number of emails from groups.yahoo.com. He suggested that we start discarding that email.
After verifying that he would take the blame, I started discarding the mail. Sure enough, in a few minutes, it cleared up.
Makes you wonder, doesn’t it?
A possible cause: spammers will subscribe individuals to multiple email lists so that they get flooded with email. Perhaps someone did that to us.