As you may or may not have noticed fork.freebsd.cx (which hosts various sites, jails etc.) has had a recurring problem over the past few weeks, where we’re seeing some filesystem corruption which causes the box to panic and reboot. We have a pair of 400GB disks in there (mirrored) attached to a decrepit ATA/33 controller so the resulting filesystem check takes the best part of an hour – longer if there’s a problem and it has to be re-run.
Annoyingly I can’t seem to forcibly recreate the error and so haven’t been able to nail it down. I kind of suspect hardware given how frequent it’s now occurring, but I’m not sure.
Anyway, we plan on replacing this box soon with a shiny new server running Solaris and Xen instead of the current FreeBSD / Jails setup. More on that to come, but as it stands I suggest everyone backs up their stuff and be wary that you might not be able to rely on service such as access to email and websites.
** UPDATE: I think I might have fixed it. We had one particular corrupted inode (that fsck didn’t pick up) which I’ve cleared following yesterday’s outage and so far it’s been OK. Finger’s crossed.