Disk Error

[Slicker]

Installed a new 42U rack which shouldn't have affected the Collatz server which is located elsewhere, but someone (me) bumped the power cord to the server while sliding the rack into place and crashed the server. It came back up with hard drive failure. I'm working on it now.

[Slicker]

All is back to normal now.

[Slicker]

And down again... Multiple DRDY ERR messages. Looks like there may be some bad blocks on one of the drives.

[Slicker]

Looks like the boot drive is toast. I have a spare drive, but it is going to take a little while to install Debian, Apache, PHP, MySQL, etc. and get Collatz back up and running. I doubt that it will be a one evening job. My guess is it may not be running until sometime this weekend.

[DrBob]

Do you believe the DB is intact or should we kill all work in progress?

[Slicker]

The DB appears to be intact and I also have the last nightly backup. The bigger question is whether the upload/download folders are corrupted. I kicked off a backup this a.m. and am crossing my fingers that it ran OK. I'll find out tonight.

[Slicker]

It isn't normal yet, but it is running. Unload your Collatz caches while you can.

The boot drive will get replaced this weekend as I don't have the time tonight or tomorrow night to work on it. The work generator is turned off since I don't want to chance corrupting 50K files which would take way too long to sift through and resubmit them if there is another problem between now and when I get it swapped out.

Multiple backups have been done, so worst case, if the server crashes again, I at least know who had what WUs and will be able to grant credit even if they can't return them - or at least that's the plan. Had to run fsck repeatedly before the errors supposedly are cleared and it now boots but still complains the HDD needs to be replaced.

Good thing I didn't have anything planned for this weekend except honey-do's.