Collatz is down

[EmSti]

Looks like things stopped about 20:35 Eastern.

[Slicker]

Naturally,I'm away for the weekend taking advantage of the MLK holiday. Looks like it is server related as I can VPN in to my home network but there's no response to ping or rsh to the Collatz server. That's the good news. The bad news is that the off site backups don't appear to have worked for a while so I hope the backup (a.k.a. previous Collatz server) has a good copy from the nightly job and/or that I can get the current server back up and running w/o losing much data. I'll know more after I get back home later this evening.

[Duke of Buckingham]

I have 2 tasks of Collatz that are now outdated, should I abort them or let it go? :confused:

They were finished on time but the fact Collatz have been down for so kong surpassed the expiration date. :((

[shiva]

I have 2 tasks of Collatz that are now outdated, should I abort them or let it go? :confused:

They were finished on time but the fact Collatz have been down for so kong surpassed the expiration date. :((I would keep them, I think you will get credit for them

[Slicker]

The server is two hundred miles away and 100% unresponsive. I don't have a magic crystal ball with me to read the future so I have no idea whether it can be repaired or whether I'll need to set up a new BOINC server from scratch. Until I figure that out, it would be like me trying to guess how much Obama will raise my taxes next year. I just really don't know.

[Duke of Buckingham]

The server is two hundred miles away and 100% unresponsive. I don't have a magic crystal ball with me to read the future so I have no idea whether it can be repaired or whether I'll need to set up a new BOINC server from scratch. Until I figure that out, it would be like me trying to guess how much Obama will raise my taxes next year. I just really don't know.

I hope computers are more trustable than politicians or are they lying to me for all those years?

The computers of course because about the politicians I have no doubth that they are lying, they never learned how to speak the truth anyway.

[EmSti]

I know the feeling - how long how long how long....

Once asked a boss to step out into the hallway and away from where we were troubleshooting so we could talk. Then closed the door when he went through the doorway and we went back to work, in peace. To his credit, he took the hint and left us be.

Good luck with it, hope it turns out well for us, if not oh well. It has only been a few days.

[Slicker]

The server drives are pretty much toast. Only one is recognized at boot and that doesn't help much when it is in a raid 5 array. Having just replaced two of them a several months back, I'm thinking it may also be controller related so the controller and all drives are being replaced. I'll probably add a couple hot spares this time just to be safe. Unfortunately, my 31 year old furnace is being replaced tomorrow so I can't run over to MicroCenter, TigerDirect, or CDW to pick up the new hardware until that's done. Also, Docsis 2 to Docsis 3 upgrade should also take place between now and the time the server is ready to go back online so the network speed should be better when the server does come back online. Regardless, it will probably be a couple weeks to rebuild the server and get all the BOINC and Collatz software rebuilt/re-installed.

[shiva]

ooffff well, one can only do what one can. Good luck with the rebuild Slicker!

[EmSti]

I have 130 gpu wus left to crunch, should I suspend them now or crunch them until done?

I have 1850+ gpu wus completed and waiting to upload. I will just let them sit until servers are back online and see what happens.

[shiva]

I'm just going to let mine run, don't know how many I have but Slicker will do everything he can to get everyone their credit.

[Slicker]

My suggestion would be to abort any WUs that aren't already completed. I should know within the next day or two what to do with the completed ones. That all hinges on how recent a restore I can do which isn't looking real good right now. Looks like the disk corruption occurred over several days getting worse and worse so the most recent backups may not be any good. Now it comes down to whether I can read the workunit or results tables from the bad drive and whether the user and host records associated with the workunit and results can also be read. If I were a betting man, I'd bet on something else.

[shiva]

well I think I will take the advise of the master :) at this point. I will be wishing you well through your current challenges.

[STE\/E]

Just give us all a Billion Credits to make up for any loss of Credits ... :D

[Bryan]

Just give us all a Billion Credits to make up for any loss of Credits ... :D

+1 I like the way you think and I'm sure I had about that amount in my cache =))

[conf]

Have about 100 WUs completed and they all try to upload and the deadline is near.
Maybe we can get credits for the WUs sent from Collatz before it has gone down.
I really want to get them out of my uploads cause I dont see anything else and it slows the system.

[EmSti]

Posted on the collatz site: "The Collatz server is currently down due to a hard drive controller failure. New hardware should arrive in the next day or two. A complete rebuild of the operating system, BOINC, and the Collatz software and project setup will then need to be done prior to the project coming back online. As this will be a complete rebuild, there is no sense hanging on to any completed workunits at this time. So, go ahead and abort any workunits you have whether they are complete or not. I expect the project will remain down for the next 10-15 days while I set it all back up and test it. Sorry for any inconvenience. And yes, I will make up for the lost credits once everything is back up and running. "

[Fire$torm]

Hey,

FYI: Some of our newer members may not realize that our team member Slicker is the project admin of Collatz. He always posts here on the forums regarding any issues with Collatz, when RL allows.

[Duke of Buckingham]

Slicker ... Collatz ??? :confused:

Sorry for all the bad words I have posted, I am new in here ...:o)

Joking of course, I really hope you can solve this problem as soon as possible for us to get to our usually crunching at Collatz Slicker. :o

Everyday I go the project to see if it came back, I had abort some tasks when the project had problems. It were my computer problems maybe my Hard Drive that is pushing to much from the PSU and all system is coming down. I have now a new PSU and hope to get a decent Hard Drive as soon I understand what is the best I can achieve. :p

[STE\/E]

Project is down again, must have been down for 6 Hr's now, I've only got about an Hr left to run then will have to switch back over to Moo on that Box ...

[STE\/E]

Back up again ... :)

[Fire$torm]

I think Slicker did an update to the apps based on that survey he had posted on the Collatz forums. Hence the reason for shutting down the server.

[Slicker]

I think Slicker did an update to the apps based on that survey he had posted on the Collatz forums. Hence the reason for shutting down the server.

It also does nightly backups, but that shouldn't take more than a few minutes. Every few weeks I export the results and re-optimize the database tables as MySQL pretty much sucks as as soon as it grows to a size greater than the physical RAM. (If it has to reside in RAM, how then to me it isn't a real database. A real database should be able to be 10-50 times the size of physical RAM and still run OK. Oracle and SQL Server can both do that. MySQL doesn't even come close - at least not if you more than 3 concurrent users.)

[Fire$torm]

I'm having a problem, can't get any nVidia work on any of my boxes.... :((

Never Mind... PEBKAC

[EmSti]

I am in the same boat. My problem started right after the preferences saving was fixed. After the fix, collatz sent AMD instead of Nividia (there is an AMD, but it is doing DiRT). Fixed the preferences from all to Nividia/cuda, aborted the AMD wus and nothing since. I have waited over 24 hours now, thought I may have pissed the servers with the abort. Attempting reattach now.

[EmSti]

Reattach didn't work.

1027 2/20/2014 9:50:41 AM Fetching configuration file from http://boinc.thesonntags.com/collatz/get_project_config.php
1028 http://boinc.thesonntags.com/collatz/ 2/20/2014 9:50:49 AM Master file download succeeded
1029 http://boinc.thesonntags.com/collatz/ 2/20/2014 9:50:54 AM Sending scheduler request: Project initialization.
1030 http://boinc.thesonntags.com/collatz/ 2/20/2014 9:50:54 AM Requesting new tasks for CPU and NVIDIA and ATI
1031 Collatz Conjecture 2/20/2014 9:50:56 AM Scheduler request completed: got 0 new tasks
1032 Collatz Conjecture 2/20/2014 9:50:56 AM No tasks sent
1033 Collatz Conjecture 2/20/2014 9:50:56 AM Tasks for CPU are available, but your preferences are set to not accept them
1034 Collatz Conjecture 2/20/2014 9:50:56 AM Tasks for AMD/ATI GPU are available, but your preferences are set to not accept them
1035 Collatz Conjecture 2/20/2014 9:50:56 AM Tasks for Intel GPU are available, but your preferences are set to not accept them
1036 Collatz Conjecture 2/20/2014 9:50:56 AM New computer location: school

[FourOh]

I'm having a problem, can't get any nVidia work on any of my boxes.... :((

Can't get AMD work either boooooo. Guess I'll work on Milkyway.

[Al]

I'm getting no task available too. Fortunately I have 2 days on board.

[EmSti]

hhhmmmm collatz server status is all green with tasks to send.

[Fire$torm]

My solution was to allow for all forms of work and allowed "If no work for selected applications is available, accept work from other applications?" All cards have plenty of work atm.

Edit: Btw, It's all OpenCL work.

[EmSti]

Problem solved: I spooted this comment from Slicker - "I found a bug that when multiple venues existed, if any venue was set to ignore a plan class, then all venues would ignore the plan class. A fix has been released. Let me know if you run into any issues."

That comment came in after my last update on preferences. I deleted all my venues. Changed the default to Nvidia/cuda only, changed agents to default, did project update and I got wus again. I would recommend deleting as reseting you venues.

[Fire$torm]

Problem solved: I spooted this comment from Slicker - "I found a bug that when multiple venues existed, if any venue was set to ignore a plan class, then all venues would ignore the plan class. A fix has been released. Let me know if you run into any issues."

That comment came in after my last update on preferences. I deleted all my venues. Changed the default to Nvidia/cuda only, changed agents to default, did project update and I got wus again. I would recommend deleting as reseting you venues.

Ah, good to know. Thx for the info.

[Fire$torm]

Damn, server is down. Slicker isn't having much luck with that puppy as of late. Hope it's not serious.

[cineon_lut]

Yeah, bummer. App wise it's one of the (if not THE) best written, but the server ain't giving much love this month.


Vic (mobile)

[zombie67]

Looks like collatz has been down since sometime yesterday.

[Slicker]

Looks like collatz has been down since sometime yesterday.
Something is going on with the overnight backups that causes MySQL to not allow connections after it is done. The backup completes but I have to stop and restart mysql in order for it to allow multiple connections. Because of that, the BOINC daemon don't start and since they don't start, the project stays in maintenance mode. This appears to be my year for having every computer I own give me multiple problems. I just got done rebuilding my i7 desktop (hard drive failure) and didn't even have all the software loaded when my i7 laptop screen died. The replacement arrived today and is the wrong fracking one! No backlight and while the resolution is supposed to be HD (and it might be), the frequency is screwed up such that the screen shows up 6 times each about 2" tall. Now the Collatz server is adding to my woes. Maybe I should start mowing lawns for a living. Oh wait. All three of my lawn mowers are currently broken. At least that isn't an issue yet due to the non-stop polar vortex we've had for the last 4 months.

[Bok]

How are you doing the backups? And what if anything does the mysqld.log say? Are you using persistent connections?