More Dirt issues. [Archive]

Fire$torm

08-30-11, 09:09 AM

If you haven't noticed credits for Dirt are not showing up in the stats.

Looks like they are having server problems

Project server status page (http://boinc.freerainbowtables.com/distrrtgen/server_status.php)
distrrtgen_validator1 -- iris -- Disabled
distrrtgen_validator2 -- iris -- Disabled
distrrtgen_validator3 -- iris -- Disabled
distrrtgen_validator4 -- iris -- Disabled
distrrtgen_validator5 -- iris -- Disabled
distrrtgen_validator6 -- iris -- Disabled
distrrtgen_validator7 -- iris -- Disabled
distrrtgen_validator8 -- iris -- Disabled
distrrtgen_work_generator1 -- iris -- Not Running
distrrtgen_work_generator_hybrid -- iris -- Running
distrrtgen_work_generator2 -- iris -- Disabled
distrrtgen_work_generator4 -- iris -- Disabled

Mike029

08-30-11, 09:49 AM

I'm going to suspend what I have left and run something else till they finish with the 280 wu's I have pending. I'm worried that they have a major hardware failure and we lose all our eggs in this one basket. :eek:

zombie67

08-30-11, 10:38 AM

The server status page is not accurately reflecting the server's actual status. No major HW problems. In fact, they just upgraded, and will be doing more upgrades in the near future. And longer running WUs will reduce the server load even further.

https://www.freerainbowtables.com/phpBB3/post18264.html#p18264

We moved to 1 validator, 1 assimilator, and 1 transitioner. bin/status shows them running but the server_status page does not, for reasons not yet known. This move was mostly so we did not crash and burn handling backoffs expiring for clients as well as handling the immediate inrush. Our inbound bandwidth is still climbing likely on the 300k length chains coming back as the 400k length chains should be 50% slower - unless more bandwidth constrained clients are now turning in more work. I'm hoping to see some leveling off at some point and can return daemons as needed.

08-30-11, 02:49 PM

Seeing the same thing. I currently have almost 700 wu's pending. Should really throw off the stats bell curve when they finally validate.

Fire$torm

08-30-11, 04:03 PM

Seeing the same thing. I currently have almost 700 wu's pending. Should really throw off the stats bell curve when they finally validate.

I just counted (3:51 EDT) and I have 361 wu's waiting for validation.

Beerdrinker

08-30-11, 04:14 PM

I just counted (3:51 EDT) and I have 361 wu's waiting for validation.

We are gonna skyrocket once they validate...I could be great if they could put us over the french...! ;)

Mr. Hankey

08-30-11, 05:22 PM

Yea, I got 620 waiting to validate for me so far.

Mumps

08-30-11, 08:29 PM

We are gonna skyrocket once they validate...I could be great if they could put us over the french...! ;)

Hah! Then they'll accuse us of bunkering again! :)

zombie67

08-30-11, 08:34 PM

1100 pending for me so far. Do I win? When it finally breaks loose, it will be a magnificent score day. ;)

valyn42

08-30-11, 11:11 PM

I think you win Z, and I agree with that. 1080 pending for me.:D

Fogle

08-31-11, 07:30 AM

OH OH! the latest task that just validated gave me only 184 credits. Hopefully it's just a fluke, all my WU's are for CPU only. Anybody else seeing this? :(

Fire$torm

08-31-11, 08:56 AM

OH OH! the latest task that just validated gave me only 184 credits. Hopefully it's just a fluke, all my WU's are for CPU only. Anybody else seeing this? :(

Nope. My last set of CPU wu's received 600 credits each.

spingadus

08-31-11, 01:20 PM

Looks like a bug.

https://www.freerainbowtables.com/phpBB3/post18279.html#p18279

Beerdrinker

08-31-11, 02:05 PM

Well, I got granted about 100.000 Credits today. All WU´s OK. 600@piece for the CPU and 3K for GPU tasks.

spingadus

08-31-11, 02:14 PM

I counted about 20 instances of the low credit bug on my account over the last week. That's about 60k credits. Hope it really is fixed.

Fire$torm

08-31-11, 02:53 PM

I counted about 20 instances of the low credit bug on my account over the last week. That's about 60k credits. Hope it really is fixed.

They say that it cannot be fixed but I know that other projects (Like PG & MW) have done this. It is a manual operation, which seems to equal "Not gonna happen".

Mike029

09-01-11, 08:40 AM

I got 1.56 credits for this wu. Going to have to keep an eye on this. Any more like that and I may have to move on till they fix the problem. Only happened once and with a cpu wu.

Name 6291721 ntlm_hybrid2(alpha#1-1,loweralpha#5-5,loweralpha-numeric#2-2,numeric#1-3)#0-0_0_60000x500000 - 59559500000_1
Workunit 9378273
Created 29 Aug 2011 23:43:40 UTC
Sent 30 Aug 2011 0:24:22 UTC
Received 30 Aug 2011 13:50:56 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x0)
Computer ID 21886
Report deadline 31 Aug 2011 0:24:22 UTC
Run time 4,427.51
CPU time 4,092.81
Validate state Valid
Credit 1.56
Application version Distributed Rainbow Table Generator (distrrtgen) v3.36
Stderr output

<core_client_version>6.12.33</core_client_version>
<![CDATA[
<stderr_txt>
09:43:41 (3984): called boinc_finish

</stderr_txt>
]]>

Mike029

09-01-11, 08:49 AM

I'm now up to 550 wu pending. I think it's a big number considering that I only have one Cuda card. Over 90% are gpu wu's. Going to be a nice bump if they ever validate. :eek: Some people have to be well over 1k in pending by now.:-o

zombie67

09-01-11, 10:52 AM

I have over 2200 pending. No big deal. It will all be awarded eventually.

As for the few tasks with small credits, there was a bug in the validator, and has been fixed.

DrPop

09-01-11, 11:42 AM

The weird thing is, I've been crunching it on 10 CPU cores, + a GTX 460, and I haven't been awarded any credit for days; but...I don't see that I have any pending either? That's kind of odd... :-??

spingadus

09-01-11, 02:08 PM

The weird thing is, I've been crunching it on 10 CPU cores, + a GTX 460, and I haven't been awarded any credit for days; but...I don't see that I have any pending either? That's kind of odd... :-??

The pending credit 'view' link off the main account page doesn't show anything. You have to go into the tasks 'view' link and click pending.

DrPop

09-01-11, 02:56 PM

Ah, thanks for that! :D

DrPop

09-01-11, 04:09 PM

I just noticed my CUDA GPU sitting idle. Looks like DiRT has plenty of CPU WUs for me, but their GPU WUs have stopped coming in. I'd recommend everyone allow PG to get some new work on their CUDA rigs if you haven't beat me to it already. ;)

Mike029

09-01-11, 04:57 PM

Cruncher Pete

09-01-11, 07:17 PM

The pending credit 'view' link off the main account page doesn't show anything. You have to go into the tasks 'view' link and click pending.

Unfortunately, this like a lot of other sites do not show you how much credit is in pending. You need to actually access you Account and view your computers on it. In my case, I have over 800 units sitting in "Awaiting Validation" and have not been given a credit for over three days. Hope it will resolve soon and in particular we will not waste our efforts. Still, I am not that trustworthy after all, and have issued a NNW on all my Dirt projects.

zombie67

09-01-11, 08:07 PM

Unfortunately, this like a lot of other sites do not show you how much credit is in pending.

You can thank DA for that.

You need to actually access you Account and view your computers on it. In my case, I have over 800 units sitting in "Awaiting Validation" and have not been given a credit for over three days. Hope it will resolve soon and in particular we will not waste our efforts. Still, I am not that trustworthy after all, and have issued a NNW on all my Dirt projects.

I am curious why you and some others here feel so concerned about this. Has this project ever stolen credits from you? Has it happened at another project, that causes you to think it will happen here? Why so worried?

For my personal experience, I have crunched just about every project ever, and find it hard to come up with an example of much significance.

Cruncher Pete

09-01-11, 09:53 PM

You can thank DA for that.

I am curious why you and some others here feel so concerned about this. Has this project ever stolen credits from you? Has it happened at another project, that causes you to think it will happen here? Why so worried?

For my personal experience, I have crunched just about every project ever, and find it hard to come up with an example of much significance.

@zombie67. To satisfy your curiosity, it is of no consequence to me how or where the credits are shown up. Having said that, what is worrying me is at this stage I have over 800 units in Pending validation that has been waiting for the last three days to be validated.. In all honestly, I can not afford to wait any longer to see if I am going to be credited for these in view (times for submission has expired) of previous times when I lost credits and hoped that in time all will be good. In my experience twice I have worked for credits for days to weeks and on the end the whole lot was wiped. Aqua was one and FreeHAL was another. This is my reason to be cautious.In this light, I suggest that like I did is to stop new work units coming in and if necessary crunch others until this problem is fixed. On the other hand, there is no problem if you keep thrusting this project and just crunch away, so far it has always credited your entitlement as due. I am just being cautious and move to more secure projects whilst it is available and more promising as to credit outcome..Hope that answers my point of view why we are doing things the way we are doing.

Keep on crunching and have a nice day...

zombie67

09-01-11, 10:19 PM

@zombie67. To satisfy your curiosity, it is of no consequence to me how or where the credits are shown up. Having said that, what is worrying me is at this stage I have over 800 units in Pending validation that has been waiting for the last three days to be validated.. In all honestly, I can not afford to wait any longer to see if I am going to be credited for these in view (times for submission has expired) of previous times when I lost credits and hoped that in time all will be good.

Just to be clear, validation does not matter about due dates, if you have returned the task by the due date. Validation could happen a year later and it would still award credits.

Also, the number of days for tasks to be validated was already known in advance, and should not be a surprise. They explained their schedule several days ago. (http://boinc.freerainbowtables.com/distrrtgen/forum_thread.php?id=46) It is not like the number of days are an indication of anything wrong. They are on schedule so far.

Think about it this way. If this were a normal BOINC project, with a quorum of 3, you could be waiting MONTHS for a task to be awarded credit. A week or two is nothing extra ordinary.

In my experience twice I have worked for credits for days to weeks and on the end the whole lot was wiped. Aqua was one and FreeHAL was another. This is my reason to be cautious.In this light, I suggest that like I did is to stop new work units coming in and if necessary crunch others until this problem is fixed. On the other hand, there is no problem if you keep thrusting this project and just crunch away, so far it has always credited your entitlement as due. I am just being cautious and move to more secure projects whilst it is available and more promising as to credit outcome..Hope that answers my point of view why we are doing things the way we are doing.

Keep on crunching and have a nice day...

No need to stop crunching, as the project stopped issuing new work a day or two ago. Any new work out there is just re-issues due to people missing the deadlines. They stopped it for you. ;)

I have been crunching FreeHAL since day 1. Losing free credits (non-CPU) didn't strike me as a problem. As long as everyone crunching the project lost equally, and we did, where is the harm? For AQUA, I had already hit my 5M goal there, so I missed this issue. I thought they restored the credits from a few days previous. Maybe I am confusing it with another project? In any case, everyone lost equally right? *shrug* I guess I just don't see the need to worry. But then again, I am usually crunching for alpha projects. So I am almost always living with the risk of losing credits. Or sometimes never getting any credits at all when they close before ever exporting stats.

Cruncher Pete

09-01-11, 11:52 PM

Just to be clear, validation does not matter about due dates, if you have returned the task by the due date. Validation could happen a year later and it would still award credits.

Also, the number of days for tasks to be validated was already known in advance, and should not be a surprise. They explained their schedule several days ago. (http://boinc.freerainbowtables.com/distrrtgen/forum_thread.php?id=46) It is not like the number of days are an indication of anything wrong. They are on schedule so far.

Think about it this way. If this were a normal BOINC project, with a quorum of 3, you could be waiting MONTHS for a task to be awarded credit. A week or two is nothing extra ordinary.

No need to stop crunching, as the project stopped issuing new work a day or two ago. Any new work out there is just re-issues due to people missing the deadlines. They stopped it for you. ;)

I have been crunching FreeHAL since day 1. Losing free credits (non-CPU) didn't strike me as a problem. As long as everyone crunching the project lost equally, and we did, where is the harm? For AQUA, I had already hit my 5M goal there, so I missed this issue. I thought they restored the credits from a few days previous. Maybe I am confusing it with another project? In any case, everyone lost equally right? *shrug* I guess I just don't see the need to worry. But then again, I am usually crunching for alpha projects. So I am almost always living with the risk of losing credits. Or sometimes never getting any credits at all when they close before ever exporting stats..

You might well be right, but in view of the saying once bitten twice shy, I tend to be careful with projects like this that takes a lot of time to fix their problems or loose all credits or some because of their inability to fix the problem. In the case of FreeHAL I have wasted weeks of computing times whereas others might only have wasted hours, yet we lost the lot, not because of a computer error but because the Admin decided that we are all cheating.(They are the words the Admin used at the time.) Therefore, I tend to be on the cautious side and place all WU's on NNW until the project recovers when something like this happens. It is really only a safeguard for during this time I can crunch an alternative until the problem is resolved....

Keep on crunching and have a nice day...

Fire$torm

09-02-11, 12:12 AM

I am curious why you and some others here feel so concerned about this. Has this project ever stolen credits from you? Has it happened at another project, that causes you to think it will happen here? Why so worried?

For my personal experience, I have crunched just about every project ever, and find it hard to come up with an example of much significance.

I counted about 20 instances of the low credit bug on my account over the last week. That's about 60k credits. Hope it really is fixed.

So ya, at least one person has lost credits. I have not checked my own mainly because if I find it then I'll be irate. And yes, I know it happens which is why I stay away from projects that take this sort of thing for granted. MW usually does a great job of correcting errors which in my opinion is the reasonable thing to do.

NJCaNS

09-02-11, 12:13 AM

Just noticed that SIC put up 15.7 million today vs 7.6 for SUSA. They'll take over first place tomorrow.

http://stats.free-dc.org/stats.php?page=proj&proj=drt

zombie67

09-02-11, 12:25 AM

It is really only a safeguard for during this time I can crunch an alternative until the problem is resolved....

Fair enough reason. I asked and you answered. Thanks.

zombie67

09-02-11, 12:36 AM

So ya, at least one person has lost credits. I have not checked my own mainly because if I find it then I'll be irate. And yes, I know it happens which is why I stay away from projects that take this sort of thing for granted. MW usually does a great job of correcting errors which in my opinion is the reasonable thing to do.

I know 60k sounds a lot. And it *is* for CPU projects.

But for the high paying GPU projects, that is just a rounding error. Even with the validation bug, this project is still paying more than any CUDA alternative.

So what is the alternative (if the bug was still there, which it isn't)? Crunch a lower paying project like PG, which everyone was already even more pissed off at? Why?

Folks, this is a hobby, we do it for *fun*. Mistakes happen from time to time, without malice. They want your help, after all. Why be perpetually worried or pissed off? Have fun with it! :)

zombie67

09-02-11, 12:39 AM

Just noticed that SIC put up 15.7 million today vs 7.6 for SUSA. They'll take over first place tomorrow.

http://stats.free-dc.org/stats.php?page=proj&proj=drt

That assumes a steady availability of work. It was all turned off for a while. What you are just seeing now are just short bursts of output from the validator.

Cruncher Pete

09-02-11, 12:39 AM

Just noticed that SIC put up 15.7 million today vs 7.6 for SUSA. They'll take over first place tomorrow.

http://stats.free-dc.org/stats.php?page=proj&proj=drt

This is another reason why I am worried about my efforts for not being credited for over 800 Units sitting in "Awaiting validation" and no credits for three days. Never the less, I will continue to crunch with my GPU's on this until it is resolved...In the mean time to make sure I will not loose all, I have suspended all CPU units for it for I have other important tasks to perform as well. Just a safety measure...

DrPop

09-02-11, 01:35 AM

I am sure we'll see an uptick when all gets validated...and yes, this is a hobby and it is...oh yeah, it is still fun. Sometimes!!! :D :)) ;)

Fire$torm

09-02-11, 01:47 AM

Just to be clear. The best day I have on Dirt was Aug. 26 with 361,095 credits. That divided by 60K is approx 1/6. To me a 15%+ drop is not a good thing, especially if it was not my doing. I do understand the projects mostly try and that they do want our help and nothing is perfect......

Bryan

09-02-11, 02:17 AM

Shoot, they are just having growing pains. I don't think they understood the magnitude of what was going to happen when they raised the credits equal to the PG of old. The credits have actually gone down a hair ... the 300000 wu took less time to crunch when they 1st came out but still pay the same 3000. I haven't seen a single 400000 wu validated yet so it will be interesting to see if it pays higher since it takes longer to crunch.

zombie67

09-03-11, 11:15 PM

Looks like they turned the validator back on. My bending task count is down from 2k+ to less than 400. Lot's o' credits today! Weee!

Fogle

09-16-11, 08:26 AM

Arrgh! I'm out of DiRT wu's check your settings to be sure your not sitting idle! :mad:

NJCaNS

10-04-11, 11:50 PM

Update: DistrRTgen (Dirt) for Team SUSA just passed 500 million credits today (8th SUSA project to reach this level). This breaks a 4 way tie for second at this team Mega Milestone and puts us into a tie for 1st with SG.

Way to go you Dirt crunchers. :-bd Pour yourselves a cold one. \m/**==

Fire$torm

10-05-11, 12:52 AM

DrPop

10-05-11, 12:57 AM

Why, thank you! :D

Cruncher Pete

10-05-11, 05:23 AM

Oh, I need some of that Coldie, so I better change to Dirt for although some machines are on it, most of my ATI's are on DNETC. I still need to be convinced that they give better credit?

Keep on crunching and have a nice day..
CP

Beerdrinker

10-05-11, 02:52 PM

I am back and forth between PG and DistTRGen for the time....Since Dist can´t come to a solution on how to run - I am busy trying to maintain a decent production!!!

Anyway, CONGRATS TEAM!! I did notice it the other day. But had no time to post...

Mike029

10-06-11, 08:15 AM

We would have done it faster it the project stayed up more often. :cool: Now onto 1B. :p

Fire$torm

10-11-11, 02:33 PM

If ya haven't noticed Dirt is down again. Something is FUBAR with the database as it wasn't exporting stats after the I/O upgrade.... :(

Fire$torm

10-24-11, 03:07 PM

Looks like Dirt is down for the moment.... :(

spingadus

10-25-11, 03:27 AM

Ugh! I just noticed around 90 invalid wu's today that ran to completion and now no work. Many hours of crunching down the drain.

STE\/E

10-25-11, 11:27 AM

Same here, almost 1400 Validation Error Wu's since the 16'th, Suspended the Project for now, not sure if I'll ever start it back up again since this seems to be a reoccurring problem ... :mad:

spingadus

10-25-11, 03:44 PM

STE\/E

10-25-11, 03:55 PM

I doubt it, they never have in the past & they say they have no way of doing it. I wondered why my Credit was down some there but never looked at my Wu's Until somebody sent me a PM at the Project notifying me of the Validation errors. Right now I haven't got any more since they supposedly fixed the problem ...

Beerdrinker

10-25-11, 04:12 PM

Yikes! That's a lot of lost credit. I wonder if they'll re-validate them. I think I may need to find a new nvidia gpu project as well.

I would not bet any money on that.

I have been thinking that same thougth the last couple of days...I think Dist is acting up more than usual these days...

DAD

10-25-11, 06:22 PM

I noticed something with Dirt.. if I used any GPUs that were ONLY SP Math, I ended up with a lot of invalid work (using 8800s and 250s from nvidia)... but if I use any of the 500 cards which do DP math, I rarely get any errors out of Dirt.

trigggl

10-25-11, 06:52 PM

DAD

10-25-11, 07:10 PM

Yeah the 9xxx vs 8xxx cards are basically new bioses :) you can turn pretty much any 8K card into a 9K if you know what you are doing, but there's no real gain for it.

I know there's a comparison someplace, but I think NV GPUs only went to DP math with the mid-high end 400s and 500s if I remember correctly. ahh here's the list...

2.0

GF100, GF110
GeForce (GF100) GTX 465, GTX 470, GTX 480, Tesla C2050, C2070, S/M2050/70, Quadro Plex 7000, GeForce (GF110) GTX570, GTX580, GTX590

2.1

GF104, GF114, GF116
GeForce GTS 450, GTX 460, GTX 550 Ti, GTX 560, GTX 560 Ti, 500M, Quadro 600, 2000, 4000, 5000, 6000

Fire$torm

10-25-11, 10:57 PM

Same here, almost 1400 Validation Error Wu's since the 16'th, Suspended the Project for now, not sure if I'll ever start it back up again since this seems to be a reoccurring problem ... :mad:

Damn, I thought spingadus's problem was an isolated incident. I see 77 errored wu's on my account. Not good.

I also found this on Dirt's website

status update on server errors
We have essentially the web site, scheduler, and validator running. The assimilator, perfecting, etc. are not running. The error rate has been 0 since we dropped the other daemons, watched the load dropped, and restarted the machine. However, this is not to say that the problem(s) are resolved as we are testing to see if our current theories have correct.

Martin, our host, and I have been exploring particularly thermal and power issues. (Martin found the upload handler was writing 16KiB chunks to disk but optimization to that helped load but didn't change uploaded files getting corrupted.) Also, the upload handler is essentially the least changed, untouched, and until now basically the upstream boinc code. At present the load is staying low and so are the CPU thermals.

Our I/O upgrade only changed the RAID controller and the drives but 3 of the drives were running hotter than they should have been. This extra heat of course impacts the heat in the system.

We have the disk space to keep running in a minimal mode like we are now. Our host already has a heavier power supply and high cfm fans ready. We also are planning some contingencies if we still have problems at that point.

25 Oct 2011 21:10:07 UTC

Edit: Just found the RSS feed for this buried in my list of unread posts. Gotta catch up...

spingadus

10-26-11, 01:09 AM

FWIW, I haven't had any new errors for the last 8 hours. I've split my gpu between distrrtgen and PG now. It irks me to not get credit for the crunching that I've done. The project could just give everyone credit for the invalid wu's if they wanted. I may move away from chasing the high credit gpu projects and spread my cards over the other nvidia projects for my MM's.

STE\/E

10-26-11, 04:01 AM

I've had 3 that I can find since yesterday, looks like quel wrote a script to grant credit to the Validation Error Wu,s, so they could have all along done that but just didn't want to but the problem was so severe yesterday they almost had to ...

spingadus

10-26-11, 06:44 AM

Fire$torm

10-26-11, 10:39 AM

Well what do ya know :) That's pretty cool. I didn't expect them to give credit.

To paraphrase: Happy crunchers, happy project.