PDA

View Full Version : BOINC: DA admits CreditNew is really a random number generator...



zombie67
11-07-11, 07:25 PM
...at least in the sort term...whatever that means.

Here are some messages from the boinc-dev mail list. Some of the project admins are not happy.


From: Richard Haselgrove
Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
Date: November 7, 2011 10:25:04 AM PST

Part 2 of this research: Credit

Because the NumberFields tasks are so variable (from 2 ro 300,000 seconds), I've been looking at the rate that credit has been awarded - Credits per hour (of runtime) gives a nice human-scale value.

Here are the results for my four hosts at NumberFields. All four were attached at the same time (note the consecutive HostIDs): in particular, the two Q6600 hosts have identical hardware.

http://img46.imageshack.us/img46/826/numberfieldscreditperho.png

The problem is that the rate at which credit is granted depends critically on the host APR value. With a non-deterministic project, especially in the early days after attachment, APR is heavily influenced by the random processing time of the early tasks. The credit/hour for all hosts were tightly grouped for the first 10 tasks, when APR is effectively ignored, but thereafter they diverge spectacularly. Hosts 1288 and 1289 happened to get short tasks first, so APR was artificially high when it was first used for credit calculation: hosts 1290 and 1291 happened to draw longer-running tasks.

It was host 1290 which received the 300,000 second task (http://numberfields.asu.edu/NumberFields/result.php?resultid=292439), and was awarded 4,500 credits (in round figures). At very much the same time, the identical host 1291 returned http://numberfields.asu.edu/NumberFields/result.php?resultid=317447, getting almost the same credit for just 43,000 seconds of work.

It's discrepancies like that which lead users, and project administrators, to distrust CreditNew. I think it needs more work, especially if BOINC is going to continue to support non-deterministic projects.


From: David Anderson
Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
Date: November 7, 2011 11:12:12 AM PST

The goals of CreditNew involve long-term averages.
It makes no promises about individual jobs or about credit/hour.

If a project has highly variable jobs,
this translates into highly variable credit for individual jobs.
But the long-term average stuff should still hold.

If anyone has a specific suggestion for how to make credit
less variable on the short term, while still preserving the
long-term goals, let me know.

BTW: the server maintains the variance of elapsed time
and turnaround on a (host, app version) basis.
For everything else it maintains only the mean.
Variance isn't currently used for anything.

-- David


From: Travis Desell
Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
Date: November 7, 2011 10:53:51 AM PST

We've had similar issues with the n-body simulations we're doing on milkyway@home. The runtime of the workunits is fairly dependent on the initial random distribution of bodies, and the runtimes can vary from a couple hours to a couple days.

It would be nice to get away from having to specify the RSC_FPOPS_EST for each workunit, especially as in our case it can cause workunits to be terminated prematurely by the clients. I don't suppose you've run into this problem as well?



From: Janus Kristensen
Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
Date: November 7, 2011 2:36:54 PM PST

Joining in to say that the BURP based projects sometimes experience this
exact issue as well. The setup is similar to the other projects
mentioned in this thread:
1) fpops estimate cannot be given in advance but has to be fixed to
something (in our case this is roughly equal to 1 CPU hour since that is
a reasonable average)
2) Workunit run time varies randomly in unpredictable ways for the same
app and even for a single series of workunits (in our case typically
from 5 secs to 5 days)
3) Credit is granted in a way that - to the users at least - seems to be
based on luck. Sometimes it is off by a factor of around 10 or more.


If a project has highly variable jobs, this translates into highly variable credit

And this is probably what Richard is pointing out as the core of the
problem. People see this as a flaw in the credit system - and I have to
agree to some extent that credit consistency within a single project at
any given time is a very important treat in a credit system.
Long term and even cross-project credit stability is desirable, but if
it comes at the cost of short term credit stability then we have to
consciously weigh them against each other and figure out what is more
important to us.
People perceive instability (be that short or long term) as "unfair".


If anyone has a specific suggestion for how to make credit less
variable on the short term, while still preserving the long-term goals,
let me know.

What if it turns out that these two things are inherently opposites? Can
we ignore the problem?

I'm really feeling lost on this issue. The very old "fpops*cpu_secs +/-
correction"-scheme had the interesting property that apart from the
difficulty in measuring the fpops capability of hosts it was at least
somewhat stable for each project.

I'm afraid I cannot provide the magical solution, though. Maybe it is
somewhere in the middle between what we have now and what we had back in
the very beginning.

-- Janus

DAD
11-07-11, 07:30 PM
the guy should just resign from the project and let someone else run it....

DrPop
11-07-11, 09:46 PM
It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig. :cool:

Mike029
11-07-11, 09:50 PM
It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig. :cool: <------- Being a simple guy I actually understand you DrPop.



The other explanation read like a bunch of bonk.

DrPop
11-07-11, 10:42 PM
<------- Being a simple guy I actually understand you DrPop.

The other explanation read like a bunch of bonk.

Confucius say: Truth is simple.

DAD
11-08-11, 12:37 AM
It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig. :cool:

Pg cw sieve does this. 384 credits be it CPU, gpu, be it 30s or 3 hrs. 364 (or something lose to that ) per WU

STE\/E
11-08-11, 03:42 AM
It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig. :cool:

That's fine if all the Wu's take about the same time to run but if like Numberfields where some can take just a few Seconds & others many Hours to run it wouldn't work. It wouldn't take long for some people to figure out which were the short Wu's & they would just abort the longer ones with a Script or Manually ...

DrPop
11-09-11, 01:32 PM
Ah, OK, I see the issue there. So that project would indeed have to be based off some kind of runtime, then. That kind of muddies the waters, when the WUs are different sizes. Could their server just say this WU is .1X in size, or .5X in size, so it gets .1(X) or .5(X) credits respectively?:confused:

I guess what I'm trying to get at, is surely there is a simple way to do this, isn't there?

zombie67
11-09-11, 07:40 PM
Ah, OK, I see the issue there. So that project would indeed have to be based off some kind of runtime, then. That kind of muddies the waters, when the WUs are different sizes. Could their server just say this WU is .1X in size, or .5X in size, so it gets .1(X) or .5(X) credits respectively?:confused:

I guess what I'm trying to get at, is surely there is a simple way to do this, isn't there?

The problem is that some projects (including this one) do not know ahead of time, which are the long and which are the short. So if you just go off of claimed time * benchmarks, you open it up to cheating.

zombie67
11-09-11, 07:44 PM
FWIW, here is my response to the list. Of course, DA will never respond or discuss this kind of stuff in the open. Still, it felt good to write.


From: zombie67
Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
Date: November 7, 2011 6:19:59 PM PST


The goals of CreditNew involve long-term averages.
It makes no promises about individual jobs or about credit/hour.

I have been following the CreditNew discussion from the beginning. And this is news to me. Seriously. I had no idea that it allows for this kind of behavior, at least not beyond the first several tasks for a particular host. I thought everything would get about the same credits very quickly.

Some thoughts:

1) Seemingly randomly awarded credits are going to piss off volunteers. Wait, that sounds hypothetical. Let me rephrase. It has already pissed off volunteers at every project where it has been implemented (check out the message boards). And I am not talking about the "credits are too high/low" old nonsense. I am talking about complaints about how the credits are not consistent. I assume because it grates on the idea of fair play. Maybe explaining (every time) that long-term averages are better than the other guy getting 10x credits right now, will work. I doubt it. But in the end it is up to the poor admins to try to defend the behavior of CreditNew. And it is usually a surprise to the admins as well.

2) So it is expected that credits for individual jobs appear to be awarded randomly, to the volunteers. Fine. This should be documented somewhere, so that the admins understand that this is *expected* behavior, and so they know what they are getting into *before* implementing. I looked on the trac wiki, but didn't see anything that said so explicitly (maybe I missed it?): <http://boinc.berkeley.edu/trac/wiki/CreditNew>. I think it needs to be added...in BOLD.

3) There needs to be some kind of boiler plate made available to the admins, to help them explain this behavior to their volunteers. Because they *will* be challenged on this. Repeatedly.

4) There should be something that shows the volunteers which credit system a project is using, with a link to the wiki. Perhaps on the server status page? It could help to curtail the inevitable "what's going on with the credits?" questions.

Bryan
11-09-11, 08:55 PM
You are right on about the admins being inundated with questions and complaints. POEM just now implemented credit new and the fun has already begun on their forums.

DrPop
11-09-11, 10:33 PM
4) There should be something that shows the volunteers which credit system a project is using, with a link to the wiki. Perhaps on the server status page? It could help to curtail the inevitable "what's going on with the credits?" questions.

Thank you for posting that in public wherever you did, and I would like to add for the record, I am all for boycotting projects that utilize CreditNew until it is fixed, (if that is even possible), or it is abolished (would be my preferred way of handling it).

rgathright
11-11-11, 02:18 PM
Thank you for posting that in public wherever you did, and I would like to add for the record, I am all for boycotting projects that utilize CreditNew until it is fixed, (if that is even possible), or it is abolished (would be my preferred way of handling it).

OCCUPY CREDITNEW!
http://www.minnpost.com/_asset/7nk4hd/mp_main_wide/OccupyWallStreet452.jpg

DrPop
11-11-11, 03:40 PM
Hahahahahah!!! ROFL! =))........:)).....Oh man! OCCUPY CREDITNEW! Why didn't we all think of that? :P Wonder if we'd get any donations or press coverage?:P Oh...wait, that's only for the liberal media's pet projects.:rolleyes: Oh well.