Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: BOINC: DA admits CreditNew is really a random number generator...

  1. #1
    Diamond Member
    zombie67's Avatar
    Join Date
    October 24th, 2010
    Location
    Reno, NV
    Posts
    6,850

    BOINC: DA admits CreditNew is really a random number generator...

    ...at least in the sort term...whatever that means.

    Here are some messages from the boinc-dev mail list. Some of the project admins are not happy.

    From: Richard Haselgrove
    Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
    Date: November 7, 2011 10:25:04 AM PST

    Part 2 of this research: Credit

    Because the NumberFields tasks are so variable (from 2 ro 300,000 seconds), I've been looking at the rate that credit has been awarded - Credits per hour (of runtime) gives a nice human-scale value.

    Here are the results for my four hosts at NumberFields. All four were attached at the same time (note the consecutive HostIDs): in particular, the two Q6600 hosts have identical hardware.

    http://img46.imageshack.us/img46/826...reditperho.png

    The problem is that the rate at which credit is granted depends critically on the host APR value. With a non-deterministic project, especially in the early days after attachment, APR is heavily influenced by the random processing time of the early tasks. The credit/hour for all hosts were tightly grouped for the first 10 tasks, when APR is effectively ignored, but thereafter they diverge spectacularly. Hosts 1288 and 1289 happened to get short tasks first, so APR was artificially high when it was first used for credit calculation: hosts 1290 and 1291 happened to draw longer-running tasks.

    It was host 1290 which received the 300,000 second task (http://numberfields.asu.edu/NumberFi...esultid=292439), and was awarded 4,500 credits (in round figures). At very much the same time, the identical host 1291 returned http://numberfields.asu.edu/NumberFi...esultid=317447, getting almost the same credit for just 43,000 seconds of work.

    It's discrepancies like that which lead users, and project administrators, to distrust CreditNew. I think it needs more work, especially if BOINC is going to continue to support non-deterministic projects.
    From: David Anderson
    Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
    Date: November 7, 2011 11:12:12 AM PST

    The goals of CreditNew involve long-term averages.
    It makes no promises about individual jobs or about credit/hour.

    If a project has highly variable jobs,
    this translates into highly variable credit for individual jobs.
    But the long-term average stuff should still hold.

    If anyone has a specific suggestion for how to make credit
    less variable on the short term, while still preserving the
    long-term goals, let me know.

    BTW: the server maintains the variance of elapsed time
    and turnaround on a (host, app version) basis.
    For everything else it maintains only the mean.
    Variance isn't currently used for anything.

    -- David
    From: Travis Desell
    Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
    Date: November 7, 2011 10:53:51 AM PST

    We've had similar issues with the n-body simulations we're doing on milkyway@home. The runtime of the workunits is fairly dependent on the initial random distribution of bodies, and the runtimes can vary from a couple hours to a couple days.

    It would be nice to get away from having to specify the RSC_FPOPS_EST for each workunit, especially as in our case it can cause workunits to be terminated prematurely by the clients. I don't suppose you've run into this problem as well?

    From: Janus Kristensen
    Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
    Date: November 7, 2011 2:36:54 PM PST

    Joining in to say that the BURP based projects sometimes experience this
    exact issue as well. The setup is similar to the other projects
    mentioned in this thread:
    1) fpops estimate cannot be given in advance but has to be fixed to
    something (in our case this is roughly equal to 1 CPU hour since that is
    a reasonable average)
    2) Workunit run time varies randomly in unpredictable ways for the same
    app and even for a single series of workunits (in our case typically
    from 5 secs to 5 days)
    3) Credit is granted in a way that - to the users at least - seems to be
    based on luck. Sometimes it is off by a factor of around 10 or more.

    If a project has highly variable jobs, this translates into highly variable credit
    And this is probably what Richard is pointing out as the core of the
    problem. People see this as a flaw in the credit system - and I have to
    agree to some extent that credit consistency within a single project at
    any given time is a very important treat in a credit system.
    Long term and even cross-project credit stability is desirable, but if
    it comes at the cost of short term credit stability then we have to
    consciously weigh them against each other and figure out what is more
    important to us.
    People perceive instability (be that short or long term) as "unfair".

    If anyone has a specific suggestion for how to make credit less
    variable on the short term, while still preserving the long-term goals,
    let me know.
    What if it turns out that these two things are inherently opposites? Can
    we ignore the problem?

    I'm really feeling lost on this issue. The very old "fpops*cpu_secs +/-
    correction"-scheme had the interesting property that apart from the
    difficulty in measuring the fpops capability of hosts it was at least
    somewhat stable for each project.

    I'm afraid I cannot provide the magical solution, though. Maybe it is
    somewhere in the middle between what we have now and what we had back in
    the very beginning.

    -- Janus
    "Bohnam beats those drums like they owe him money."

    Avatar source
    Twitter: @zombie_67


  2. #2
    Silver Member

    Join Date
    October 9th, 2011
    Location
    Charlotte, NC
    Posts
    323

    Re: BOINC: DA admits CreditNew is really a random number generator...

    the guy should just resign from the project and let someone else run it....

  3. #3
    Past Administrator
    DrPop's Avatar
    Join Date
    October 13th, 2010
    Location
    SoCal, USA
    Posts
    7,482

    Re: BOINC: DA admits CreditNew is really a random number generator...

    It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig.

  4. #4
    Past Admin
    Mike029's Avatar
    Join Date
    October 24th, 2010
    Location
    Bronx, New York
    Posts
    3,377

    Re: BOINC: DA admits CreditNew is really a random number generator...

    Quote Originally Posted by DrPop View Post
    It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig.
    <------- Being a simple guy I actually understand you DrPop.



    The other explanation read like a bunch of bonk.



  5. #5
    Past Administrator
    DrPop's Avatar
    Join Date
    October 13th, 2010
    Location
    SoCal, USA
    Posts
    7,482

    Re: BOINC: DA admits CreditNew is really a random number generator...

    Quote Originally Posted by Mike029 View Post
    <------- Being a simple guy I actually understand you DrPop.

    The other explanation read like a bunch of bonk.
    Confucius say: Truth is simple.

  6. #6
    Silver Member

    Join Date
    October 9th, 2011
    Location
    Charlotte, NC
    Posts
    323

    Re: BOINC: DA admits CreditNew is really a random number generator...

    Quote Originally Posted by DrPop View Post
    It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig.
    Pg cw sieve does this. 384 credits be it CPU, gpu, be it 30s or 3 hrs. 364 (or something lose to that ) per WU

  7. #7
    Friend of SETI.USA
    Join Date
    November 15th, 2010
    Posts
    2,452

    Re: BOINC: DA admits CreditNew is really a random number generator...

    Quote Originally Posted by DrPop View Post
    It's about time they all implemented a fixed credit system like Primaboinca, etc. A WU is worth X credit. If your rig crunches Y WUs per day, then you are duly awarded X*(Y) credits. Simple. Bullet proof. Incentive for purchasing / building / tweaking a faster rig.
    That's fine if all the Wu's take about the same time to run but if like Numberfields where some can take just a few Seconds & others many Hours to run it wouldn't work. It wouldn't take long for some people to figure out which were the short Wu's & they would just abort the longer ones with a Script or Manually ...
    Last edited by STE\/E; 11-08-11 at 04:44 AM.

  8. #8
    Past Administrator
    DrPop's Avatar
    Join Date
    October 13th, 2010
    Location
    SoCal, USA
    Posts
    7,482

    Re: BOINC: DA admits CreditNew is really a random number generator...

    Ah, OK, I see the issue there. So that project would indeed have to be based off some kind of runtime, then. That kind of muddies the waters, when the WUs are different sizes. Could their server just say this WU is .1X in size, or .5X in size, so it gets .1(X) or .5(X) credits respectively?

    I guess what I'm trying to get at, is surely there is a simple way to do this, isn't there?

  9. #9
    Diamond Member
    zombie67's Avatar
    Join Date
    October 24th, 2010
    Location
    Reno, NV
    Posts
    6,850

    Re: BOINC: DA admits CreditNew is really a random number generator...

    Quote Originally Posted by DrPop View Post
    Ah, OK, I see the issue there. So that project would indeed have to be based off some kind of runtime, then. That kind of muddies the waters, when the WUs are different sizes. Could their server just say this WU is .1X in size, or .5X in size, so it gets .1(X) or .5(X) credits respectively?

    I guess what I'm trying to get at, is surely there is a simple way to do this, isn't there?
    The problem is that some projects (including this one) do not know ahead of time, which are the long and which are the short. So if you just go off of claimed time * benchmarks, you open it up to cheating.
    "Bohnam beats those drums like they owe him money."

    Avatar source
    Twitter: @zombie_67


  10. #10
    Diamond Member
    zombie67's Avatar
    Join Date
    October 24th, 2010
    Location
    Reno, NV
    Posts
    6,850

    Re: BOINC: DA admits CreditNew is really a random number generator...

    FWIW, here is my response to the list. Of course, DA will never respond or discuss this kind of stuff in the open. Still, it felt good to write.

    From: zombie67
    Subject: Re: [boinc_dev] APR, DCF and non-deterministic projects
    Date: November 7, 2011 6:19:59 PM PST

    The goals of CreditNew involve long-term averages.
    It makes no promises about individual jobs or about credit/hour.
    I have been following the CreditNew discussion from the beginning. And this is news to me. Seriously. I had no idea that it allows for this kind of behavior, at least not beyond the first several tasks for a particular host. I thought everything would get about the same credits very quickly.

    Some thoughts:

    1) Seemingly randomly awarded credits are going to piss off volunteers. Wait, that sounds hypothetical. Let me rephrase. It has already pissed off volunteers at every project where it has been implemented (check out the message boards). And I am not talking about the "credits are too high/low" old nonsense. I am talking about complaints about how the credits are not consistent. I assume because it grates on the idea of fair play. Maybe explaining (every time) that long-term averages are better than the other guy getting 10x credits right now, will work. I doubt it. But in the end it is up to the poor admins to try to defend the behavior of CreditNew. And it is usually a surprise to the admins as well.

    2) So it is expected that credits for individual jobs appear to be awarded randomly, to the volunteers. Fine. This should be documented somewhere, so that the admins understand that this is *expected* behavior, and so they know what they are getting into *before* implementing. I looked on the trac wiki, but didn't see anything that said so explicitly (maybe I missed it?): <http://boinc.berkeley.edu/trac/wiki/CreditNew>. I think it needs to be added...in BOLD.

    3) There needs to be some kind of boiler plate made available to the admins, to help them explain this behavior to their volunteers. Because they *will* be challenged on this. Repeatedly.

    4) There should be something that shows the volunteers which credit system a project is using, with a link to the wiki. Perhaps on the server status page? It could help to curtail the inevitable "what's going on with the credits?" questions.
    "Bohnam beats those drums like they owe him money."

    Avatar source
    Twitter: @zombie_67


Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •