Yafu: new MT app available, and help needed

[zombie67]

The good news: They implemented a multi-threaded app! Run times are much smaller now, with lots of cores. Love it!

The bad news: The credits have gone from bad to horrible. It uses CreditNew still, so it is impossible to tell what is going on. My theory: The wrapper system it uses is somehow messing with the CPU time value, and under-reporting CPU time as if it was a single CPU app. So you are using all (say) 8 cores, for (say) an hour, and it is awarding credits as if it was a single CPU being used for an hour. Read more about it here (http://yafu.dyndns.org/yafu/forum_thread.php?id=79&nowrap=true#307), if you want.

Both apps are still available (http://yafu.dyndns.org/yafu/apps.php) (MT and non-MT), but there is no way to manually chose one or the other via project preferences. Yoyo asked DA how to do this, and DA said there is no way. The BOINC server is sending everyone the MT app. The only way around this is via anonymous platform.

So I need some help, please. Can someone please write an app_info.xml file? It is beyond me. The application itself is the same for both MT and non-MT, actually. (http://yafu.dyndns.org/yafu/forum_thread.php?id=81) But it uses a wrapper, and I just don't know how to do all that in an app_info.xml.

FYI, we are still in the #1 spot. And now that everyone's production has been cut WAY down due to the MT app, us (just me?) getting back to the non-MT app would be an easy opportunity to increase the lead.

[Fire$torm]

I doubt that I can help but this might point someone in the right direction. Can you please post the contents of that job.xml file.

[zombie67]

There are several files:

yafu_w64_02.ini

ggnfs_dir=.\
ecm_path=ecm.exe
R=1


yafu_w64mt_02.ini

ggnfs_dir=.\
ecm_path=ecm.exe
R=1

yafujob_0.02.xml

<job_desc>
<task>
<application>yafu</application>
<stdin_filename>in</stdin_filename>
<stdout_filename>out</stdout_filename>
<checkpoint_filename>nfs.dat</checkpoint_filename>
</task>
</job_desc>

yafujobmt_0.04.xml

<job_desc>
<task>
<application>yafu</application>
<stdout_filename>out</stdout_filename>
<command_line>-threads $NTHREADS -batchfile in</command_line>
<multi_process/>
<checkpoint_filename>nfs.dat</checkpoint_filename>
</task>
</job_desc>

[Fire$torm]

I take it no one has answered your request on the Yafu forum?

Well it looks like yafujobmt_0.04.xml is the file that initiates multi-threading with this line <command_line>-threads $NTHREADS -batchfile in</command_line>.

I wonder if you tell BOINC not to check file sizes in the cc_config.xml if I could just copy the contents of the yafujob_0.02.xml file into yafujobmt_0.04.xml..... I can't try it now but maybe over the weekend just to see.

[spingadus]

So, did anyone figure out how to get the non threads tasks to run? I was thinking of finishing up my 250k MM for this project before it ends, but I don't want to waste all my cores with a mt task that doesn't grant any more credit than a single core task.

[trigggl]

Set your computer to 1 processor only then set your cache to 10 days. Set NNW and open back up all cores.

When I limit one of my dual cores to one CPU sim1 sends me non-mt tasks.

It's worth a shot.

[spingadus]

Cool I'll try it when I get back. Its cheap pizza Tuesday at mountain Mike's

Sent from my Galaxy Nexus using Tapatalk

[spingadus]

Trigggl's method worked! Only issue so far is that it only downloads 4 tasks and then says I've reached a limit. Maybe I need to complete a few tasks before I can download more? Either way, you da man Trigggl!

On a side note, prior to this, I tried to edit the <commandline...-THREAD... part out of the yafujobmt_0.04.xml file. It does indeed check the file size and found that it was wrong. So, I padded the file with white space until it reached the correct file size. Unfortunately, it still downloaded the mt tasks, so I gave up. Don't have time to fiddle as I have a mid term tomorrow.

As soon as my current tasks complete, I'll try Trigggl's method again.

[Fire$torm]

Trigggl's method worked! Only issue so far is that it only downloads 4 tasks and then says I've reached a limit. Maybe I need to complete a few tasks before I can download more? Either way, you da man Trigggl!

On a side note, prior to this, I tried to edit the <commandline...-THREAD... part out of the yafujobmt_0.04.xml file. It does indeed check the file size and found that it was wrong. So, I padded the file with white space until it reached the correct file size. Unfortunately, it still downloaded the mt tasks, so I gave up. Don't have time to fiddle as I have a mid term tomorrow.

As soon as my current tasks complete, I'll try Trigggl's method again.

If they keep limiting the number of tasks you could try using multiple VMs. Initially set each VM CPU to one core, DL the tasks, shut down the VMs, up the cores to 4~16 and then restart the VMs.

[spingadus]

The vm approach would require a lot of micro managing. It appears the Yafu will only let me download 4 at a time as I tried it again. So, I may have to just do the mt tasks. It may not be too bad. I just looked at some other crunchers completed tasks and it looks like they do count the total run time now for all cores.

[zombie67]

The vm approach would require a lot of micro managing. It appears the Yafu will only let me download 4 at a time as I tried it again. So, I may have to just do the mt tasks. It may not be too bad. I just looked at some other crunchers completed tasks and it looks like they do count the total run time now for all cores.

Interesting. I took a look at some tasks, and I am seeing mostly the opposite. For example (http://yafu.dyndns.org/yafu/result.php?resultid=648937).

Run time 25,276.24
CPU time 7,437.88

Those numbers should be reversed.

[trigggl]

Interesting. I took a look at some tasks, and I am seeing mostly the opposite. For example (http://yafu.dyndns.org/yafu/result.php?resultid=648937).

Run time 25,276.24
CPU time 7,437.88

Those numbers should be reversed.

Must be a Mac thing.

[Slicker]

I would think an app_info.xml would solve the problem. When my 24 core web server was crunching AQUA, running more than 8 cores on a task was a waste as the cores wouldn't be fully utilized. I let it download some tasks and then used the sched_reply file as a template to create an app_info file. I changed the max and average cpus to be 8 rather than 64 so it would run one WU for every 8 cores. The same thing should work here by using just 1 cpu.

[zombie67]

Must be a Mac thing.

No, that was a random windows machine from the top machines page. Most (but not all) machines on that list have tasks that are backwards like that.

FWIW, there is no mac app, so it definitely not a mac thing.

[trigggl]

No, that was a random windows machine from the top machines page. Most (but not all) machines on that list have tasks that are backwards like that.

FWIW, there is no mac app, so it definitely not a mac thing.

I figured it probably wasn't, but seemed like the thing to say at the time.

[spingadus]

Ok, so is the credit at this project based on run time or cpu time? It's hard to tell with the numbers i'm getting.

[zombie67]

Ok, so is the credit at this project based on run time or cpu time? It's hard to tell with the numbers i'm getting.

1st: It is CreditNew, so regardless, credits will appear mostly random.

2nd: I believe CPU time is what is used for the calculation.

[spingadus]

1st: It is CreditNew, so regardless, credits will appear mostly random.

2nd: I believe CPU time is what is used for the calculation.

k thanks. My cpu vs run times are all over for the few mt tasks i've run. The last task I did had a larger cpu time than the run time, but the others did not. So, I have no idea how the tasks times are being recorded.

http://yafu.dyndns.org/yafu/results.php?hostid=43&offset=0&show_names=0&state=3&appid=

I should have stopped at 100k instead of 150k and then I wouldn't feel the need to get my 250k MM. :(

[zombie67]

Right. This is why I started this thread. The problem goes away with the non-MT app.

[spingadus]

Well, if anyone has the time to create a working app_info file please post it here :)

[zombie67]

Well, if anyone has the time to create a working app_info file please post it here :)

+1

In the mean time, once I get my 100k @ Sim1, I will re-task my VMs, changed to 1 thread, to yafu.

[spingadus]

Isn't that a lot of overhead? Do you run your vms concurrently with boinc on the main OS?

[zombie67]

Isn't that a lot of overhead? Do you run your vms concurrently with boinc on the main OS?

No, no additional overhead. For example, on an 8 thread CPU, I would set BOINC to use 7 threads, and the VM to use 1 thread.

[spingadus]

No, no additional overhead. For example, on an 8 thread CPU, I would set BOINC to use 7 threads, and the VM to use 1 thread.

I see. I thought you were going to run more vm's than that. Yafu credit is low, so 1 vm per pc would be slow going, unless you have a lot of pcs.

[zombie67]

I see. I thought you were going to run more vm's than that. Yafu credit is low, so 1 vm per pc would be slow going, unless you have a lot of pcs.

Only about 4-5 running VMs. But with fast SB threads, it will go fast enough.

[zombie67]

Update:

The fact that CPU time reporting is messed up, has nothing to do with the credits awarded. Credits are based on run time and the number of threads (MT app). You can read more about it here (http://yafu.dyndns.org/yafu/forum_thread.php?id=87).

Anyway, after doing some more data analysis, there is an upside. Credits awarded are not so horrible after all. On a Core2 CPU, credits range from 20-40 credits/hour/thread. Considering that the old "normal" method of awarding credits awarded ~20 c/h/t*, it is not worse than it used to be, once the machine settles down under CreditNew....at least for this project. I validated this across three different machines.

So I guess this is a long way to say it is safe to come back to yafu, if you want. It would be nice to keep our #1 spot there (http://boincstats.com/stats/team_stats.php?pr=yafu&st=0). Remember, it is a finite project. So we can keep that #1 forever.

*Seriously. Compare to some of the older projects like amelegrid, sztaki, etc