BOINC server point of view

[pinhodecarlos]

Is there away to create offline wu's for a certain BOINC project and then add them to the BOINC server? Imagine a BOINC project being hard hit and the server can't keep the pace with the work demanded by the clients connected. There must be away to run in an offline mode the BOINC wu generator.

I can't find an answer to my question. Anyone knows of something related?

Carlos

[yoyo]

It depends what you mean with creating a WU.
A WU is
- a set of files which must be copied into the download directories of the server
- an entry in the workunit table of the server
From my point of view this must be done on the server.
Other things you can make offline, e.g. preparing all files for all wus.

Afterwards the transitioner creates results (as much as your initial replication).

yoyo

[Slicker]

Is there away to create offline wu's for a certain BOINC project and then add them to the BOINC server? Imagine a BOINC project being hard hit and the server can't keep the pace with the work demanded by the clients connected. There must be away to run in an offline mode the BOINC wu generator.

I can't find an answer to my question. Anyone knows of something related?

Carlos

Increasing the shmem on the server will help (shmem_work_items in the config.xml). Also, changing the work generator so it has a much larger number of WUs ready to deploy is also helpful (#define CUSHION 5000 or more).
If the server is IO bound, make sure to have enough RAM so that the ENTIRE mysql database fits into RAM. Adjust the mysql settings accordingly. Also switch to innodb as it is much faster and allows multiple reads at one time.
Put the wu and result files on their own drives, preferrably RAID to improve speed. Put the database and the database log files on separate drives. Set the uldl_dir_fanout in the config.xml to at least 1024. Too many files in a directory slows down access.

If the issue is generating the input files for the WUs, those can be created off-line and placed where the work generator can access them. The upload_url and download_url config.xml settings can be changed so that the files can be located on different servers. Those servers will need to be running apache and have the BOINC cgi etc. on them.

For more info, check the BOINC wiki and/or suscribe and post questions to the boinc_projects email list. I believe there is an entire section on the wiki dedicated to tuning and best practices.

[pinhodecarlos]

yoyo,

I'm talking about the text file that contains the flags that give the orders to run the application.

For example, the NFS@Home project runs the lattice sievers. The usual flags are something like this:
lasieve5f_1.10_x86_64-pc-linux-gnu -r -f 963388000 -c 1000 -R S2m1037.poly -o S2m1037_963388_0_0

.poly file is external and must be downloaded once
-f flag gives the start of the wu
-c flag gives the length of the wu
-o flag writes the output

Carlos

[zombie67]

yoyo,

I'm talking about the text file that contains the flags that give the orders to run the application.

For example, the NFS@Home project runs the lattice sievers. The usual flags are something like this:
lasieve5f_1.10_x86_64-pc-linux-gnu -r -f 963388000 -c 1000 -R S2m1037.poly -o S2m1037_963388_0_0

.poly file is external and must be downloaded once
-f flag gives the start of the wu
-c flag gives the length of the wu
-o flag writes the output

Carlos

Primegrid does this, but not using BOINC.

- User to go a forum
- reserves a range
- downloads the app
- runs it command line with the defined range
- upon completion, uploads the results to the project FTP server.

[yoyo]

I don't know the setup and tuning of your Boinc server.
But I have some experience with my yoyo@home Boinc server, which is a very small one (single core, 1GB RAM, 75GB disk) and which was also already hit by some big team races. But the server was still able to handle it

First you should check where the bottleneck of your server is. (with top and iostat)

In my case it was nearly 100% busy on IO and I found out, that it is the mysql DB which needs to handle all result- and wu state changes. Therefore I did the following to tune the server:

1) Tune mysql settings
The standard mysql installation uses a config file for very small systems (64MB RAM or so). I assume your server has more ram and should use much more for mysql. Mysql is shipped with some example config files for bigger systems. I used than the appropriate one.

2) mysql fine tuning
After running than mysql for at least 24h I used http://www.day32.com/MySQL/tuning-primer.sh, which analysis your mysql settings and suggests changes based on your mysql usage. With this you can adjust mysql caches and so on. The main thing which runs on my server is mysql. So I use more than 50% of the RAM for mysql.

3) long running results
Create workunits which runs longer. Than the clients doesn't connect so often to your server. So e.g. instead running only 2h per result, let them run 6h or 8h.

4) keep the DB small
Ensure, that you have less workunits and less results in your DB. This speeds up the DB and reduces IO. To reach this you can:
- reduce deadline
- don't create so much workunits in advance
- delete workunits and results soon after they are assimilated.

yoyo

[pinhodecarlos]

zombie67, I understood want you meant. Passed a few years sieving for Riesel and PSP before it went to Primegrid so I know how to use all srsieve clients but I was not asking on that perspective.

yoyo, thank you. Where are the wu's stored on your server, on the workunit table? If so how can I add more entries in an offline mode?

[yoyo]

In my workunit table only wus stored which are ready to sent. All other wus are stored just in files or zips in a way that they only have to be copied to the download directory and added to the workunit table. If the amount of workunits which is available to sent out decreases below a threshold, the Boinc server creates from the files new workunits.
You can see here how this goes up and down:

yoyo