2,1037- sieve

[pinhodecarlos]

Right now NFS@Home is sieving a big task : 2,1037- (second hardest number).
16e Lattice Sieve is being used to sieve 2,1037- from q=~25M and 16e Lattice Sieve V5 from q=1000M. The former will go until q=1000M and the latter will go until necessary.
Picking up the sieve range from 25M to 1000M, where each workunit manages 1k Q values, it totals 975,000 workunits to be crunched. Each wu on my core i5 750@3.4 GHz takes an average of ~2000 seconds (64-bit linux). By myself it would take me 5642 CPU days to sieve all range. With ~800 equal cores it would take 28.1 days. Nice stats!

If q range goes to q=1200M, 34 days with ~800 cores.
If q range goes to q=1400M, 40 days with ~800 cores.

As I write:

Last 16e Lattice Sieve application wu received is at ~q=110.962M
Last 16e Lattice Sieve V5 application wu received is at ~q=1,042.327M

In the last 24 hours 6,416 16e Lattice Sieve wu's were completed. At this sieve speed the range to sieve with 16e Lattice Sieve application will be done within 139 days.

Lot's of work to be done...

Carlos

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=117.555M
Last 16e Lattice Sieve V5 application wu received is at ~q=1,046.571M

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=186.131M (goal to 1000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,080.533M

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=288.112M (goal to 1000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,111.760M

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=400.000M (goal to 1000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,230.000M (goal to 1400M)

[Duke of Buckingham]

Thanks to keep us update Carlos.

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=470.000M (goal to 1000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,250.000M (goal to 1400M)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=555.000M (goal to 1000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,290.000M (goal to 1400M)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=566.000M (goal to 1000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,295.000M (goal to 1400M)

Would be awesome to see more 16e Lattice Sieve application crunchers. Greg told that after 16e V5 reaches 1400M it will work down from the top of the 16e range until they meet in the middle.

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=583.000M (goal to 1000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,300.000M (goal to 1400M)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=600.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,308.000M (goal to 1,400M)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=609.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,310.000M (goal to 1,400M)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=633.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,326.000M (goal to 1,400M)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=668.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,348.000M (goal to 1,400M, then backwards from 1,000M until meet 16e in the middle)

Received means downloaded but not yet processed. After 16e Lattice Sieve V5 reaches 1,400M it will go down starting at 1,000M until it crosses range from 16e Lattice Sieve, or using other words, 16e V5 will then work down from the top of the 16e range (1,000 M) until they meet in the middle.

1st day of the challenge completed.

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=691.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,361.000M (goal to 1,400M, then backwards from 1,000M until meet 16e in the middle)

[pinhodecarlos]

After third day of the challenge part I, 3 days left:

Last 16e Lattice Sieve application wu received is at ~q=711.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,373.000M (goal to 1,400M, then backwards from 1,000M until meet 16e in the middle)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=737.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,387.000M (goal to 1,400M, then backwards from 1,000M until meet 16e in the middle)

[Mumps]

Last 16e Lattice Sieve application wu received is at ~q=737.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,387.000M (goal to 1,400M, then backwards from 1,000M until meet 16e in the middle)

So, estimating less than 7 days based on the production since your last post?

737-711 = 26M produced in 23 hours
1387-1373 = 14M produced in 23 hours

With only 263+13 (276M) to go, at 40M in 23 hours that's 7*23 hours or about 6 3/4 days left. Presuming production doesn't change much. Did I get that right?

And at that point, the project cluster at the University takes over and does about 2 months of post processing to do the factoring from the sieves we've produced?

[pinhodecarlos]

Mumps, that is the state of the sieve effort as received, not taking into account the aborted wu's and the ones left behind when you detach the project, etc. I think if you add 1-2 days to clean those abort one plus the number of wu's that were not aborted with deadline of 5-7 days, you can say that the sieve of 2,1027- will be done by your estimate time, if the numbers of wu's processed keeps up, plus equivalent days of the deadline, that is 5 or 7 days. I would say in total 15 days to finish everything. Did you understood? I am now quite sleepy...lol
So I ask people that want to leave the challenge to abort the wu's, not to detach the project so that wu's can be quickly spread out again.

The wu's received by NFS@Home server are every hour gathered to last state of the big file called the relations. Then Greg puts it as you said on university cluster first to try to build a matrix. Sometimes the estimates are wrong ( the range 20M-1400 could be needed to go up to 1500M, for example) and more wu's are necessary because the matrix can't be built for the algebra phase (one of the stages of the post-processing), I hope not. I think the post-process will take like 3 weeks if Greg uses the 512 core cluster.

Carlos

[pinhodecarlos]

Because I am sleepy... Just received the 16e V5 wu called S2m1037_1391346_0, this one will be processed by me within 26 hours. My cache sizes 200, I can do ~180 per day.

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=761.000M (goal to 1,000M)
Last 16e Lattice Sieve V5 application wu received is at ~q=1,399.000M (goal to 1,400M, then backwards from 1,000M until meet 16e in the middle)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=782.000M.
Last 16e Lattice Sieve V5 application wu received is at ~q=959.000M (second chunk restarted at 950M going to 1,000M: meaning going backwards from 1,000M until meet 16e in the middle).
Last 16e Lattice Sieve V5 application already sent all wu's from 1,000M to 1,400M (first chunk).

Overall Q range sieve of 2,1027- started at 20M to 1,400M.

Q range situation is this:
20M-782M (sent through 16e application, remaining wu's close to be done)
782M-950M (unsent)
950M-959M (sent, remaining wu's close to be done)
959M-1000M (unsent, will be sent through 16e V5 application)
1000M-1400M (sent through 16e V5 application, remaining wu's close to be done)

In terms of work still to be done we are talking about ~209k wu's left to be crunched, ~113k already created, ~46k in progress.

[pinhodecarlos]

At current pace we would finish 2,1037- sieve within ~7.5 days but we need to count with the aborted wu's and the ones left behind when detaching the project. So I bet 15 days to complete.

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=801M.
Last 16e Lattice Sieve V5 application wu received is at ~q=970M (second chunk restarted at 950M going to 1,000M: meaning going backwards from 1,000M until meet 16e in the middle).
Last 16e Lattice Sieve V5 application already sent all wu's from 1,000M to 1,400M (first chunk).

Overall Q range sieve of 2,1037- started at 20M to 1,400M.

Q range situation is this:
20M-801M (sent through 16e application, remaining wu's close to be done)
801M-950M (unsent)
950M-970M (sent, remaining wu's close to be done)
970M-1000M (unsent, will be sent through 16e V5 application)
1000M-1400M (sent through 16e V5 application, remaining wu's close to be done)

In terms of work still to be done we are talking about ~179k wu's left to be crunched, ~82k already created, ~46k in progress.

Considering my machine (core i5 750 with cache set to 200, 180 wu's daily done) the leading edge of the undone wu's to the finish ones is about 11M without taking into consideration the repetitive wu's that probably are in there mixed.

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=837M.
Last 16e Lattice Sieve V5 application wu received is at ~q=991M (second chunk restarted at 950M going to 1,000M: meaning going backwards from 1,000M until meet 16e in the middle).
Last 16e Lattice Sieve V5 application already sent all wu's from 1,000M to 1,400M (first chunk).

Overall Q range sieve of 2,1037- started at 20M to 1,400M.

Q range situation is this:
20M-837M (sent through 16e application, remaining wu's close to be done)
837M-950M (unsent)
950M-991M (sent, remaining wu's close to be done)
991M-1000M (unsent, will be sent through 16e V5 application)
1000M-1400M (sent through 16e V5 application, remaining wu's close to be done)

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=849M.
Last 16e Lattice Sieve V5 application wu received is at ~q=932M (third chunk restarted at 30M going to 950M: meaning going backwards from 1,000M until meet 16e in the middle).
Last 16e Lattice Sieve V5 application already sent all wu's from 950M to 1,400M.

Overall Q range sieve of 2,1037- started at 20M to 1,400M.

Q range situation is this:
20M-849M (sent through 16e application, remaining wu's close to be done)
849M-930M (unsent)
930M-932M (sent, remaining wu's close to be done)
932M-950M (unsent)
950M-1400M (sent through 16e V5 application, remaining wu's close to be done)

Lot's of aborted wu's being done on the range below 800M.

If you are only running lasieve5f application please consider also checking lasievef application. Linux user will run both, windows users only lasievef.

Carlos Pinho

[Mumps]

Last 16e Lattice Sieve application wu received is at ~q=849M.
Last 16e Lattice Sieve V5 application wu received is at ~q=932M (third chunk restarted at 30M going to 950M: meaning going backwards from 1,000M until meet 16e in the middle).
Last 16e Lattice Sieve V5 application already sent all wu's from 950M to 1,400M.

Overall Q range sieve of 2,1037- started at 20M to 1,400M.

Q range situation is this:
20M-849M (sent through 16e application, remaining wu's close to be done)
849M-930M (unsent)
930M-932M (sent, remaining wu's close to be done)
932M-950M (unsent)
950M-1400M (sent through 16e V5 application, remaining wu's close to be done)

Lot's of aborted wu's being done on the range below 800M.

If you are only running lasieve5f application please consider also checking lasievef application. Linux user will run both, windows users only lasievef.

Carlos Pinho

Just curious, did we actually finish the 11,301- Sieving during these challenges? I see what looks like the number listed as Sieving, but Detailed status indicates it's moved to Post Processing.

Also, 2,1037- doesn't match up to anything I can find on the Detailed status pages for sieve_d or sieve_e. Do they not show all the numbers in the Detailed status pages?

EDIT: Yeah, there it is. 5 days ago in the Project News. Which I don't remember seeing in our RSS feed here... :)

[pinhodecarlos]

Just curious, did we actually finish the 11,301- Sieving during these challenges? I see what looks like the number listed as Sieving, but Detailed status indicates it's moved to Post Processing.

Also, 2,1037- doesn't match up to anything I can find on the Detailed status pages for sieve_d or sieve_e. Do they not show all the numbers in the Detailed status pages?

EDIT: Yeah, there it is. 5 days ago in the Project News. Which I don't remember seeing in our RSS feed here... :)

Post-processing of 11,301+ was started before the challenges 1st of December, check attach with log file. Application used was lasievee (44 points per wu, less than 1GB necessary).
11,301- sieve was ended during the challenges. Currently being post-processed by M. Vang on a core i7 with 16 GB of memory. Relation file zipped is 19 GB. Last wu's gathered entered it at 22nd of December.

For integers being sieved by applications lasievef and lasieve5f it is very difficult to estimate how much wu's are necessary so there's no progress or detailed page as are for applications lasieved and lasievee. Here we are working on advancing the state of the art of factoring. As I said before, more wu's for 2,1037- can be necessary to generate if a matrix, for the linear algebra phase, is not built.

Carlos Pinho

1222

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=910M.
Last 16e Lattice Sieve V5 application already sent all wu's from 930M to 1,400M.

Overall Q range sieve of 2,1037- started at 20M to 1,400M.

Q range situation is this:
20M-910M (sent through 16e application, remaining wu's close to be done)
910M-930M (unsent throgh 16e application)
930M-1400M (sent through 16e V5 application, remaining wu's close to be done)

16e V5 application started another number (2,1049+) from q=1000M because all wu's for 2,1037- have been distributed. Only missing the left overs wu's. Also I only gave order to my CPU to do the 16e Lattice Sieve wu's instead of both 16 and 16e V5.

Carlos Pinho

[pinhodecarlos]

Last 16e Lattice Sieve application wu received is at ~q=921M.
16e Lattice Sieve V5 application already sent all wu's from 930M to 1,400M.

Overall Q range sieve of 2,1037- started at 20M to 1,400M.

Q range situation is this:
20M-921M (sent through 16e application, remaining wu's close to be done)
921M-930M (unsent through 16e application)
930M-1400M (sent through 16e V5 application, remaining wu's close to be done)

16e V5 application started another number (2,1049+) from q=1000M because all wu's for 2,1037- have been distributed. Only missing the left overs wu's. Also I only gave order to my CPU to do the 16e Lattice Sieve wu's instead of both 16 and 16e V5 applications respectively.

In conclusion, 2,1037- will be completely sieved by the end of the year.
After next phase (post-processing) finishes by using the 512 core cluster, Greg usually writes a simple 2-3 pages article about the factorization with final figures. Hope I can give you guys a little surprise. Thank you all for this month of full crunching.

Carlos Pinho

[pinhodecarlos]

This is the number, 31376, the number of wu's still left to be done on 2,1037- sieving. This number represents the aborted ones, the left ones, etc.
Let's point to 7-8 days to fully complete the 2,1037- sieve due to the increase of the wu's deadline from 4 days to 7 days.

Carlos

[pinhodecarlos]

It's down to 20948 wu's left (16e+16e V5).

[Fire$torm]

Just curious, did we actually finish the 11,301- Sieving during these challenges? I see what looks like the number listed as Sieving, but Detailed status indicates it's moved to Post Processing.

Also, 2,1037- doesn't match up to anything I can find on the Detailed status pages for sieve_d or sieve_e. Do they not show all the numbers in the Detailed status pages?

EDIT: Yeah, there it is. 5 days ago in the Project News. Which I don't remember seeing in our RSS feed here... :)

hmmm..... How did that not happen........?

OK, all better now.

[pinhodecarlos]

Greg said:


Down to 12724 remaining. There are only a couple hundred lasieve5f wu's left, and I'll finish those off today.

Carlos: If you like, I can direct some remaining lasievef wu's to your computers. Let me know how many...

I'm in, who can help me (STE\/E)? I think I need by PM some info I think, your nickname on NFS@Home or ID.

Carlos

[pinhodecarlos]

I aborted all my 2,1047+ wu's so I can start cleaning the left overs of 2,1037- Greg is sending me to my ID. He can only send me 250/day because I can only crunch ~180 wu's per day. Anyway, some have already been automatically cancelled by the server due to the fact that someone else already did them. That's good that BOINC and the server can handle and detect and not duplicate work.

I think that more help from a few SETI.USA fellows would be appreciated so we can easily crunch those 12k wu's needed to finally finish 2,1037- sieve and to Greg start the post-processing phase.

From my side I can guarantee to Greg we can always send me the left overs of 2,1037-.

Carlos Pinho

[STE\/E]

I'll help too at least until I get to 10 Million maybe more ...

[pinhodecarlos]

I'll help too at least until I get to 10 Million maybe more ...

If you can manually abort in a few machines 20-30 2,1047+ wu's (those S2p1049_xxxxx_0 wu's) that would be great. Greg sent to you ~450 2,1037- wu's. With your current CPU power you can easily clean 6-7k.

What do you think?

Carlos

[STE\/E]

the S2m1037 are the ones Greg wants run ???

[pinhodecarlos]

the S2m1037 are the ones Greg wants run ???

Yes. You might already have a few ones in your queue. I also had a few, didn't abort them, only the S2p1047 ones.

[STE\/E]

Yes. You might already have a few ones in your queue. I also had a few, didn't abort them, only the S2p1047 ones.

Yes I see some on at least 1 Box, will check them all ...

[pinhodecarlos]

The left overs can be called something like this :S2m1037_767510_23816_1356609309_0. It has more numbers but I putted in bold the main characteristic of the wu. I don't know what the 23816_1356609309 means.

[pinhodecarlos]

If you click on the left over wu you will then see that it was distributed to two clients. The first one crunching it earns the points, the other the server cancels it. Both of us will always get the old wu's but for example Greg already sent me 350 wu's, and by my stats, at least 9 were already done by others. No duplicate work is done. I already did 14, 9 were cancelled by the server for my wingman pair who received it a few days ago. So, once again no duplicate work is done.

[pinhodecarlos]

The left overs can be called something like this :S2m1037_767510_23816_1356609309_0. It has more numbers but I putted in bold the main characteristic of the wu. I don't know what the 23816_1356609309 means.

The wu detail is S2m1037_767510_2 (last number differ, it means second time sent to clients)
Wu task detail is that enormous S2m1037_767510_23816_1356609309_0.

Here's a good example of a wu sent to me and 20 mins later cancelled by the server. My ID computer is 28521.

http://escatter11.fullerton.edu/nfs/workunit.php?wuid=20678799

I think more 2-3 days and we clean all old S2m1037 wu's because with dedicated machines to clean the left overs it is more quickly.
Steve, what you can do is put 1-3 machines only doing the old S2m1037 wu's. I added mine, Greg is also doing the same.

[STE\/E]

I cleaned out a lot of the 1049 Wu's but that's all I get back ... If Greg can He can go into my Computers and clean them out himself & send me the 1037's, if that's possible ...

[pinhodecarlos]

I cleaned out a lot of the 1049 Wu's but that's all I get back ... If Greg can He can go into my Computers and clean them out himself & send me the 1037's, if that's possible ...

I already asked him about it a few weeks ago but that's not possible. The server cannot abort wu's for a dedicated client, it can only abort wu's when someone else did it as we are doing now with the old wu's of S2m1037. He cannot enter our boinc manager to abort wu's. I understand it's very pain in an ass manually abort a bunch of wu's in a bunch of machines. Greg can only specify for a certain client, not computer, a certain type of work.

Those 1049 aborted wu's are helpful, now it's from Greg side to do the work. Thank you Steve, you can now go to sleep.

Carlos

[STE\/E]

More like get dressed & go get the Snow Blower out, we had a Winter Storm blow thru yesterday afternoon & last night, still snowing but have to try & get rid of some of it before it gets to deep. Not to bad right now, maybe 4" or so ... http://www.darkgrid.com/wafen/images2/snowblower.gif

[pinhodecarlos]

More like get dressed & go get the Snow Blower out, we had a Winter Storm blow thru yesterday afternoon & last night, still snowing but have to try & get rid of some of it before it gets to deep. Not to bad right now, maybe 4" or so ... http://www.darkgrid.com/wafen/images2/snowblower.gif

I don't know where you have your machines but I was thinking that you could use that snow to cool the inside air where you have your machines...I have to think more. Just post here the details of the space you have the computers.
Anyway, I'm moving my things over to my parents home because in January I will be moving to another country so I am also busy, so don't worry if you don't get a quick reply from my side.

Carlos Pinho

[STE\/E]

13 are in my Garage which I can keep as cool as I want in the winter, 2 are in an upstairs Bedroom where I keep a window open & my only Laptop in in the living room where I can monitor the activity of all the Box's from a single Box ...

[pinhodecarlos]

13 are in my Garage which I can keep as cool as I want in the winter, 2 are in an upstairs Bedroom where I keep a window open & my only Laptop in in the living room where I can monitor the activity of all the Box's from a single Box ...

What do you use, an axial vent?

Back to the topic, 2,1037- is almost done. We (Greg, Steve and Carlos) are cleaning the last wu's.

[Mike029]

13 are in my Garage which I can keep as cool as I want in the winter, 2 are in an upstairs Bedroom where I keep a window open & my only Laptop in in the living room where I can monitor the activity of all the Box's from a single Box ...

Brilliant!!!

[pinhodecarlos]

Here's the situation:

As of 22:55 GMT there are 381 wu's left. I have 260 and Steve has 121. Cancelled rate of 20 %. Less than 12 hours to completely finish 2,1037- sieve.

Good work Steve!

Carlos

[Duke of Buckingham]

Nice that you achieved your main goal Carlos and had a HAPPY CHRISTMAS. Good that STE\/E and the team could help you for the best and STE\/E is still helping.

Nice to see a part of the work done. That is a battle we are all about to won. Congratulations to all that helped.

[pinhodecarlos]

Next goal while Greg runs post-processing of 2,1037- on the cluster is to sieve 2,1049+. This latter number is harder than the former. More 6 months sieving it....lol

Carlos

[pinhodecarlos]

Here's a reply from Greg:


There are a handful of somewhat problematic WU's still out. I'll give those a chance to finish while I collect the other relations. But yes, we are basically done.

For a number this large I do the collection and duplication removal in stages, so I don't have a total raw relation count at hand. I will report the size of the duplicate-free file and the number of unique relations when I have them. I'm processing the last half-million result files now.

[Duke of Buckingham]

Damn memory, I don't remember where is the post that I promised a video about Rio de Janeiro. Well the video, is very simple but it is here Carlos.

And Carlos the light of the city is amazing, no pictures make justice to beauty of Rio. Now I want to go there also...


http://www.youtube.com/watch?v=UFGzpZAZx7o

I will make a better video on a near future.

[pinhodecarlos]

Thank you Ricardo.

Greg started the filtering on January the 4th. After dup removal, the file is

-rw-rw-r-- 1 childers childers 90724638183 Jan 3 15:11 msieve.dat

90.7 billion bytes is about 85 GB. Dup removal found

Found 680357182 unique, 414377950 duplicate (37.9% of total), and 3722 bad relations.

So the total file would have been about 135GB in size.

I already asked Greg if LA is underway.

Carlos

[pinhodecarlos]

The LA has started.

matrix is 67008555 x 67008732 (29495.3 MB) with weight 8405533452 (125.44/col)

linear algebra at 0.1%, ETA 447h56m

[pinhodecarlos]

The LA has started.

matrix is 67008555 x 67008732 (29495.3 MB) with weight 8405533452 (125.44/col)

linear algebra at 0.1%, ETA 447h56m

LA is using 576 cores on Trestles at the San Diego Supercomputing Center.