In love with gpu crunching

[DAD]

I hope more projects make cuda WUs (I'll take open cl, but it sort of sucks)

A good nv gpu, can outwork even an OCed i7 with all it's cores

Example a pg cw sieve... I can do 2 every 1.75-2.25mins on 2 OCed 560TIs. My i7 CPU can do 8 in... Gah id have to go look... I think it's like 1.5-2hrs

Let's just say 1.5 for 8.... My gpus can do 8 far less time

As long a projects continue support cuda, I think future rigs will have as many pcie slots as possible with a cheap CPU. I'd REALLY like to build a 6-8 590 gpu rig. Mmmm 12-16 gpus going all at once... Powergasm :)

Although I'm not sure if the nv drivers can deal with more than 4 gpus. I know 4 is the sli limit, but if they aren't in sli, then I would think no limit would apply. If anyone knows for SURE, let me know

Once I get back fom my cruise, I'll seek out a mobo that can handle 6-8 590s with an intel CPU... Ya I'm biased. I don't do amd or ati. Then slowly populate it over time with only dual core gpus

[Fire$torm]

Sounds like a good plan. Ya gotta love disposable income... :D

[DAD]

Lol ya, but it's being disposed on the cruise, wedding, and house repairs first

But I can still build the core rig pretty cheap and start it off with one 590. Then add more over time. I just have to find out if nv drivers can deal with 12-16 gpus

[Fire$torm]

JPM would know for sure as well a other members but if IIRC nVidia limits the number of GPUs in a system to 8. So you can run four 590s. And personally that about all you should run unless you use a customer liquid cooling solution although GPU cooling is much more expensive then CPU solutions.

[DAD]

Did some digging on nvidia forums. As long as its newer drivers, a newer bios with a 64bit CPU, 8 pcie slots capable of 8x8 speed then up to 16 gpus are supported in 8x2 card configurations. Win 7 x64 also a requirement. Dunno about vista x64

The old limit was 8 back in 2009 with nvida saying they got 12 working but official driver support was 8. The new drivers can do 16 provided the above conditions are met

But yeah air cooling won't work for 8 590s. Air would work for 4 590s in a 8 pcie system spaced apart probably. For 8, custom liquid cooling needed with multiple pumps and water tanks

I just want to build the ultimate gpu rig and see it blow the num 1 computer off the charts in pg etc (based on RAC)... Those num 1 systems have 3-4 gpus at most. Probably 4 590s would be enough to do it ;)

[DrPop]

Dude! :-o:p That would be insane. ;) But, very, very cool crunching under the SETI.USA flag. :D **==

I know you're really knowledgable and into computers, but I would definitely have some PMs with JPM before you buy anything just in case. Simply because he built a hog with 2 GTX 590s in it, water cooling, etc - and he might have found out some pointers the hard way over time. ;)
Plus - he LOVES talking about hardware. :D

[DAD]

I've seen options that put a card in your computer that goes to box where you can put more gpus so another rig isn't needed but those cost like 2-3k! Lame


I think I'd be happy with a 4 590 system. Just have to find the right case and mobo to space them properly

[Fire$torm]

I've seen options that put a card in your computer that goes to box where you can put more gpus so another rig isn't needed but those cost like 2-3k! Lame


I think I'd be happy with a 4 590 system. Just have to find the right case and mobo to space them properly

How about a Mountain Mods case???? (Page Link (http://www.mountainmods.com/computer-cases-ascension-c-21_85.html))

http://www.jpcomputersolutions.com.au/media/catalog/product/cache/1/image/aa8fb1bcb5e9e39f5de349b5d3896bf6/a/s/ascensionb3.jpg

A former team member named JerWA turned me on the MM cases. Just can't afford one.......

[John P. Myers]

FastraII (http://fastra2.ua.ac.be/) 12 TFLOPS of power using 13 GPUs in a single PC. It's not bad at all. However, if you notice the GPUs they used, you can immediately see their design can be improved upon, just cost a bit more money. Doing some calculations, if they had used the equivalent 500 series GPUs, they could've gotten 18.8235 TFLOPS of computing power, at stock speeds. And if they had used my motherboard which has 8 PCIe slots instead of 7, they could've gotten 21.3118 TFLOPS. :p

[Mike029]

...And if they had used my motherboard which has 8 PCIe slots instead of 7, they could've gotten 21.3118 TFLOPS. :p

What motherboard was that again? :x

[Fire$torm]

What motherboard was that again? :x

For Sandy Bridge ---> http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=100007627+600142440&QksAutoSuggestion=&ShowDeactivatedMark=False&Configurator=&IsNodeId=1&Subcategory=280&description=&hisInDesc=&Ntk=&CFG=&SpeTabStoreType=&AdvancedSearch=1&srchInDesc=

For AMD ---> None (Ooops)

[DrPop]

Wait a sec... I thought JPM was a Gigabyte man?:confused:

[Mike029]

For Sandy Bridge ---> http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=100007627+600142440&QksAutoSuggestion=&ShowDeactivatedMark=False&Configurator=&IsNodeId=1&Subcategory=280&description=&hisInDesc=&Ntk=&CFG=&SpeTabStoreType=&AdvancedSearch=1&srchInDesc=

For AMD ---> None (Ooops)

SWEET! I wonder how many stones that one box could kick out? Hmm, DiRT is down now sooo, I'll say 2 mil. in PG alone even with reduced credits. :cool:

[Fire$torm]

Wait a sec... I thought JPM was a Gigabyte man?:confused:

LOL, yes he is... except when Gigabyte doesn't produce a product he has to have :D

[DAD]

I dont think that mobo can run 8 GPUs.. look at the PCI specs:

8 x PCI Express 2.0 x16 slots - ok great... 8 slots "capable of x8" but then there's the kicker:

* PCI_E1 supports up to PCIE x16 speed (when PCI_E3 is empty)
* PCI_E2 supports up to PCIE x1 speed
* PCI_E3 supports up to PCIE x8 speed
* PCI_E4 supports up to PCIE x1 speed
* PCI_E5 supports up to PCIE x16 speed (when PCI_E7 is empty)

The description is cut off for slots E6 and E7 and E8.. so if we continue their logic:
E6 is X1
E7 is X8
E8 is X1


So basically only PCI slots 1, 3, 5, 7 can handle GPUs... however... all their slots are right up next to each other... so 4 double wide GPUs would cover up all the X1 slots and be crammed together and air cooling wouldn't work

I still need to find a TRUE mobo that will run all 8 lanes at X8 and all 8 lanes are double spaced.. not sure that exists :)

[Fire$torm]

I dont think that mobo can run 8 GPUs.. look at the PCI specs:

8 x PCI Express 2.0 x16 slots - ok great... 8 slots "capable of x8" but then there's the kicker:

* PCI_E1 supports up to PCIE x16 speed (when PCI_E3 is empty)
* PCI_E2 supports up to PCIE x1 speed
* PCI_E3 supports up to PCIE x8 speed
* PCI_E4 supports up to PCIE x1 speed
* PCI_E5 supports up to PCIE x16 speed (when PCI_E7 is empty)

The description is cut off for slots E6 and E7 and E8.. so if we continue their logic:
E6 is X1
E7 is X8
E8 is X1


So basically only PCI slots 1, 3, 5, 7 can handle GPUs... however... all their slots are right up next to each other... so 4 double wide GPUs would cover up all the X1 slots and be crammed together and air cooling wouldn't work

I still need to find a TRUE mobo that will run all 8 lanes at X8 and all 8 lanes are double spaced.. not sure that exists :)

I think you misunderstand the bandwidth requirements for GPU crunching. PCIe at x1 speed is more than sufficient to keep a GPU well fed. So an 8 GPU cruncher can be easily achieved.

As to the spacing requirements, that to is easily solved by using PCIe riser cables used every other slot like these ---> http://www.orbitmicro.com/global/riser-cards-c-33_898.html?kw=&p[116][]=813&show=all

[shralper]

GPU crunching is seriously awesome, its very satisfying to watch the projects fly to completion and finish in a quarter of the expected time. I wish world community grid would utilize GPU's!

[DAD]

Ahhhh if they only need x1.... Then I'm all for it. I assumed they needed x8

[John P. Myers]

Wait a sec... I thought JPM was a Gigabyte man?:confused:


LOL, yes he is... except when Gigabyte doesn't produce a product he has to have

Exactly! :D

[John P. Myers]

Ahhhh if they only need x1.... Then I'm all for it. I assumed they needed x8

Actually they don't even need x1, it's just that it happens to be the slowest PCIe speed available. People crunch with plain PCI slots as well, with half the bandwidth of a PCIe x1.