How?: Dual xeon with 7x GTX 580s

[zombie67]

Check out the top two hosts on GPUGRID (http://www.gpugrid.net/top_hosts.php). They are dual Xeon with seven GTX 580s (not 590s!). How is this done? Which mobo assuming it is home built? If it is an OEM machine, which one?

[dan]

Check out the top two hosts on GPUGRID (http://www.gpugrid.net/top_hosts.php). They are dual Xeon with seven GTX 580s (not 590s!). How is this done? Which mobo assuming it is home built? If it is an OEM machine, which one?

Someone recently made a post about running 4 5870s in a box. Supposedly you only need a 1x connection for PCIe 2.0, so he used extenders. I imagine if you have a board with 7 PCIe 2.0 slots you could put 7 580s in them. The thing about it is he has to have some custom power, at least a 20 amp circuit with two big PSUs in a server box.

Hmmm.... ain't using that Freezer down stairs wonder if I could use the 20 amp circuit off it :-)

[Maxwell]

Notice, too, that all the boxes with a crapton of GPUs in them are running Linux. Not sure what they do to accomplish that, but it works...

[Fire$torm]

Check out the top two hosts on GPUGRID (http://www.gpugrid.net/top_hosts.php). They are dual Xeon with seven GTX 580s (not 590s!). How is this done? Which mobo assuming it is home built? If it is an OEM machine, which one?


Someone recently made a post about running 4 5870s in a box. Supposedly you only need a 1x connection for PCIe 2.0, so he used extenders. I imagine if you have a board with 7 PCIe 2.0 slots you could put 7 580s in them. The thing about it is he has to have some custom power, at least a 20 amp circuit with two big PSUs in a server box.

Hmmm.... ain't using that Freezer down stairs wonder if I could use the 20 amp circuit off it :-)

That would be me. It was the Hardware Template for a Linux GPU Super Cruncher. (http://www.setiusa.us/showthread.php?1587-Hardware-Template-for-Linux-GPU-Super-Cruncher.) That 7 GPU setup is probably using 3 PSUs to distribute the required power.

[dan]

That would be me. It was the Hardware Template for a Linux GPU Super Cruncher. (http://www.setiusa.us/showthread.php?1587-Hardware-Template-for-Linux-GPU-Super-Cruncher.) That 7 GPU setup is probably using 3 PSUs to distribute the required power.

Pretty nice read. Made me rethink about how to approach lots of points.

[DrPop]

Pretty nice read. Made me rethink about how to approach lots of points. And makes me rethink the dream machine on my vision board! ;)

[YoDude9999]

Could be something similar to.....

http://www.jrti.com/products/workstations/velocitymicro/vsc455_images.html

.....could have started with a barebone chassis and the right backplane.

If it's true, the guy probably has 5k+ invested into each system. In any case, this is not a consumer grade setup.
You'd likely have to be looking into commercial grade workstations to get a setup like that. (I would tend to think anyway.)

I suppose, using extenders could get you something that would work, but it would probably be more cost effective to
go with a workstation instead.

[dan]

Could be something similar to.....

http://www.jrti.com/products/workstations/velocitymicro/vsc455_images.html

.....could have started with a barebone chassis and the right backplane.

If it's true, the guy probably has 5k+ invested into each system. In any case, this is not a consumer grade setup.
You'd likely have to be looking into commercial grade workstations to get a setup like that. (I would tend to think anyway.)

I suppose, using extenders could get you something that would work, but it would probably be more cost effective to
go with a workstation instead.

Nice pic! The pic is with Telsas. Their using 580s in the build zombi references. Reading the specs at Nvidia, the Teslas seem nowhere near the performance of the 580s, plus cost $2,000 to $3,000 each vs $500 for a 580. True? So the build your suggesting would cost in the mid 20s.

[dan]

http://www.newegg.com/Product/Product.aspx?Item=N82E16813130595

H (http://www.newegg.com/Product/Product.aspx?Item=N82E16813130595)ere you go. Eight slots. Do them water cooled, probably in a server box with two big ass power supplies and you can hold the top spot at GPUGrid. Probably only cost around $6k

[YoDude9999]

Nice pic! The pic is with Telsas. Their using 580s in the build zombi references. Reading the specs at Nvidia, the Teslas seem nowhere near the performance of the 580s, plus cost $2,000 to $3,000 each vs $500 for a 580. True? So the build your suggesting would cost in the mid 20s.

Yes, I agree with what you're saying, but what I meant was that, if he started with a system that was, "barebone", he could put whatever boards into it he wanted. 580s being MUCH less expensive than Teslas, would fill the bill quite nicely. So, 7 580 boards * $500 = $3500 + another 1.5k for box, power supplies, accessories, and whatever else, there ya go. There may even be enough change left over for a beer or 12 to celebrate with.

I went looking at his stats on Bonic and I didn't see anything that made me say, "WOW". His averages aren't anything spectacular, nor anything else for that matter. It is 3am my time however so maybe I'm just seeing cross-eyed LOL.

[YoDude9999]

http://www.newegg.com/Product/Product.aspx?Item=N82E16813130595

H (http://www.newegg.com/Product/Product.aspx?Item=N82E16813130595)ere you go. Eight slots. Do them water cooled, probably in a server box with two big ass power supplies and you can hold the top spot at GPUGrid. Probably only cost around $6k

Only one problem with that MB, is that the 580 cards are double wide cards. You'll never fit 7 of them on that MB. It'll take a commercial grade backplane to handle it. You could get 4 580 boards on that MB, but that would be the limit for double wide cards. Now, if there's a version of 580 that's single wide, well, then you'd have it made for sure. Cooling could be an issue however.

[dan]

Yes, I agree with what you're saying, but what I meant was that, if he started with a system that was, "barebone", he could put whatever boards into it he wanted. 580s being MUCH less expensive than Teslas, would fill the bill quite nicely. So, 7 580 boards * $500 = $3500 + another 1.5k for box, power supplies, accessories, and whatever else, there ya go. There may even be enough change left over for a beer or 12 to celebrate with.

I went looking at his stats on Bonic and I didn't see anything that made me say, "WOW". His averages aren't anything spectacular, nor anything else for that matter. It is 3am my time however so maybe I'm just seeing cross-eyed LOL.

Totally agree. And I thought I was bad posting at 4:30 AM.

[YoDude9999]

Even THIS (http://www.evga.com/products/moreInfo.asp?pn=270-WS-W555-A2&family=Motherboard%20Family&series=All%20Motherboards&sw=5) won't handle it.

[zombie67]

Only one problem with that MB, is that the 580 cards are double wide cards. You'll never fit 7 of them on that MB.

Right. I already have a Marshall board in my latest system. No way that will hold 7 580s.

FWIW, 1M/day in GPUGRID on a single machine is spectacular. It would probably do about 10M/day on PG.

[Maxwell]

Only one problem with that MB, is that the 580 cards are double wide cards. You'll never fit 7 of them on that MB. It'll take a commercial grade backplane to handle it. You could get 4 580 boards on that MB, but that would be the limit for double wide cards. Now, if there's a version of 580 that's single wide, well, then you'd have it made for sure. Cooling could be an issue however.
Not all of them are double-wide... http://www.newegg.com/Product/Product.aspx?Item=N82E16814130601

[Fire$torm]

http://www.newegg.com/Product/Product.aspx?Item=N82E16813130595

(http://www.newegg.com/Product/Product.aspx?Item=N82E16813130595)Here you go. Eight slots. Do them water cooled, probably in a server box with two big ass power supplies and you can hold the top spot at GPUGrid. Probably only cost around $6k


Only one problem with that MB, is that the 580 cards are double wide cards. You'll never fit 7 of them on that MB. It'll take a commercial grade backplane to handle it. You could get 4 580 boards on that MB, but that would be the limit for double wide cards. Now, if there's a version of 580 that's single wide, well, then you'd have it made for sure. Cooling could be an issue however.

The MSI BBM is a great board but it has a limitation, you cannot populate all seven slots. The limit is 6, IIRC, due to the NB chipset.


Even THIS (http://www.evga.com/products/moreInfo.asp?pn=270-WS-W555-A2&family=Motherboard%20Family&series=All%20Motherboards&sw=5) won't handle it.


Right. I already have a Marshall board in my latest system. No way that will hold 7 580s.....

Both of you are correct until one factors in the PCIe extenders. Those would make it possible, at least to the limit of the MB in terms of the number of active PCIe slots it can support.

There is a way around the PCIe slot limit. Connect a video card or two via USB ports. A USB port would give you a PCIe 1x connection. Take a look at this [H]ardOCP post. (http://hardforum.com/showthread.php?t=1522180)


FWIW, 1M/day in GPUGRID on a single machine is spectacular. It would probably do about 10M/day on PG.

That is truly amazing crunching power. If only I had that kind of money...........

[YoDude9999]

Ah yes, PG, GPUGrid, I sometimes forget myself. I stand corrected, thx zombie.
...and I stand corrected again, single wide 580s, thx max.

Ok, only $4983 for 7 580s....399+ for a MB....$500 (ish) for a pair of 1200watt PSs.....Case $180(or so), plumbing for all that hardware???....$6000+ per system. Yes it could be done.....Oh yeah, plus tax! AND the electric bill!

Thanks guys for setting me straight on that one.....And I thought I knew what I was talking about #-o

[YoDude9999]

There is a way around the PCIe slot limit. Connect a video card or two via USB ports. A USB port would give you a PCIe 1x connection. Take a look at this [H]ardOCP post.

This has my eyebrow raised. Nice find FS

[YoDude9999]

Available here:

http://www.hwtools.net/Adapter/PE4H.html

[Fire$torm]

This has my eyebrow raised. Nice find FS

Thanks but I cannot take credit for the find. It was originally posted on the "Old Forum" buy either JerWA or JPM. But it is a neat bit of tech.

On a related note......... To the powers that be: There is a TON of valuable info on the old board. I hope it can be archived here sooner rather then later.

[John P. Myers]

The MSI BBM is a great board but it has a limitation, you cannot populate all seven slots. The limit is 6, IIRC, due to the NB chipset.
As you may recall, i have the MSI Big Bang Marshal board. It has 8 PCIe x16 slots and yes, all 8 can be populated. The NB does have a limitation, but they added an additional PCX chip to add a few more lanes. So yes, you can actually put 8 580s on this board. Either 8 of the single-slot water cooled ones, or 8 double-wide ones with PCIe x16 extension ribbons and a rack mounted 4 inches or so above the mobo to actually mount the GPUs too. completely doable.

Also there's the EVGA SR2 and some UD9 board from Gigabyte that can also run 7 580s at once. Probably 1 or 2 other board out there somewhere as well. It's easy to do. The hard part is just figuring out where to put all those GPUs :p

All dual GPU cards count as 2 cards to the OS and to any OCing and temp reporting software. So in my system, the OS sees 5 GPUs, because there are, in fact, 5 different GPU chips (2 590s and a 460). 4 GPUs is by no means a limit.

[YoDude9999]

Well I think we've all been somewhat enlightened with this thread a little, I know I have. But, in any case, or without ( :) ) , it's still gonna cost a pile of dough to bring up a system like that. Finding used items on ebay may save you a small pile, but still, you'll pay dearly for it.

Lets face it, some people just have more cash just laying around than most of us do.

[DrPop]

All dual GPU cards count as 2 cards to the OS and to any OCing and temp reporting software. So in my system, the OS sees 5 GPUs, because there are, in fact, 5 different GPU chips (2 590s and a 460). 4 GPUs is by no means a limit.

@JPM - OK, I just caught up reading this. So, if you can put 5 GPUs in, that shatters the "4 GPU Myth" out there. This leads to the next question...could one *in theory* put 8 GTX 590s on that board? (I know where would you put them, heat, power issues, etc), but I'm talking strictly the interface of having 8 x16 PCI-e slots, could it connect to "16 GPUs", or is there some limit out there...just greater than or equal to 5, since we know the limit is not 4 now?:confused::cool:

[Dandasarge]

@JPM - OK, I just caught up reading this. So, if you can put 5 GPUs in, that shatters the "4 GPU Myth" out there. This leads to the next question...could one *in theory* put 8 GTX 590s on that board? (I know where would you put them, heat, power issues, etc), but I'm talking strictly the interface of having 8 x16 PCI-e slots, could it connect to "16 GPUs", or is there some limit out there...just greater than or equal to 5, since we know the limit is not 4 now?:confused::cool:


Nvidia's drivers will not allow for more then 8 cores on a single computer. You in theory can split it with ATI cards and work but I have never seen it done so that is only in theory.

[John P. Myers]

Nvidia's drivers will not allow for more then 8 cores on a single computer. You in theory can split it with ATI cards and work but I have never seen it done so that is only in theory.

13-GPU supercomputer (http://www.bit-tech.net/news/bits/2009/12/15/astra-builds-13-gpu-supercomputer/1)

Unless we can get our hands on that special BIOS, the limit for the rest of us is 8, as Dandasarge said. And since it's a BIOS limit, i doubt mixing ATI and Nvidia will help.
Of course, the limit for SLI or Xfire is 4.

[Dandasarge]

13-GPU supercomputer (http://www.bit-tech.net/news/bits/2009/12/15/astra-builds-13-gpu-supercomputer/1)

Unless we can get our hands on that special BIOS, the limit for the rest of us is 8, as Dandasarge said. And since it's a BIOS limit, i doubt mixing ATI and Nvidia will help.
Of course, the limit for SLI or Xfire is 4.

I've always thought it to be a VC Bios issue not the MB, I've never had a 8 lane MB to try it out. I never thought of the fact that the motherboard manufactures would make the VC companys write the VC "rules" in the MB Bios. they have the Bios on this page http://fastra2.ua.ac.be/?page_id=628

Hmmm you would deffently have the fastist cruncher. You could probably out produce 99% of the farms out there in RAC with one of those systems.