Dual E5 v3 build

[zombie67]

I am considering a dual E5 v3 cruncher build.

After looking at all the flavors of E5 v3s and their costs, it seems like the E5-2630 v3 is the price performance sweet spot. 8 core, 2.4ghz, $650 each. Does anyone have a different perspective?

Which mobo? E5 v3 requires the C61x north bridge, I think. It would be going into a HAF-X, so size is not important. It will hold any of them. I would be using at least 8gb DIMMs, so with dual 8 core, I would not need more than 8 DIMMs total (8 cores x 2 threads per x 2 chips = 32gb). I want it to be able to hold at least two full length, dual-width GPUs.

After a light bit of googling, it looks like there are only a few other brands that are currently offering v3 server mobos:


Newegg has only supermicro with C612 north bridge. Of those, MBD-X10DAI-O (http://www.newegg.com/Product/Product.aspx?Item=N82E16813182938) seems to be the best for PCIe cards, cos $390
Asus Z10PE-D8 WS (http://www.asus.com/us/Commercial_Servers_Workstations/Z10PED8_WS/), for $550
ASRock has a bunch, none of which look PCIe-friendly.
Gigabyte has this one (http://b2b.gigabyte.com/products/product-page.aspx?pid=5134#ov). But I don't see it for sale yet.


Thoughts?

[John P. Myers]

The Gigabyte board is a beat. A $770 (https://www.cdw.com/shop/products/GIGABYTE-MD-70-HBO-SVR-MB/3534865.aspx) beast. The reason it's so high is because the Intel X540 adapter it comes with would cost ~$500 by itself and another $350 for the LSI SAS controller. Both of those things are built in to the MD70-HB0. You may want the MD70-HB1 instead, as it should be less than half the price, but not sure exactly.

[zombie67]

Thanks for the catch! Without either available for sale, and being able to see costs, I was not clear on the differences.

That said, which mobo that *is* for sale? And which CPUs? You know best!

[John P. Myers]

I'd get the Asus board. I know it's more expensive but it's worth it. Asus and Gigabyte will both be more compatible with a wider range of RAM and GPUs than SuperMicro. Also both Asus and Gigabyte will allow for overclocking where SM won't. They'll also allow memory clock speeds that simply won't be supported by SM. Asus and Gigabyte will also have a better CPU power delivery structure and will have more effective heatsinks.And since only the Asus is available at the moment, get that :) Definitely worth the extra money, and is $100 cheaper than the same board (http://www.asus.com/us/Motherboards/Z9PED8_WS/) for the previous gen Xeons was. And here's (http://b2b.gigabyte.com/products/product-page.aspx?pid=4468#ov) Gigabyte's previous board for comparison purposes. Was hoping they'd release a v3 version of it but haven't seen anything.

I agree with the e5-2630 v3 being the best choice right now. With the Asus board, you can give them a 200MHz nudge and suddenly they become e5-2640's, which cost $300 apiece more :p

[zombie67]

Thanks for all the advise!

I'm still on the fence for pulling the trigger, for now. It's a lot of $$!

[John P. Myers]

Thanks for all the advise!

I'm still on the fence for pulling the trigger, for now. It's a lot of $$!

Yes it is. I began planning my dual Xeon system a year and a half ago with that Gigabyte board in mind. I bought a case for it (http://www.lian-li.com/en/dt_portfolio/pc-d8000/) even, but never bought the first component for it. The E5 v2's were about to be released and didn't want to feel the upgrade bug after having the v1's for only a month or 2. Then when the v2's finally came out, i couldn't bring myself to buy them either, knowing the v3's weren't compatible with the mobo's chipset or DDR3.

Things are different now though. DDR4 is here and it'll be the standard for many years. Also with the v3's, there's an upgrade path to the next-gen CPUs. Another bonus is the re-sale value on this stuff after it becomes obsolete in 2-3 years is usually pretty close to what you paid for it new. In the case of the high-end boards, sometimes they'll sell for more than you paid. Of course this is true of any of the better chipsets (Z77 and X79 for example). So at least for me, pretty sure i'll finally build the dual Xeon this time, though i admit i'm still waiting to see if another motherboard comes available that's a solid contender...which makes me wonder what ever happened to EVGA's line of dual Xeon mobos, like the SR-2 and SR-X...

[pinhodecarlos]

I've been thinking of this since I moved to UK and all my calculations (investments, electricity bill, space, time to babysit computers in function of client used, etc) point out to be better to have a dual E5 v3 cruncher than buying 5 or 10 individual machines to match the number of cores of the server. I used this site (http://www.scan.co.uk/3xs/configurator/tower-server-intel-xeon-e5-2600-v3-asus-ts500-e8-ps4) to get the prices and build the machine. My problem is the initial investment, money I don't have at the moment (£7000).

Carlos

[FourOh]

I am considering a dual E5 v3 cruncher build.

After looking at all the flavors of E5 v3s and their costs, it seems like the E5-2630 v3 is the price performance sweet spot. 8 core, 2.4ghz, $650 each. Does anyone have a different perspective?


I've considered a setup like this before, but ultimately it was just out of my $$$ range. My plan was to start with a single E5-2670 v3 which is the least expensive 12-core chip from the v3 lineup and has a few slight architecture advantages over the 2630. Then when I had another $1700 laying around :p I'd pop in the 2nd processor. They are about twice the cost of the CPU you are considering, but a complete dual-CPU system would have 48 threads versus 32. The nice thing about 48 threads is that, even with 4 threads tied up driving GPUs, you can earn 1000 hours a day on a single machine. Either will make a great CPU cruncher... best of luck making your decision!

[John P. Myers]

Just to crap on everyone's parade (including my own), here are a couple CPUs from the E5-2600 series you may not have known about (neither are available to buy yet):

Insanity (http://ark.intel.com/products/81060/Intel-Xeon-Processor-E5-2698-v3-40M-Cache-2_30-GHz)
Severe Insanity (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cache-2_30-GHz)

As you can see, the links are official. These aren't rumors.

[Bok]

I have a friend over at XtremeSystems who is running a pair of those 18core xeons. Consistently the #1 machine at WCG..

[zombie67]

impressive. I'm sure they will be impressively expensive too.

[Bok]

impressive. I'm sure they will be impressively expensive too.

Indeed. Luckily he didn't pay for the build at all I don't think :)

[John P. Myers]

Impressive indeed. Even more impressive since i'm assuming he's achieving #1 with engineering samples since according to Intel the status of those 2 processors is "Announced" and not "Launched" as it is with all other v3 chips?

[Bok]

Impressive indeed. Even more impressive since i'm assuming he's achieving #1 with engineering samples since according to Intel the status of those 2 processors is "Announced" and not "Launched" as it is with all other v3 chips?

Yes they are. I think the board he has was also engineering from SuperMicro too. I've seen a pic of the cpu's in the board (X10DAX ?) and the cpu's have Intel Confidential written on them. Board is a beast with 16 Dimm slots. i know he has 8x8Gb DDR4 in there... I was drooling!!

[zombie67]

The XtremeSystems team have a few guys who are on the list for engineering samples when intel is testing out new CPUs. They always have the new, unreleased stuff. Often they will get configurations of chips that never go to market.

[DrPop]

That stuff is just insane. Wonder how much the final chips will be for those? Can't imagine...I guess it's a farm all in one rig though! :P

[Fire$torm]

So Z,

Have you decided on taking the plunge?

[zombie67]

So damn coincidental. Just started looking again today.

[zombie67]

Question about the Z10PE-D8 WS (http://www.asus.com/us/Commercial_Servers_Workstations/Z10PED8_WS/overview/):

It has 8 DIMM slots, but will work with 2, 4 or 8 DIMMs (with two CPUs). Is there any performance penalty using less than all 8? it is quad channel, if that helps answer the question.

I as because I could save about $90 going with 4x16gb vs 8x8gb. It would also allow for adding memory in the future without having to remove DIMMs too.

[zombie67]

http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=20964449

[John P. Myers]

There is a performance penalty but i doubt it'd be noticeable

Sent from my SGH-M919 using Tapatalk

[Fire$torm]

That's a nice chunk o change.....

[John P. Myers]

I looked everywhere i could think of and Newegg beats everyone on the price of that RAM

[zombie67]

Thanks JPM! I see the mobo cheaper by a bit elsewhere. It's crazy expensive, probably because there is no competition yet.

[Aux10]

I wish I had that kind of money...

[zombie67]

http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=20964449

I just pulled the trigger. Stay tuned.

[Aux10]

Can I have half? I have cookies!

[zombie67]

The best I can do is post pictures. :)

[DrPop]

I just pulled the trigger. Stay tuned.

Wahooooooooooooooooooooooooo!!! :D

[zombie67]

I ordered two Kraken x41s. I assume I can fit them both somewhere in the HAF-X.

[Al]

With a SawzAll you can make anything fit. :p Sweet machine Z, can't wait to see what it can do.

[DrPop]

With a SawzAll you can make anything fit. :p Sweet machine Z, can't wait to see what it can do.

That's a pretty spendy case to take a SawzAll to! ;) haha :))

[Fire$torm]

SWEET! Can't wait to see the finished build.

BTW Z,
If you have any build issues, you can just pack the whole kit and kaboodle and ship it my way. I know I can make it all work....

[zombie67]

Thanks for the selfless offer! I'll keep that in mind. ;)

[Bryan]

And after he gets it working I'd more than happy to stress test it for you =))

[zombie67]

I just pulled the trigger. Stay tuned.

DIMMs arrived.

[Mumps]

So, how are they crunching for you? For that kinda money, they should BOINC on their own! :D

[Aux10]

Speaking of DIMM crunching, we need another RAM based project like FreeHal was.

[zombie67]

Mobo and CPUs arrived today. The final items, two X41 coolers, ETA Friday.

[Duke of Buckingham]

Four LGA-1150 Motherboards Under $60
http://www.tomshardware.co.uk/asrock-b85m-dgs-asrock-h81m-hds-asus-h81m-e-msi-h91m-e34,review-33170.html

[Aux10]

So, do you have any computer porn for us yet?

[zombie67]

The x41 should arrive in a few hours. Then the build will start.

[zombie67]

Dammit! I didn't notice that they shipped only one of the two I ordered!!

Their amazon page says they are out of stock, with no estimate. This is very annoying!!

[Aux10]

Nooooooooooooooooooooooooooooooooooooooooooooooooo o!!!

[John P. Myers]

Dammit! I didn't notice that they shipped only one of the two I ordered!!

Their amazon page says they are out of stock, with no estimate. This is very annoying!!

Not cool at all. Damn you Amazon!!!

[Duke of Buckingham]

Do you remember me zombie67 with half of your memories.
http://www.theblaze.com/wp-content/uploads/2012/05/Screen-shot-2012-05-15-at-6.05.43-PM.png
Very Bad Duke [Black Sheep]
2035

[zombie67]

Okay guys, I need help. I am leaving in my H60 for now, for one CPU. But I can't find any place in the FAH-X to mount the X41 for the second CPU. Along the top, all the holes are for 200mm fans.

If I can't make this work, I will have to switch to traditional fan/heatsinks. So would like some recommendations for 2011.

Thanks!

Edit: Maybe something like this would work? http://www.performance-pcs.com/bitspower-plastic-fan-adapter-200mm-to-140mm-120mm-ice-black.html

Edit2: Even with the adapter, I don't think it will work. I Just did some measurements. The one coming down from the top needs to be in the same space as the top of the one mounted to the back. I am going to have to go with normal fan/heatsinks I think.

Which brings me back to recommendations please. It has been many many years since I built a machine with those, and I am no longer familiar with what's good.

[Aux10]

There's no image so meh but what about that side panel vent?

[zombie67]

Sorry Aux10, there is no place in the side panel either. There is a fan there already, 220mm, I think. But it is right over the GPUs.

Heatsinks: I did some measurements. Not a lot of room in between the DIMMs. Room for maybe an 80mm fan in either the x or y direction. So I think it will have to be something like this form factor:

http://www.newegg.com/Product/Product.aspx?Item=N82E16835103204

And even then, 120mm is the max. 140mm is too big, and they would reach each other in the middle.

Suggestions anyone? Anything wrong with that coolermaster? Is there a better one like that?

[Aux10]

Why not go all out and try submersion cooling! Hehe. (it's been done btw)

[Al]

If you plan on OCing it doesn't look like a real strong performer. Though not cheap, I've always been pleased with Noctua coolers. Would this one fit? (http://www.newegg.com/Product/Product.aspx?Item=N82E16835608025&nm_mc=KNC-GoogleAdwords-PC&cm_mmc=KNC-GoogleAdwords-PC-_-pla-_-CPU+Fans+%26+Heatsinks-_-N82E16835608025&gclid=CPD5nomt7cQCFdgGgQodXl0Abw&gclsrc=aw.ds)

[zombie67]

Good news! I remembered that Thanos has an H100, the double wide version that mounts to the top. Duh. So I pulled it out, cleaned it up, and it fits with the H60. The X41 is significantly thicker. If they were both kraken, it may not have fit. So I will put the X41 in Thanos. All is good on the cooling front now.

Next problem No way the main/large power cable will reach to the mobo connector. I didn't notice it was on the TOP of the mobo. I am going to have to get an extender to make it work. Maybe Frys will have one. I will check tomorrow.

Similar to any home project, these things always take three trips to the store. Unfortunately, "store" is mail order for me, normally. Grrrr.

Now to put Thanos back together.

[Al]

Glad you got the cooling sorted out...........we will wait patiently for pictures and results.....................got any yet?

[John P. Myers]

In the top of my HAF X i left the 200mm fans in the upper section. Below the 200mm fans i mounted a 360mm rad and 3 120mm fans are mounted to that. All you'd need for the extra x41 is a 120mm to 140mm fan adapter.

Edit: like this http://www.performance-pcs.com/fan-ducts-adapters/bitspower-plastic-fan-adapter-120mm-to-140mm-black.html

[zombie67]

Whelp, it all powered up. I got the main power cable to reach.

But no video out at all. The LED says 67, which means "starting up". I think I am done for the night. Maybe something will be obvious in the morning.

The X41 is now installed in Thanos. This is my first Kraken, so I will play with it a bit before shipping it off to the buyer of the set. But not until I get this new system up and running.

[zombie67]

Ding ding ding!

https://dl.dropboxusercontent.com/u/55884901/Screenshot%202015-04-11%2010.36.33.png

https://dl.dropboxusercontent.com/u/55884901/IMG_1406.jpg

https://dl.dropboxusercontent.com/u/55884901/IMG_1408.jpg

https://dl.dropboxusercontent.com/u/55884901/IMG_1407.jpg

[Aux10]

Sweet! Just stick two more GPUs in there and you're golden.

[zombie67]

I have 4 DIMMs for a total of 64gb. You can see in the task manager. I just had two when I took the picture while I was debugging.

[John P. Myers]

What are your CPU temps like under load vs. each cooler? I can't read that first image too well :/

[Mumps]

What are your CPU temps like under load vs. each cooler? I can't read that first image too well :/
If you right click on the image and choose "View Image" it clears up real nice. And you can zoon manually if you still have issues. :)

[John P. Myers]

If you right click on the image and choose "View Image" it clears up real nice. And you can zoon manually if you still have issues. :)

Ah I see now. I did enlarge the image before, but I was viewing it on my phone since i'm at work. I couldn't quite squint enough to see it. Now i'm on my 28" 1080p work monitor and I can see it just fine :D

[DrPop]

Wow! :) That's just incredible looking. I know I live a tech sheltered life, but never seen anything like it, 64GB RAM...and I always thought 16GB was a lot. :P