View Full Version : dual xeon build
MindCrime
12-06-14, 04:10 PM
I've been interested in a dual cpu build for a while now and I think I've found the sweet spot for me, where performance meets the wallet and that's the x5650 area. I think a few active forum users have these in use. I'm getting near the point of buying and did as much research as I could think of. So in my mobo search it needed to be dual1366 and i wanted to have at least one pcie2.0+ slot, this is the best thing I came up with: http://www.ebay.com/itm/191310694068 and the specs http://www.supermicro.com/products/motherboard/QPI/5500/X8DTi-F.cfm
That's a package, they also sell it broke down all the way to the mobo. They also sell it with cpus but i think I'll buy them separately for the savings. I asked the seller about the PSU it comes with, as I didn't think it had dual 8pin psu connectors but he tells me it does and has been tested with it. My big concern is unforeseen costs, it looks like ram isn't cheap for registered ecc, thats what I want right? Also if I need a lot, or else whats the point of 24 threads? I was thinking at least 24gbs (6x4gb), pretty sure the mobo manual said optimum is 3dimm + 3 dimm or 6 + 6 (triple channel?)?
Right now my unforeseen costs are RAM, ~150$?
HDD: i might have something here. (SAS = scsi on sata connector?)
GPU: i have a pcie2.0 gtx 560 that needs a home or this 270x
Case: plywood if i have too
OS: linux
Peripherals: have monitor,kb, mouse I don't think it has onboard audio.
Any experience or input would be greatly appreciated.
zombie67
12-06-14, 04:21 PM
I don't think you have to use ECC memory. If true, I would just get regular memory. Make sure you get at least 2gb per CPU thread.
The key is "Registered." That's what says "Server RAM" right there, and that's what makes it more expensive. I don't think I've ever run across Registered RAM that wasn't also ECC. And I also doubt that, if you did hunt it down, that it'd be less expensive than the more popular ECC Registered RAM found in most Servers from that era. Also, going to that Unbuffered RAM may be more cost effective, but again, because it was a rarely used option (as far as I ever saw) there's not gonna be tons of it available because there wasn't enough demand to have created an inventory for aftermarket like this.
So MindCrime, this won't just be a cruncher? Only reason to have Sound I'd think. :)
Also, where did you get SAS from? The both link specs only indicate SATA. SAS actually stands for Serial Attached SCSI (http://en.wikipedia.org/wiki/Serial_attached_SCSI) and is pin compatible with SATA, but the controller must support it. For example, most of the HP RAID controllers will work with either SAS or SATA drives, you just can't have both types on the same bus at once. Basically, SAS evolved from SCSI whereas SATA evolved from ATA/IDE. :)
MindCrime
12-06-14, 07:03 PM
I'd be happy to use regular, non ecc/registered ram. Especially if I should use 2GBs per thread, that's 48gb. Any input on sourcing ddr3 ~1300mhz in that size... six x 8gbs or twelve x 4gbs?
It's pretty much going to be a workstation that will crunch, capable but rarely used for anything too demanding other than crunching. Since this thread started I was steered toward the dell t5500s which are pretty much the hardware I want all wrapped up. And putting things in the price range I want. Much appreciated for the heads-up.
The SAS thing was me thinking about these unknown drives I have in a box somewhere. I thought they were SAS drives but they are Sata 1.5gbs... probably staying in the box.
Update: been doing some browsing any I'm leaning toward one of these http://www.ebay.com/itm/121474655596 adding lots of non-ecc ram, and adding whichever gpu plays best, gtx 750ti, gtx 560, or r9 270x.
Just keep in mind. Registered memory (http://en.wikipedia.org/wiki/Registered_memory) is a unique beast. You can't use any old DDR3 DIMMs in there. They must state that they are "Registered." Registered memory has the key "notch" in a different spot than normal DDR3 DIMMs.
Compatibility
Usually the motherboard must match the memory type: registered memory will not work in a motherboard not designed for it, and vice-versa.
zombie67
12-06-14, 08:29 PM
You can get by with just 1gb per thread. But just know there are a handful of projects that need more. For those you could either limit the number of tasks, or turn off hyper threading. For me, I always go with 2gb so I don't have to worry about it. It also helps with running multiple VMs at the same time.
John P. Myers
12-06-14, 11:40 PM
You can only use registered ECC RAM with the Dell T5500. 72GB limit (9 slots) if you have the riser card for the 2nd CPU. 48GB (6 slots) without it. That's right, you only get 3 RAM slots for the 2nd CPU (lolDell). Also your GPUs are limited to 150W and it's quite likely you may only be able to use Quadro/Firepro/FireMV/Tesla GPUs, not the standard ones.
Though as a bonus, it does have audio on-board ;)
I was looking a couple of days ago on Ebay and there was a listing for 24G of DDR3 ECC memory for $150. 6 sticks IIRC. The memory was for a T5500.
My T5500 has 36Gb RAM, I've rarely wanted more - but then I don't run VMs. But it has two x16 PCIe slots and I've used everything from dual 5850s to a 7950 + 7790 with no problems. Not an elegant rig, but gets the job done
John P. Myers
12-07-14, 08:17 AM
My T5500 has 36Gb RAM, I've rarely wanted more - but then I don't run VMs. But it has two x16 PCIe slots and I've used everything from dual 5850s to a 7950 + 7790 with no problems. Not an elegant rig, but gets the job done
Good to know. I guess Dell just didn't bother updating their compatibility list, or they didn't want to associate their server with "common" GPUs to appear more professional.
1894
cineon_lut
12-08-14, 01:38 AM
Good to know. I guess Dell just didn't bother updating their compatibility list, or they didn't want to associate their server with "common" GPUs to appear more professional.
1894
I'm not the biggest Dell fan either, but the rigs are built pretty tough. Those are haunted forest's weapon of choice
Vic (mobile)
I just bought one also. Should be here on the 11th, just in time for the FB New Year. :)
MindCrime
12-08-14, 02:42 PM
Good to know. I guess Dell just didn't bother updating their compatibility list, or they didn't want to associate their server with "common" GPUs to appear more professional.
1894
Maybe that 150w is 75w(pci power only)x2 ? You know how they like to be vague and omit reasons and if/this/then/that stuff.
I imagine these workstations and other socket 1366 server gear is hitting their EOL or end of contract (leases?) and thats why 1000$ msrp cpus are now ~75$? I wonder why the ram hasn't dropped as fast as the other hardware. For those people still upgrading ram before they make the next platform upgrade?
Well I'm ready to pull the trigger I just want to make sure I can get all my other christmas shopping done and still buy this present for me. The question is do i buy now and put off shopping until payday, friday, or go shopping now and buy this on friday...oh the decisions.
I'd like to learn and get some experience with VMs with this rig and I'll probably end up with the 2gbs of ram per thread but I think in the mean time I'll get one with 4/6/8 and add whatever i need to 24gbs...ECC registered, pretty much right away. And as I learn and get things setup and going I'll see about maxing out the ram.
I read these x5600s OC very well. But on a dell mobo, and for crunching with that many threads, its just not really needed and probably not easily done or worth it?
The T5500 was available with 2 graphics cards of up to 150 Watts each, so for instance a Quadro 4800 is rated at 150W and has only one 6-pin (75W) PCI-e connector. Notionally, it gets the other 75 Watts from the slot. Since my box only has one hard drive, I have a few leftover 15-pin SATA power connectors - so I use a dual SATA power to 8-pin PCI-e to get additional power to GPUs as needed. I've had an OC'd 7950 and OC'd 7790 installed together - together they exceeded 300 Watts but I never had any issues.
You're right - I don't believe you can OC on a Dell workstation/server mobo. What you give up in tweakability you gain in stability. To me, having the most "up" time is most important for a crunching rig like this.
Good luck!
pinhodecarlos
12-08-14, 03:28 PM
Anyone living in the UK? If so please check this site: https://www.bargainhardware.co.uk/
What you give up in tweakability you gain in stability. To me, having the most "up" time is most important for a crunching rig like this.
Since you so highly recommended the T5500 I decided I needed one :D Actually it was Al's fault!
Adding 24 threads :D
MindCrime
12-09-14, 07:34 PM
Can anyone confirm if the dual socket t5500s come with the memory riser card or if its an option? I was just looking through the manual (memory stuff is conveniently at the beginning) and found the DIMM population to be a bit in depth. Oops, looks like the 2nd cpu is on the riser along with the memory.
ftp://ftp.dell.com/Manuals/Common/precision-t5500_setup%20guide_en-us.pdf
Also worth noting is the following:
Your computer uses 1066 MHz and 1333Mhz DDR3 unbuffered or registered ECC SDRAM memory. DDR3 SDRAM, or double-data-rate
three synchronous dynamic random access memory, is a random access memory technology. It is a part of the SDRAM family of technologies, which is one of many DRAM
(dynamic random access memory) implementations, and is an evolutionary improvement over its predecessor, DDR2 SDRAM.
Does the part in bold confuse anyone else? Is that unbuffered NON ecc ram and registered ecc ram? Or is that unbuffered ecc ram and registered ecc ram? I found some listings for some unregistered memory for t5500s but it was ddr3 1066 but if you search ebay or anywhere for that matter for "dell t5500 memory" you pretty much get ECC registered only. Everyone runs ecc/reg memory in these, right?
Does the part in bold confuse anyone else? Is that unbuffered NON ecc ram and registered ecc ram? Or is that unbuffered ecc ram and registered ecc ram? I found some listings for some unregistered memory for t5500s but it was ddr3 1066 but if you search ebay or anywhere for that matter for "dell t5500 memory" you pretty much get ECC registered only. Everyone runs ecc/reg memory in these, right?
Both the unbuffered and registered are ECC memory. The ECC stands for Error Correcting Code and it has the ability of correcting single bit errors that may come from a memory location. It is like parity but better.
In buffered/registered RAM there is a buffer (active circuit) between the memory stick and the memory controller. That reduces the load on the memory controller's IO lines when you have boat loads of memory modules.
I don't know of a single server that doesn't use ECC memory modules since the RAM is error correcting - at least to a point!
I would guess that the riser is part of a 2nd processor kit. If you look on Ebay you will find a number of the kits for sale ... not cheap either. I assume they have the riser, heat sink, and fan. I doubt any of that comes with a single processor unit.
The above is shaking cobwebs since I haven't been involved with this stuff for over 10 years :D
MindCrime
12-09-14, 10:08 PM
Thanks for the confirmation on the ram, I'm just trying to get my ducks in a row so I can order it too and get them both around the same time. Can anyone share if they used a particular ebay/amazon/.. vendor for their t5500 ram?
I've bought T5500 memory from this seller: http://www.ebay.com/usr/1-800-4-memory
But I usually just find a listing with the "Make Offer" option and try to get it for around $5 per Gb.
Al and I ran some benchmarks of the T5500 against my I7 3930K @ 4.2G. We used Universe@home since the WU are short and a consistent length. Here are the results.
3930K @ 4.2G
1814.6s time per WU
2116.5 credits per thread
25,398 credits/day total
T5500 E5650 2.66G
3005.6s time per WU
1277.5 credits per thread
30,659 credits/day total
T5500 E5675 3.06G
2683.6s time per WU
1431 credits per thread
34346 credits/day total
Either T5500 will out perform the I7 3930K and the entire system costs less than the I7-3930K processor alone :D
Yeah, but then ya gotta pay for the electricity to run it! :)
That's why I parted with all my gpus. ;)
I tested my dual X5650 T5500 a few days ago - it pulls 250-285 watts measured at the UPS with a full CPU load and idle GPU. Mine typically bounces between 2.8 and 2.9GHz... Intel lists the max turbo speed as 3.06GHz.
Has anyone tried the upgraded Dell performance heatsink? I'm curious if that would allow the Xeons to stretch their legs a bit.
And I turned off all my miners :)
I tested my dual X5650 T5500 a few days ago - it pulls 250-285 watts measured at the UPS with a full CPU load and idle GPU. Mine typically bounces between 2.8 and 2.9GHz... Intel lists the max turbo speed as 3.06GHz.
Has anyone tried the upgraded Dell performance heatsink? I'm curious if that would allow the Xeons to stretch their legs a bit.
Got a link to the upgraded heatsink?
https://www.google.com/search?q=U016F+dell&rlz=1CDGOYI_enUS590US590&espv=1&hl=en-US&source=lnms&tbm=isch&sboxchip=Images&sa=X&ei=XIeLVJCjMYGHNtGfgsAO&ved=0CAkQ_AUoAw&biw=320&bih=504
This is the stock heatsink: https://www.google.com/search?q=t021f%20dell&rlz=1CDGOYI_enUS590US590&espv=1&hl=en-US&source=lnms&tbm=isch&sboxchip=Images&sa=X&ei=zIeLVKKEO8SqgwSNk4OACQ&ved=0CAkQ_AUoAw&biw=320&bih=504
MindCrime
12-20-14, 05:38 PM
This thread started with me trying to fish for input on a dual xeon build from mobo up. So i thought I'd give it an update where you can see all your input manifested.
I now have a dell t5500 next to me. It has dual x5650s and currently 36gb of ram (until 8gb 2r gets cheaper). It came with a wd 10k raptor (could have been worse) and no OS. I have been playing around with a bunch of Linux distros this past month and linux mint mate went on it right away and easily, (this is with 12gb of ram at the time). I added a few projects to boinc, WCG, yafu and some others. I didn't give it too much time to settle in and get a lot of different apps/projects/tasks but I was really impressed by how little ram it used. I decided i didn't want to fuss with more up to date nvidia drivers on linux and installed win 7 64bit after some tail chasing (when installing from USB drive partition issues).
I popped a gtx 750ti in it between Linux and Windows, it has at least 1 pcie 6pin, i thought I saw two but i got a near reference model that requires no extra power. It is still limited to 38.5watts in vbios though.
So right now its pretty happy except the 2nd cpu well cpu1 reads anywhere from 15c-30c higher than my cpu0 cores. Cpu0 doesn't ever seem to break 70c while Cpu1 spends most of its time in the upper 70s and 80s. Unless I'm mistaken cpu0 is on the mobo and cpu1 is on the riser. The mobo mounted cpu doesn't even have a fan attached to the heatsink, while cpu1 on the riser has a much fancier heatsink and well who knows how good that fan is. I'm not quite sure what my temp solution is going to be but there will definitely be a change. I've ran it with the side panel off and on, not much different for cpu1, wasn't paying attention to cpu0. In a new case I don't see why an aftermarket 1366 cooler like a Cooler master 212 wouldn't work/fit? Maybe dimm clearances?
Let us know if you cure the heat problem. Both Bryan and I see the same issue with cpu 1. To be honest, I'm not too concerned about it.
Edit: I did change the thermal paste on that processor with no change in temps at all.
MindCrime
12-21-14, 01:00 AM
Edit: I did change the thermal paste on that processor with no change in temps at all.
That thought crossed my mind but when you said you and bryan both have the 2nd cpu high temps too I figured it wasn't thermal paste, wish it was that easy. My theory is that all the hot air generated from the first cpu continues on to "cool" the 2nd cpu, but obviously its now hotter. I thought an AIO would be good for the 2nd cpu if you can make it work. I figure it gives you the space savings on the cpu mounting area and is near immune from the exhaust of the 2nd cpu depending on where the AIO res is mounted.
There looks to be space for two 90mm (my guess) fans right above the I/O area inside the case. I'm going to wait for a day off to crack it open and actually pull the riser and have a look. In the meantime I have one 120mm fan mounted(taped) over the area i think is for the smaller fans. It definity pulls a lot of hot air out, temps dropped a few C but a permanent solution is needed.
Whats funny is the 2nd cpu seems to be favored, more work load on those cores than on the first cpu, could just be the current WU assignments though.
John P. Myers
12-21-14, 01:10 AM
This should solve your problem: http://www.performance-pcs.com/nexus-80-or-92mm-to-120mm-fan-adapter.html
Only 21 left in stock at this massive discount. Better get on it. This will allow you to mount a 120mm fan/rad to an 80mm or 92mm case exhaust port and it's low profile.
Keep in mind, it's a Dell and you may not have the connectors to power much additional fanage.There is an additional molex and 2 pci-e. The free molex means at least 2 should be possible.
MindCrime
12-22-14, 02:00 AM
Keep in mind, it's a Dell and you may not have the connectors to power much additional fanage.There is an additional molex and 2 pci-e. The free molex means at least 2 should be possible.
This is exactly the hurdle im on right now. I'm using the free molex for the 120mm fan but yeah the mobo looks to not have any open fan headers. And the ones it does have are 5pin...wtf. The riser has two fan headers on it one for the ram blower and one for the cpu cooler. Don't know how they're controlled. I really haven't noticed much fan speed variance at all as a matter of fact. It's pretty damn quiet of a machine before I added that 120mm fan. I wonder if a program like hardware monitor will show sensors for the fans. I have a four channel fan controller that can fit in one of the free front drive bays, but of course if the mobo has to control the cpu fans thats not gonna work.
How do water cooled builds defeat CPU FAN errors?
John P. Myers
12-22-14, 04:43 AM
How do water cooled builds defeat CPU FAN errors?
The pumps on water cooled units plug into the CPU fan header. Not sure how that would work on loldell's proprietary pin configurations though. What i would do is relocate the fan that's plugged into the CPU fan header, but leave it plugged in there. Then run a water cooled (or air cooled) CPU cooler off of your fan controller.
This is exactly the hurdle im on right now. I'm using the free molex for the 120mm fan but yeah the mobo looks to not have any open fan headers. And the ones it does have are 5pin...wtf. The riser has two fan headers on it one for the ram blower and one for the cpu cooler. Don't know how they're controlled. I really haven't noticed much fan speed variance at all as a matter of fact. It's pretty damn quiet of a machine before I added that 120mm fan. I wonder if a program like hardware monitor will show sensors for the fans. I have a four channel fan controller that can fit in one of the free front drive bays, but of course if the mobo has to control the cpu fans thats not gonna work.
How do water cooled builds defeat CPU FAN errors?
I have the same temperature difference on my T5500 - obviously it's a design issue. As long as the 2nd CPU isn't consistently over 85-90C I don't think you'll have any problems. If you are trying to reduce temps in order to stop the CPUs from throttling, additional fans may help. As I mentioned earlier, there is a high-performance heatsink available (Dell part U016F) but only for CPU 0... I don't think that will help CPU 1 though. I think your best bet is to add an exhaust fan or two at the bottom rear of the case - I measured mine and it looks like two 80mm fans will fit the hole pattern.
To add a bit more intake, you can install a hard drive fan assembly (unless yours already has one). There is a fan header for this on the mobo up by the SATA connectors. If you only have one hard drive, you can install it in this caddy and remove the hard drive tray for better case air flow.
http://www.ebay.com/sch/i.html?_mPrRngCbx=1&_from=R40&_sacat=0&_sop=15&_nkw=dell+wh216&rt=nc&LH_BIN=1
In my experience, in order to enable manual software fan control on a Dell workstation you have to disable driver signature enforcement on startup. You can do that by hitting F8 during boot and using the menus or you can download and install a Driver Signature Enforcement Overrider. Even then, it can be tricky.
Fire$torm
01-08-15, 07:37 PM
I have a couple of suggestions for your power issue.
*A tandem PSU setup.
*An optically switched on external 12V 2~4A power brick. (This would be a DIY project)
I installed the optional hard drive fan in my T5500 workstation today... sadly it did not improve CPU temperatures at all. It seems to have helped the GPU temp by 1 degree C, but I don't think that's worth the cost, trouble, and additional noise.
http://i.ebayimg.com/00/s/MzY5WDU1MA==/z/hbgAAOSwD0lUjwUA/$_12.JPG
MindCrime
01-22-15, 08:51 PM
I was going to start a new thread when i SOLVED my 2nd cpu temps but i haven't done that yet. What I have done is fooled around a bit and figured out what didn't help. I unmounted the 2nd cpu fan and removed the black shroud. I noticed how little effective surface area the cooler has. I believe the issue is purely the power dissipation capacity of the heatsink itself. I also found that with the shroud removed the fan was not mounted flush against the heatsink. I was able to drop a 2-3c on the hottest core with the black shroud completely removed and the stock fan mounted flush to the heat sink.
I also found two 80mm fans that fit perfectly in that unused fan location right above the I/O section. listed at 7$ http://www.newegg.com/Product/Product.aspx?Item=N82E16811999199 They are well reviewed, pretty much silent, and the fan blade is designed to be pulled and removed for easy cleaning.
While i was in there I took note of the socket backplate on the riser card. It looks like it would be aftermarket HSF friendly, ie theres probably enough space to mount an aftermarket socket 1366 heatsink backplate. I've been looking at some just not my top priority. Currently the highest temp is core #6 on cpu2 and it runs 79-80c and been running 24/7 for a few days.
Powered by vBulletin® Version 4.2.4 Copyright © 2024 vBulletin Solutions, Inc. All rights reserved.