Interesting problem.... any advice?

[Egilman]

Well that time of year came to do all the maintenance I've deferred for too long....

Took Blackhawk-3 down to clean the radiators and blow the dust out and other miscellaneous small issues. Two 7990's running on an Asus Crossfire V formula Z (990FX) with an FX 8350 and 8 gig of G-Skil.

After getting it all cleaned out, I reinstalled one of the cards (saving the other for the new build) and fired it up......

Watching the GPU monitor from afterburner after booting the temp on GPU #1 went straight to 104 degrees and shutdown. This is without Boinc running and took about 120 seconds. Shut it down and restarted to make sure. Computer in all other respects is normal. Boots fine, Drivers load normally. Swapped it out with the second card, same result. Flipped the bios on both the cards to the saved bios, same result.

Set them to one side.

Take down Blackhawk-2 for the same maintenance, same exact result! Upon re-installation, both cards go straight to 104 and shutdown. The only difference between the machines is that Blackhawk-2 is running on a Sabertooth 990FX R3.0 otherwise identical.

Swap cards/machines? same exact result.

Any Ideas before I conclude that they are done? I don't want to put any of the cards in the DP's and risk losing one of them.....

Last time this happened to me was last April, I lost 25% of my fleet, 5 GPU's and half the MB's.

Cards run fine till they reach shutdown.

Tried to log into bios, bios runs fine until the card reaches shutdown. (nothing obviously wrong in bios either but didn't have enough time to inspect deeply) Also this tells me it isn't a windows/software problem.

It is definitely hardware related.

Any ideas anyone? before I declare the cards trash?

WE are talking about the balance of my GPU fleet, 6 7990's 12 GPU's.

[Bryan]

Were the cards running okay before maintenance? Are the GPU fans spinning up?

[Egilman]

Yes.

They were running hot straight and normal 100% load 13 million a day before the removal/cleaning. They are AIO cooled using H55's and a custom bracket for the dual cards (for two years). Temps were in the 55-65 degree range. They were also undervolted to 1.1 volts to help control temps.

[Mumps]

Sounds like investigating the cooling solution may help. If the water cooling loop isn't pumping, that'd be a quick way for the GPU's to overheat. And if you're disconnecting things for adding/removing cards, could it be a bubble introduced in the lines that may be preventing proper cooling?

[Egilman]

I'm familiar with the bubble issue in AIO's.

If it was one or even two cards, I could see it, but four of them all at the exact same time, not likely. Besides there is a simple way to tell if they are running.

The solid probe of a mechanics stethoscope on the pump body gives you a slight whirring noise if it is running. You can actually hear them turning....

The coolers are working.

[John P. Myers]

The pumps may be whirring but that doesn't mean liquid is flowing. Air bubbles may still have the line blocked. Being that you say you've had the H55's on the GPUs for a couple of years, would i be right to assume they were used for something else before that? Fluid does evaporate out of AIOs over time and they may have run fine while undisturbed but removing them may have dislodged all the extra air that has accumulated. Being the smaller H55's, i expect evaporation to occur more rapidly when trying cool a 7990. Do you have a different, newer AIO you could test with? Or even reinstall the original heatsink/fan assembly on 1 card?

Come to think of it, didn't someone else here have issues with multiple H55's as well? I avoid Corsair so i can't relate to cooler failures

[Egilman]

The pumps may be whirring but that doesn't mean liquid is flowing. Air bubbles may still have the line blocked. Being that you say you've had the H55's on the GPUs for a couple of years, would i be right to assume they were used for something else before that? Fluid does evaporate out of AIOs over time and they may have run fine while undisturbed but removing them may have dislodged all the extra air that has accumulated. Being the smaller H55's, i expect evaporation to occur more rapidly when trying cool a 7990. Do you have a different, newer AIO you could test with? Or even reinstall the original heatsink/fan assembly on 1 card?

Come to think of it, didn't someone else here have issues with multiple H55's as well? I avoid Corsair so i can't relate to cooler failures

They were brand new when installed. I've read alot about the air bubble noise issue, but have never experienced it, I've only had experience with two brands, the Arctic Hybrids and the Corsair H-55 which are both Asatek licensed pump bodies.

I've reviewed the current opinions on AIO lifespan and the opinions run from 3-5 years. It's all opinion cause there are only reports of failure but no documented cases.

I know they will fail, I've an engineering background and anything mechanical, especially with moving parts, will eventually fail. That is a rule written in stone.

That being said, AIO's haven't been around long enough to really know and there are not actual failure reports backed up by evidence. There are just as many reports of coolers lasting 6+ years as there are reports of failures at -2 years.

In researching this I found the warranty terms revealing....

NZXT warranty is 6 years
Corsair warranty is 5 years
Swiftech warranty is 3 years
EKWB warranty is 2 years

High end pumps not using the asatek design are warranted shorter that the asatek designs at the lower end of the scale.

In many ways I'm poking around in the dark here, when I describe the problem to the Asus tech tech people they have never heard of such happening before, same as the MSI people last year... In fact I'm sending three of the cards back to MSI so they can look at them as they requested last year.... (last year I had disposed of the cards before they requested a sampling of them)

Maybe I'll get an answer this time as to what is causing this...

The best overall advice on AIO replacement is to replace when they start to not cool as well as they did new. At least that is what my research over the last 8 hours is telling me. I wish I could find a confirmed report on the coolant loss issue. But the Manufacturers tech's that are willing to talk to me are not willing to say that evap is a real issue.

Estimated absolute hardware mechanical failure on an asatek pump/cooler is 10 years. (according to asatek) But even they do not know for sure cause they haven't seen such an occurrence yet. the radiator/tubing/electronics will fail before the pump will according to them.

I don't think it was the coolers....

Anyway three of the cards are on their way back to MSI, they even are paying the freight to get them. (and one of the MB's) They want to see what the heck is going on with this combo. They agree it's a HW issue.

[Al]

They were brand new when installed. I've read alot about the air bubble noise issue, but have never experienced it, I've only had experience with two brands, the Arctic Hybrids and the Corsair H-55 which are both Asatek licensed pump bodies.

I've reviewed the current opinions on AIO lifespan and the opinions run from 3-5 years. It's all opinion cause there are only reports of failure but no documented cases.

I know they will fail, I've an engineering background and anything mechanical, especially with moving parts, will eventually fail. That is a rule written in stone.

That being said, AIO's haven't been around long enough to really know and there are not actual failure reports backed up by evidence. There are just as many reports of coolers lasting 6+ years as there are reports of failures at -2 years.

In researching this I found the warranty terms revealing....

NZXT warranty is 6 years
Corsair warranty is 5 years
Swiftech warranty is 3 years
EKWB warranty is 2 years

High end pumps not using the asatek design are warranted shorter that the asatek designs at the lower end of the scale.

In many ways I'm poking around in the dark here, when I describe the problem to the Asus tech tech people they have never heard of such happening before, same as the MSI people last year... In fact I'm sending three of the cards back to MSI so they can look at them as they requested last year.... (last year I had disposed of the cards before they requested a sampling of them)

Maybe I'll get an answer this time as to what is causing this...

The best overall advice on AIO replacement is to replace when they start to not cool as well as they did new. At least that is what my research over the last 8 hours is telling me. I wish I could find a confirmed report on the coolant loss issue. But the Manufacturers tech's that are willing to talk to me are not willing to say that evap is a real issue.

Estimated absolute hardware mechanical failure on an asatek pump/cooler is 10 years. (according to asatek) But even they do not know for sure cause they haven't seen such an occurrence yet. the radiator/tubing/electronics will fail before the pump will according to them.

I don't think it was the coolers....

Anyway three of the cards are on their way back to MSI, they even are paying the freight to get them. (and one of the MB's) They want to see what the heck is going on with this combo. They agree it's a HW issue.

Glad they are looking at it and hopefully MSI will come up with a remedy that is free to you.

As you all know, I lost a MB to a psu failure recently. When I got the replacement board I put the same H100i cooler on it. It would throttle at idle and over 75c. Prior to that temps had been acceptable. I replaced the cooler with a Hyper 212 evo yesterday and 100% load temps are less than 50c on all cores. I drilled the tanks on the H100i and got all of 1 - 2ozs of fluid out of it. So it does evaporate because there was never any sign of leakage. This is why I prefer air coolers for cpus. GPUs are another story and mine suck enough to where it was never a concern.

[Bryan]

As you all know, I lost a MB to a psu failure recently. When I got the replacement board I put the same H100i cooler on it. It would throttle at idle and over 75c. Prior to that temps had been acceptable. I replaced the cooler with a Hyper 212 evo yesterday and 100% load temps are less than 50c on all cores. I drilled the tanks on the H100i and got all of 1 - 2ozs of fluid out of it. So it does evaporate because there was never any sign of leakage. This is why I prefer air coolers for cpus. GPUs are another story and mine suck enough to where it was never a concern.

And I'm betting you bought that MB and H100 cooler from some fly-by-night outfit that won't give you any warranty either!

[Al]

And I'm betting you bought that MB and H100 cooler from some fly-by-night outfit that won't give you any warranty either!

I got it from AmaBryan....no more fly-by-night that that!