I'd say to try 10.11 or maybe even go back to 10.10. I just installed some new GPU's last week and had similar problems. Ended up going back to older drivers.