Trigggl, have you ever done a Minimal Ubuntu Install? I just found it existed! I think this could be a better platform for an Ubuntu GPU cruncher. Maybe not as compact as Gentoo but decidedly better than a full Ubuntu install.

JIC ---> http://ubuntuforums.org/showthread.php?t=1155961 and http://wiki.dennyhalim.com/ubuntu-minimal-desktop