Good Evening, Cap'n! Well hmmm...sometimes it is worth upgrading the old beast and sometimes not.

I have been round and round myself on stuff like this at many times in my life. The biggest question that comes to mind is, this being a "mini" BTX system, can you fit a good PSU in there? If not, what is the wattage of the current PSU? That is likely to be your limiting factor on an old Dell / HP / insert "Brand" here.
CPU: If you want to use the stock cooler on the CPU (i.e. you don't want to mod the case!) then the best CPU you can stick in there is the
AMD Athlon X2 6000+ for just shy of $50 - this is due to the 6400+ CPU taking a lot more watts, and the stock cooler probably can't hack it.
RAM: 4GB DDR2 800 (as in four sticks of 1GB supported) Not sure, but *sometimes* with the latest BIOS update you can get 2 sticks of 2GB working in these old things, but they only have 32Bit Memory addressing in the chipset, so the chipset is limited at 4GB (actually Windows will only see 3.5GB - no matter if you run 64bit version or not).
GPU: Pretty much sky is the limit for crunching because you have a PCI-e X16 slot w/ 8GBs/sec bandwidth. No worries there, even though it's only revision 1 of PCi-e, the card will crunch no problem! The issue here is going to be PSU - can you put a more powerful PSU in this rig? If not, you are limited to the 305W of the original Dell PSU. Not many "big" PCI-e GPUs can run off of the little you will have to give it - perhaps you could throw a GTX 460 in there, and be safe - but that would mean NO other GPUs at all - don't even think about doing anything else to it on the stock PSU.

Your 2 PCI slots you *could* fill up with the PCI version of the GTX 430 they are talking about in this thread as well. Depending on how big your primary GPU is that you stick in that PCI-e slot, heat may become a problem - or you can just leave the case off if it's in the right setting!
Hope this helps a little.
