This is the best alternative if you need a business server or high precision workstation. You can have brand-name machines from Dell, California Digital, IBM mainframes and workstations, Compaq, HP and many other PC manufacturers. The only argument against this alternative is that you will pay closeto the price of a Windows PC for it.
With some PCs selling for $400 you may be tempted to buy one of them and install Linux in the new machine. Be careful when you follow this route because some of these new cheap machines have devices for which no device driver exists in Linux. I will not be surprised if in the future some of these machines will either be Linux certified, or will come with Linux installed.
If you decide to build your own PC, you can buy a "barebones PC" for about $150 (see for example Magicmicro) and then you can buy locally a video card, network interface card (NIC), sound board, hard-drive and monitor. The motherboad and video cards are the key items you need to be concerned, for the others are easy to add to the basic configuration. Read the next alternative for details of the basic specifications.
If you are not afraid of opening your PC and doing it yourself this is the best alternative to have a client installation of Linux, dual-booting in Windows and Linux. You can use your current PC, add some new devices, or replace older ones, and install Linux. If you have an old 486 of Pentium that you are no longer using, you can upgrade it and install Linux. This is even better, for you will end up with two machines you can easily network and share one ISP connection, all for under $300.
Your main source to learn how to upgrade an existing or old PC is the The Cheap Linux Box page. It contains not only information on how to do it, but also links to places you can buy on-line the devices you need. You can have additional information on the various devices from Tom's Hardware Guide. If you plan to install RedHat check the hardware compatibility list.
Specific information you may need regarding your existing hardware can also be obtained (if available) in the Linux HOWTOs page. See also Modem-HOWTO, Sound-HOWTO, Video Card (compatible with X11, the GUI of Linux), Ethernet-HOWTO (for your NIC).