What Steven is saying is mostly true, but I will just put my 2p in anyway:
The whole PC gaming world is for me worth the initial investment to begin with, then you can essentially "drip feed" upgrades over time into your computer (as said above). Take my example:
In 2007 I built the following PC:
Intel Q6600 Quad Core 2.4ghz CPU (This is your processor)
Gigabyte Motherboard (this is the main board which your processor, RAM, graphics, hard drives etc plug into)
4GB DDR2 1066 RAM (This is your RAM, or system memory as it's also known. Not to be confused with your hard drive, which is where you store everything)
500GB Hard Drive (this is where everything is stored including your windows operating system, games, microsoft office, pictures, music, movies etc)
Nvidia 8800GT 512MB Graphics Card (this is what runs your graphics in games and on the desktop)
Now this cost me at the time about £500 (not including monitor/keyboard/mouse which you could buy now all in for £100), I had a copy of Windows from work also.
This is the timeline over the last few years, and due to having other priorities I haven't spent a great deal:
2010 - Bought 1TB Hard drive for more storage (£70)
2012 - Corsair HS1 Headset (£40) (my old headset broke)
2012 - Upgraded Graphics card to AMD 6870 1GB (£120), this roughly increased gaming performance by around 100% at the same settings (by gaming performance we mean frames per second, ideally you need to be looking at 60+, but bare in mind the Xbox 360 runs at 30fps!)
2012 - Bought a 120GB SSD (£130) to put windows and a few games on (this is a Solid State Drive, basically these are A LOT quicker than hard drives so load up windows and games in about 1/3rd of the time, it doesn't change gaming performance per-se)
2012 - Asus Xonar Sound Card (£30) (The motherboard already had onboard sound but I wanted Dolby Heaphone for gaming)
2013 - New Case (£70) (I upgraded my case to a bigger, more practical case so that when I upgrade next I can fit in more fans etc)
2014 - Before the Spring I will be buying the following: i5 4750k CPU, new motherboard, AMD 7870 2GB Graphics, 8GB DDR3 RAM (approx £400).
At the moment, (before the 2014 upgrade) I regularly play Battlefield 4 on low/medium settings and achieve around 40-45 average FPS. This is on a system that is essentially 7 years old apart from the graphics which are two years old. Compare my PC performance to an Xbox 360 and my PC is quite a bit better, however it will be behind the performance of an Xbox One.
After my 2014 upgrade I expect to play BF4 on high and average 60+ FPS (which will appear smooth), and on ultra probably 45-50fps. This will be better than an Xbox One.
This for me is the big advantage of PC's; since 2007 I've been enjoying better graphical performance and an Xbox 360, and soon I will be doing better than an Xbox One. But, I could also quite happily stick with my current system and still play BF4 with all my mates who have high end systems, it's my choice. It's nice knowing that when I do my 2014 upgrade my system will pretty much run most things I throw at it for the next few years, and if I really want to push my system to the next level all I would need to do is chuck in a second graphics card and it will eat every game for a long time.
I love knowing that much of the time I don't HAVE to upgrade, and if I do it's usually out of want than need and I can gain instant results without having to do things like re-install windows (unless we're changing a lot of stuff) as I keep my operating system very clean anyway.
For what it's worth I also own an Xbox 360, the only game I play on that at the moment is GTA5, once it comes out on PC it will be redundant on the 360.