I briefly considered spending 700,000yen on a nice, new road bike before thinking about buying second-hand. I came across a place called Suginami Green Cycle. There I bought the pictured bike for 15,000yen. The retail price is something between five and ten times that.
You can get a regular mama charie bike for a few thousand yen, and various road and mountain bikes for not much more. But why so cheap? Suginami and the other places like it work through a deal with the local government. In Japan, if a bike remains in one place for a certain amount of time someone will come by and attach a tag to it. The tag tells you to remove it if the bike is still in use/wanted. If you don't do so then it will eventually be taken away and crushed.
Anyway, if you need a bike then this is the place to go. The only downside is that it's only open three days a month, so you need to time it right. It should be the second Monday, Tuesday and Wednesday of each month (normally whichever Monday is closest to the 15th of the month). If there's a public holiday on one of those days then it'll be the following week.
The address:
Open 8am-4pm
Tel: 03 3327-2287
The two closest stations are Eifuku-cho and Shimo-Tokaido. It's very easy to get there with a map (and here is one), but if you want to be extra sure then use this pictoral guide.
I only know of one other place like this, and it's:
http://www.musashino-sc.or.jp/center/index.html
Musashino-shi Silver Jinzai Recycling Center
3-5-16 Nakamachi, Musashino-shi
Tel: 0422-51-3448
Mon-Fri 0900-1500, every 4th Saturday 1000- 1600
Nearest station: Mitaka
I've never been there, I just read the information here. If you've been there or know of any other places then drop them in the comments box.
PS: if you're interested I made it back to my town after a mere 16 and a half hours and 18 litres of water.

Hello, thank you for this amazing informative article!
I was looking to buy a bicycle in Tokyo and was already dreading having to spend the next month eating rice,rice,rice, and only rice.
How many race bicycles/fixies did you see? Is it mainly Mamachari-land?