When I first arrived in Japan, our boss took my girlfriend and I to a local onsen. It was a little embarrassing. Not because my privates were twenty feet longer than everyone elses, but because I accidentally ditched the guy when I got in there.
First of all, sexes separated, presumably to avoid boner-issues. Then we went and undressed. When I had finished unravelling myself I couldn't locate the Japanese guy anymore. I wandered into the bath room and looked around. I saw a selection of old Japanese men, none of which I had come with, and a sphincter: some guy was bent over double pointing his right at me. "huh," I promptly thought, and sat down to wash facing the other direction.
Well, it turns out the anus belonged to my boss. I guess I came across like a scared gaijin intimidated by naked nationals. To top it all after that I lost him again, so I went and got in the hot pool outside, only to find he'd gone and sat inside. Whatevs... He got fired 6 months later so never mind. That's life.
An onsen is a hot spring. Some are natural, many are not (i.e. just regular hot water). I live in Nagano prefecture, which is pretty famous for them (natural ones). I don't think there's anything particularly theraputic or special about the water (although the Japanese insist it), but some of the views from them are stunning. Most onsen in pretty areas are perched in the hills and have huge glass walls looking out over the vicinity. Very relaxing.
Onsen that are more than basic have several pools in various combinations. Usually one hot one (around 45 degrees C) and a cold plunge pool. Often there will be a pool outside, too, which is especially nice when it's really cold out there, or snowing. Sometimes there is a "special" one - collagen or some aphrodisiac or something.
Also you may find a sauna or steam room, often up around 90 degrees C. Take your friends and try to out-man them. Then get in the cold pool afterwards. I've never been so happy or comfortable to be in cold water.
Pay your ~500yen, take off your clothes, sit down, get all soapy, wash it off and get in the water. After a couple of minutes you will probably want to get out. The heat itself is okay, but the relentless steam up the nose can't be good for you, and always makes me feel a bit odd unless I'm outside. Get out, wander about flapping your huge western bodyparts, then get back in. Then, if you're me, get a bit bored, restless and confused as to why you really came in the first place...
Then the tradition is to drink milk, for some reason. I don't know why, but it does taste really good afterwards.
Update: last month I managed to find a private onsen in a beautiful ryoukan in Kyoto.