For more information about pharmacies and drugs in Japan, check out Dan's series of articles here on AccessJ:
Pharmacies and Prescriptions in Japan
Over the Counter Drug Laws in Japan
Online Drug Sale Laws in Japan
Matsumoto Kiyoshi holds the largest market share of any chain pharmacy in Japan. Their stores are located nationwide. They have an easily-identifiable yellow sign. In my experience, I most often encounter them inside shopping malls, train stations, and along downtown shopping arcades. This may be a regional thing--the places they happened to build in the areas of Japan I've lived in/the shopping areas I frequent. They certainly do also build suburban storefronts with large parking lots. :)
SunDrug is another large chain. They maintain a presence in all 47 prefectures, albeit a token, two or three store presence in several regions. SunDrug seems to be particularly well-represented in the Kanto, Aichi, and Fukuoka areas.
Cocokara Fine is an umbrella company with several branded storefronts, including Seijo, Segami, Zip Drug, Lifort, Suzuran Pharmacy, and (Kusuri no) Kodama. Aside from the requisite Kanto presence, these stores are appear to be fairly well represented in Aichi and the far western parts of Japan, including the Chugoku region and the northern half of Kyushu.
Sugi Drug Group of stores is strong mainly in the central and kansai regions of Japan, specifically Aichi, Gifu, and Osaka. They use a very plain, blatant red sign that straightforwardly reads, くすり: "medicine."
Tsuruha Drug is exclusive to Eastern Japan and is easy to find throughout Tokyo and areas northward. It is headquartered in Hokkaido, and has expanded to command an impressive domestic market share, despite its almost complete absence from half of the country.
Cosmos Pharmacy is completely absent in eastern Japan. Its stores are focused mainly within Kyushu but stretch out to Shikoku and the western portion of Honshu, including Yamaguchi, Hiroshima, Okayama, and Kobe.
This covers many of the major players. However, other notable pharmacy chains in Japan include:
Welcia (Kanto, Osaka, Niigata)
Cawachi Pharmacy (Eastern Japan)