I see what you’re saying
Over at Interesting Thing of the Day there’s a nice article on Synesthesia:
There are at least 50 different types of synesthesia, involving various combinations of senses both as the triggering stimulus and the secondary response. Some forms of synesthesia are experienced as multiple modalities of a single physical sense. For example, seeing a number might evoke a certain color for one synesthete, while in another person the same number might cause a different visual sensation, such as a pattern or shape. One sense may also trigger another, as in a tactile sensation that has a taste. But not all synesthetic experiences are restricted to the five senses. In some synesthetes, a word or sound might evoke a sensation of motion, or even a kinesthetic response, inducing the person to assume a particular physical position. There are also cases in which abstract concepts, such as days of the week or months of the year, cause the sensation of shapes, colors, or other experiences.
Go read the article or listen to the podcast (9:26, 8.64 MB, mp3).