Daniel L. Moody (no, not that D L Moody!) has been working for well over a decade on what makes a good graphical language. He brings together previous work in cognitive psychology (which gave us the much-used and much-misunderstood 7±2) with empirical studies of graphical language usability. I've referenced him in some of my talks (e.g. Code Generation 2009 keynote), but I've always been frustrated by the lack of a freely available, in-depth presentation of his research.
"In a nutshell, my paper defines a set of principles for designing cognitively effective visual notations. Visual notations have been used in software engineering since its earliest beginnings, and even before - the first one was designed by Golstine and von Neumann in 1946 - but surprisingly, this is the first time someone has tried to do this."
Daniel says he received a phone call last month from Grady Booch, one of the architects of UML (the industry standard language for software engineering), who is now Chief Scientist at IBM.
"He [Booch] told me he loved the paper and only wished he had this when they designed UML - if so, things could have been very different."
That last comment presumably refers to Moody's testing of UML based on the principles he collected. Predictably, as a black and white language relying on rectangles and solid or dashed lines, UML doesn't do very well in the analysis. You can see the slides from his SLE'08 talk on Evaluating the Visual Syntax of UML, which also form an easier introduction to the principles than the somewhat academic article above (151 references!).
Here's a picture from the slides: see how long it takes you to find the "odd one out" in the three grids below:
Another notation researcher I've referenced frequently, e.g. in the DSM book, is Alan Blackwell, whose PhD thesis showed that pictograms are better than simple geometric symbols or photorealistic bitmaps in graphical language notations. Alan is part of the wider research community of Cognitive Dimensions of Notations, whose site also has a lot to offer the graphical language designer.