TR-0801

Laboratory of Psychology and Cognitive Sciences

University of Genoa (Italy)

Technical Report 0801

A method for a cognitive analysis
of graphical presentations

Alberto Greco
 
1. Introduction
The benefit of graphical presentations (drawings, diagrams, graphs, etc.) as a powerful tool in several cognitive tasks (mathematical learning, reasoning), or as a communication aid in scientific, instructional or technical fields, or in web-based instruction, is well known (Sierpinska, 1992; Lesh, Behr, & Post, 1987; Mayer, 2003; Schnotz & Bannert, 2003; Gagatsis & Elia, 2004). Their efficacy is traditionally not questioned (“a diagram is worth ten thousand words”, Larkin & Simon 1987). Recently, however, this belief is being considered more critically, because how these supposed beneficial effects are produced is still unclear.
Some relevant data in cognitive tasks are abstract and can hardly be translated into a graphic format. Therefore, drawings are normally enriched with words, numbers, captions or other symbolic elements, which express information that cannot be conveyed by a simple drawing. The problem of representation of abstract properties in a pictorial format is specially relevant when dealing with instructional material (cfr. graphic organizers: Robinson & Kiewra, 1995; Robinson, 1998).
This enrichment is not just addition but integration, because what is added, where it is put, what information is given, are crucial aspects. The question is, then, how cognitive representations of pictures and other symbols interact in order to convey meaning. In particular, the case of the integration of verbal and pictorial elements deserves a special interest, along with studies about the cognitive properties of such two kinds of representation.
According to some scholars, the use of both verbal and pictorial elements is always beneficial (Mayer, 2003); but according to others this is not the case, either because of a possible cognitive load (Sweller, 1988; Sweller, van Merriënboer & Paas, 1998) or a possible interference between the two kinds of information (Schnotz & Bannert, 2003). Such question, in fact, is still unanswered. For example, according to the “spatial contiguity effect” (Mayer, 2003), learning is easier when words are spatially close to pictures, but it is not explained why. It seems hardly plausible that it might work just because both verbal and pictorial elements are present.
Many theories make reference to “mental integration”. The most usual explanation of integration, in the vein of the Paivio’s dual coding theory, is that the two kinds of information are processed differently, giving rise to different mental models. The integration would be the result of a mapping of elements of one model into the other, or of some dynamic process like skipping back and forth between text and picture.
Existing models, however, only account cognitive processes used to take advantage of graphic displays, but not processes used to construct pictorial representations by translating conceptual representations into a propositional format, and not how this construction is done according to the demands of different situations. Most research has been conducted about how ready-made presentations (e.g. adjunct displays) can or cannot help learning (Carney et al., 2002), but very little research exists about how people create their own presentations and how to assess their effectiveness in full representing the informational content (concept properties and relations). This cognitive problem concerns instructional designers but also learners that help themselves by constructing concept maps, diagrams or other kinds of displays by jointly employing texts and pictures.
The aim of the present work is to propose a new approach in considering representation of text and pictures, that takes into account the type of situation to be represented and its conceptual demands. The study was motivated by the idea that in order to be able to say how pictorial and symbolic elements interact, a common way to describe concepts expressed by both systems is needed.
In particular, the present study included the following steps:

(a) selecting some situations, that expressed some standard properties and relations(b) studying how situation descriptions were empirically translated from a text format into pictures(c) developing a representation system specifically designed for coping with pictorial meanings(d) assessing the effectiveness of descriptions

The last step, assessing effectiveness, was only sketched in the present analysis, because – although the meaning of the expression “to assess effectiveness” of visual presentations seems clear enough (i.e. to express the same concepts, or gist, key ideas, as the verbal input) – in fact it may have different meanings. It may refer to different specific tasks (problem solving, communication, etc.) and different dependent measurements can be adopted (e.g. correct responses/reaction times in recognition or recall, number of problems solved, etc.). A specific study will be devoted to this subject, which deserves to be treated more thoroughly.
2. Method
 
Materials
 
Our study started with sentences that referred to situations of different kinds. We considered situations that included interesting (from the pictorial/symbolic integration point of view) conceptual aspects or relations. Sentences could describe mathematical or non-mathematical situations. There were spatial situations (relations such as “above”, “nearby”, etc.), set situations (where a set is divided into parts or subsets, like in arithmetic situations), and time situations (where events occur in different stages in time).  Table 1 shows general characteristics of situations, with examples.
There were four cases: 1) no pictorial-symbolic integration was required (relations could be expressed only by pictorial elements);  2) use of numbers or conventional symbols was implicitly required; 3) information could be expressed indifferently by pictorial or symbolic elements; 4) a double representation was required in order to account for transformations that implied a first-after sequence.
Descriptions did not concern problem solving situations, because preliminary trials had shown that a solution-oriented representation involves some disadvantages: participants would have been more anxious and would give more importance to solution strategies than to representation strategies.
Table 1 – General properties of situations
case 1 – Relationships expressible by pictorial elements
a)       No relationships
b)       Topological or ordinal spatial location
e.g.: above / between / adjacent
c)       Metric spatial location
e.g.: near to…, far from…
case 2 – Properties expressible only by numbers
Non concretely countable objects
e.g.: one hundred houses
case 3 – Properties that require conventional symbols
a)       Qualification
e.g.: is red   /   is green
b)       Single belonging or possession
e.g.: Alan’s
c)       Multiple belonging or possession
e.g.: Alan & Burt’s
case 4 – Mixed (can be expressed as pictures, symbols, or both)
a)       Concretely countable objects
e.g.: two houses
b)       Comparison, difference, correspondence
e.g.: more than   /   less than   /   how many
case 5 – Transformation
Reduction, addition (in this case there are two boxes, representing first-after times)
e.g.:  Alan had …, then he gave …  /  , then he bought …
Table 2 (left column) shows sentences. Situations described by the sentences were arranged in increasing order of complexity and abstraction level.
Representations in the rightmost column in table 2 have been expressed using a predicate-argument notation. For example, the representation of sentence 14 means that, in order to fully express its meaning, one has to say that there is a first garden, possessed by Alan, which has a first set of trees  (trees1), whose number is 50 (uncountable). One must also draw the inference that there is a second garden, belonging to Burt, which has a set of trees as well (trees2), and the sentence meaning to be conveyed is that the number of trees in the first set is greater than the number of trees in the second set. This sentence is particularly difficult to be expressed using only  pictures, because of the presence of an uncountable quantity but also because an inference must be drawn and there is an unknown quantity (NUM-X). The distinction between countable and uncountable quantities has been based on commonsense assumptions (it is normally unlike to draw a picture for 100 houses); in some cases, when such assumption is not so clear (e.g. 8 books might or might not be drawn) it has not been specified.

Table 2 – Sentences and logical representation

Sentences*

Objects, properties, and relations to be represented (pred-arg notation)
1.        A house
OBJ(house)
2.        Two houses
OBJ(house), NUM-C(2)
3.        One hundred houses
OBJ(house), NUM-U(100)
4.        Alan’s house
OBJ(house), QUAL(POSS(A, house))
5.        The house where Alan, Burt, and Charles live
OBJ(house),
INF: QUAL(POSS((A,B,C),house))
6.        The house on the mountain
OBJ(house), LOC-T(ON mountain)
7.        Alan’s house on the mountain
OBJ(house), LOC-T(ON mountain), QUAL(POSS(A))
8.        Two houses on the mountain
OBJ(house), NUM-C(2), LOC-T(ON mountain)
9.        One hundred houses on the mountain
OBJ(house), NUM-U(100),LOC-T(ON mountain)
10.     A house with 100 windows
OBJ(house)
OBJ(window), NUM-U(100)
11.     A house has 50 windows at the 1st floor and 50 windows at the 2nd floor
OBJ(house)
OBJ(window1), SET(floor(1), NUM-U(50))
OBJ(window2), SET(floor(2), NUM-U(50))
12.     A house has 50 windows at the 1st floor and 50 windows at the 2nd floor. The 1st floor belongs to Alan;  Burt and Charles live at the 2nd floor
OBJ(house)
OBJ(window1),SET(floor(1),QUAL(POSS(A),NUM-U (50) )
OBJ(window2,SET(floor(2),QUAL(POSS(B,C),NUM-U (50))
13.     A house near the mountain
OBJ(house), LOC-M(NEAR mountain)
14.     In Alan’s garden there are 50 trees. Burt has more trees than Alan
SET(garden1, QUAL(POSS(A)), SET(trees1, NUM-U(50))
[INF: SET(garden2, QUAL(POSS(B), SET(trees2,NUM-X)]
COMP (MORE-THAN, NUM-U(trees1), NUM-X(trees2))
15.     Alan’s house is in Park Street between the town council and the chemist
OBJ(house), QUAL(POSS(A), QUAL(NAME “Park Street”, LOC-T(BETW t.c., chemist))
16.     There are red books and green books
SET(books1, QUAL(COL red))
SET(books2, QUAL(COL green))
17.     There are red books and green books. There are 34 red books and 85 books in all
SET(books1, QUAL(COL red, NUM-U(34)))
SET(books2, QUAL(COL green, NUM-X)
SET(all-books, NUM-U(85)))
18.     Alan has 34 books. He read 12 of them, now has 22 books to read
SET(all-books, QUAL(POSS(A), NUM-U(34)))
SET(read-books,QUAL(POSS(A), NUM(12) )
SET(to-be-read-books, QUAL(POSS(A),NUM(22))
19.     Burt had 15 books. He bought 8 more, now he has 23 books
TIME1, SET(prev-books, QUAL(POSS(B), NUM(15))
TIME2, SET(new-books, QUAL(POSS(B), NUM(8))
TIME3, SET(now-books, QUAL(POSS(B), NUM(23))
* Sentences have been adapted from Italian.
Legend
A, B, C
persons
NUM-U
number (uncountable)
COL
color
NUM-X
number (unknown)
COMP
comparison
OBJ
depictable object
INF
inference
POSS
possession
LOC-M
location (metric)
QUAL
qualification
LOC-T
location (topological)
SET
set
NUM-C
number (countable)
TIME
time sequence

Participants

40 subjects participated in the experiment. All of them were volunteers, students at the University of Genoa.

Procedure

Participants were given a 19-page booklet; each page contained a verbal description of a situation printed on top of an empty box. Their task was “to represent without words” each situation. Drawings, single letters, numbers, and symbols were allowed; only a monochrome (black) pen was available. Subjects were also told that the representation should have been “clear enough in order that another person could reconstruct the sentence from the picture only”. Sentences were presented in booklet pages in the same order as in table 2, by increasing complexity.

Results

Only 24 out of the original 40 booklets fully conformed to the requirements; the remaining 16 were discarded because either they were not completed or instructions were not consistently followed. A total of 456 (24 subjects X 19 situations) representations were thus obtained.
3. Descriptive analysis
a) Pictorial elements
Our aim at this stage was to devise a method for a descriptive analysis of how situation descriptions were empirically translated from a text format into pictures. In order to assess how situations were generally represented, the following pictorial elements were considered: type of picture, used symbols, operators, and connectors.

Type of picture
Type of picture

number

percentage
picture
403
88,4
diagram
13
2,8
picture & diagram (mixed)
8
1,8
set
4
0,9
no picture (only symbols)
28
6,1
TOTAL
456
100,0
The vast majority of representations were pictures. Diagrams were used only in a few cases for representing situations from 16 to 19. The set-theoretical notation (e.g. Wenn diagrams) was used very rarely.
In the vast majority of cases (92,8 %), pictorial representations were simple pictures. The remaining pictures can be analyzed more in detail, showing some interesting functions (strategies for expressing difficult aspects):
–          numeric correspondence (1,2%), when subjects attempted to reach a numeric correspondence even in cases where the concept could have been expressed more easily by a number (e.g. 100 windows represented as a 10 by 10 matrix);
–          partial numeric correspondence (0,9%)similar to the previous case, but where the numeric correspondence was partial and the completion was suggested by arithmetic operation (e.g. 50 windows represented as 5 windows with the symbol “X 10”);
–          symbolic function (1,9%) (e.g. a little square similar to a window picture, but used as a symbol associated with a number);
–          symbolic sequence (1,6%) (similar to the previous case, but using a sequence: e.g. “a house with 100 windows” represented using a picture of a house, the number 100, and the picture of a window);
–          iconic (1,6%) (e.g. a fire picture for representing the red color, a tree for green)

Symbols
Two kind of symbols were considered: numbers and letters. Numbers could have a cardinal or ordinal function, or they could act as simple labels. Letters generally had a label function, acting as captions for identifying something; in two cases they had the function of a legend, because the symbol was repeated outside the working area and associated with an explanation; in one case a string was used in a metonimic sense, when “Marx” was used inside a book picture in order to denote “red books”.

Operators
Operators are special symbols that denote an operation to be performed. The operation could be intended in a wide sense as any action to be done upon a representation in order to transform it, or in a restricted sense as a mathematical operation. Examples of the first kind of operators are the symbol “x” to denotate “repeat this a number of times”, or the symbols “…” to denotate “take more than this”.
Types of operators

number

percentage
general
74
69,2
mathematical
33
30,8
TOTAL
107
100,0
 

Connectors
Connectors are symbols that link pictures or other symbols each other. Arrows were by far the most frequent kind of connectors. In the majority of cases (26), they indicated what a symbol referred to. In a few cases, arrows indicated where to put a symbol that did not fit into a picture, direction, development (e.g. from 1 window to 100 windows). In five cases, they had no recognisable function and were considered pleonastic.
Other connectors were lines and enclosures (no set-theoretical space partitions, with a function of conceptual focus or reinforcement: e.g. to circumscribe a part of the picture in order to highlight it or to put it in correspondence with a symbol)
 

Types of connectors

number

percentage

arrows
35
68,6
arrow function
reference
26
indication
2
direction
1
development
1
pleonastic
5
enclosures
12
23,5
lines
3
5,9
other
1
2,0
TOTAL
51
100,0
b) Connection between pictorial elements and conceptual properties
In order to convey a complete information about situations, objects and relevant properties should have been represented.
We considered how logical properties were translated into pictorial data by using conceptual slots. To this aim, we developed a predicate-argument notational system specifically apt for coding elements with pictorial relevance (already described in Table 2).
To classify pictures, we adopted some formal techniques used for describing and implementing visual languages in human-computer interaction, with programming or modeling purposes (Marriott & Meyer, 1998).
The spatial arrangement of elements may be seen as the composition of a “visual sentence” and a sort of grammar may be devised for describing the different kind of relations between elements. By analogy with verbal languages, syntactic, semantic and pragmatic perspectives may be taken in this description (Gurr, 1999).
A visual sentence is made of visual symbols. A visual symbol is defined by three components (P,S,L), where P are the physical attributes (such as form, color, position), S are syntactic attributes.

Conceptual slots

Possible representations

OBJ(x)
PICT (x)
DIAPICT(x)
NUM-C (x)
ITER-PICT(x)
ITER-DIAPICT(x)
NUM(x)
NUM-U (x)
ITER-PICT(x)
ITER-DIAPICT(x)
NUM(x)
TIMES(x)
DOTS
QUAL (POSS(owner,possession))
owner: PICT (person/owner)
owner: LET
possession:
possession: arrow
LOC-T (ON x)
LOC-T (BETW x,y)
PICT (ON x)
PICT (BETW x,y)
SET (x)
SET-DEL:
  • DIAPICT(x)
  • ITER-PICT (x)
  • ENCLOSURE
  • SPACEPART
PICT= picture (rich)
DIAPICT = diagrammatic picture (only simplified shape)
NUM = number
LET=letter
ITER-PICT = iterated identical pictures (/C corresponding, /NC non corresponding number)
ITER-DIAPICT =  iterated diagrammatic pictures (/C corresponding, /NC non corresponding number)
SET-DEL = set delimiter
ENCLOSURE = closed line (circle or other polygon): Venn diagrams are a particular case
SPACEPART = space partition (the whole space is divided into different regions by using a line or other means, e.g. a fence)
IRR (x) = irrelevant objects (e.g. a tree when only a house should be represented)
conventional operators
TIMES  (X) = repeat this
GRTHAN/LESSTHAN (>, <)
Table 4 – Actual representation of sentences in our data
Sentences
Conceptual slots
Representation
N
1.       A house
OBJ(house)
PICT(house)
PICT(house), IRR(x)
DIAPICT(house)
4
2
2
2.       Two houses
OBJ(house), NUM-C(2)
PICT (house), ITER-PICT(house)
DIAPICT(house), ITER-DIAPICT(house)
PICT (house), IRR(x), ITER-PICT(house)
5
2
1
3.       One hundred houses
OBJ(house), NUM-U(100)
DIAPICT(house), ITER-DIAPICT(house)
PICT (house), OPER(&“x”), &NUM(100)
DIAPICT(house), ITER-DIAPICT(house), DOTS
PICT (house), &NUM(100)
PICT (house), &NUM(100), DOTS
DIAPICT(house), &NUM(100)
PICT(house), ITER-PICT(house),PERSP, DOTS
2
1
1
1
1
1
1
4.       Alan’s house
OBJ(house), QUAL(POSS(A, house))
PICT(house), PICT(person),&SYMB-ON(person)
PICT(house), PICT(person),&SYMB-ON(house)
PICT(house), PICT(person), (person,house)
PICT(house), PICT(person), IRR(x)
PICT(house), PICT(person), ARROW(person to house)
DIAPICT(house), &LET(A)
ERR
1
1
1
1
1
1
2
5.       The house where Alan, Burt, and Charles live
OBJ(house),
INF: QUAL(POSS((A,B,C),house))
PICT(house), ITER-PICT(person),&LET, IRR(x)
PICT(house), ITER-PICT(person), ARROW(person to house)
PICT(house), ITER-PICT(person)
DIAPICT(house), &LET(A,B,C)
PICT(house), PICT(person),&SYMB-ON(person)
PICT(house), &SYMB-ON(house)
ERR
1
1
1
1
1
1
2
6.       The house on the mountain
OBJ(house), LOC-T(ON mountain)
PICT(house),PICT house,mountain
PICT(house),PICT house,mountain, IRR(x)
5
3
7.       Alan’s house on the mountain
OBJ(house), LOC-T(ON mountain), QUAL(POSS(A))
PICT(house),PICT house,mountain, PICT(person)
PICT(house),PICT house,mountain, PICT(person), ARROW(person to house)
PICT(house), PICT house,mountain,PICT(person),&LET(A)
PICT(house), PICT house,mountain,PICT(person), (person,house)
PICT(house),PICT house,mountain, IRR(x)
PICT(house),PICT house,mountain
PICT(house),PICT house,mountain, IRR(x)
1
1
1
2
1
1
1
Stage 2  
 
Method
As explained in the introduction, the step concerning the assessment of picture effectiveness was only sketched in the present study, since effectiveness is not only a matter of communication and could be measured in different ways. Here we operationally defined it as making other people able to understand what situation one picture is referring to.
In the second stage, other participants acted as judges; their task was to match pictures drawn by the subjects in stage 1 (in random order) with the sentences from Table 2. The aim of this procedure was to evaluate the appropriateness of the pictures to their intended purpose, i.e. communicating information to other people. Participants in stage 2 were not taught explicit procedures, in order to encourage them to develop their own implicit procedures as they encountered more and more complex and abstract situations.
 
Participants 
8 subjects, students at the University of Genoa, participated in the experiment as volunteers.
Materials and procedure 
8 sets of sketches produced by 8 participants in stage 1 were randomly selected for this stage. Each set was composed of all 19 pictures, so that in each set all 19 sentences had been depicted and there was one correct picture-sentence match.  Each new subject here acted as a “judge”. Their task was to match each picture with the corresponding sentence. Pictures were presented in random order. Sentences were listed in alphabetical order and, in order to avoid that numbers acted as a possible cue for correct picture-sentence matching, all numbers were replaced by x,y,z,.. letters. For example, sentence number 11 became: “A house has x windows at the y-th floor and z windows at the t-th floor”.
 
Results and discussion
Table 5 shows the percentage of correct matchings picture-sentence out of the 152 total available sketches. Nine sketches were never recognized. Seven of them (corresponding in two cases to sentence 7, in two cases to sentence 12, and in other cases to sentences 3, 5, 11) did not have the features needed for representing corresponding sentences.
Table 5 – Scores from the most difficult to the easiest sentence
sentence
   Sum
 Mean
Mean
st.dev.
17. There are red books and green books. There are 34 red books and 85 books in all
25
0,39
0,49
11. A house has 50 windows at the 1st floor and 50 windows at the 2nd floor
28
0,44
0,34
12. A house has 50 windows at the 1st floor and 50 windows at the 2nd floor. The 1st floor belongs to Alan; Burt and Charles live at the 2nd floor
31
0,48
0,38
7. Alan’s house on the mountain
33
0,52
0,36
18. Alan has 34 books. He read 12 of them, now has 22 books to read
34
0,53
0,32
4. Alan’s house
41
0,64
0,39
19. Burt had 15 books. He bought 8 more, now he has 23 books
41
0,64
0,40
5. The house where Alan, Burt, and Charles live
42
0,66
0,24
10. A house with 100 windows
44
0,69
0,42
3. One hundred houses
46
0,72
0,23
8. Two houses on the mountain
46
0,72
0,31
9. One hundred houses on the mountain
47
0,73
0,40
6. The house on the mountain
47
0,73
0,44
1. A house
53
0,83
0,36
15. Alan’s house is in Park Street between the town council and the chemist
54
0,84
0,22
16. There are red books and green books
55
0,86
0,27
2. Two houses
61
0,95
0,13
13. A house near the mountain
61
0,95
0,13
14. In Alan’s garden there are 50 trees. Burt has more trees than Alan
62
0,97
0,09
References
Antonietti A. (1991) Why does mental visualization facilitate problem-solving? In Logie R.H., Denis M. Mental images in human cognition Amsterdam: North-Holland, pp.211-227.
Carney R.N., Levin J.R. (2002) Pictorial illustrations still improve students’ learning from text, Educational Psychology Review,14, 1, 5-26.
Chandler, P., Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293-332.
Gagatsis A., Elia I. (2004) The effects of different modes of representation on mathematical problem solving.Proceedings of the 28th Conference of the International Group for the Psychology of Mathematics Education, vol 2, pp 447–454.
Greco A. (1994). Rappresentazione iconica e procedura nella soluzione di problemi aritmetici: un’indagine empirica. [Iconic representation and procedure in arithmetic problem solving: an empirical study] (in Italian) Ricerche di Psicologia, 18, 3, 7-36.
Gurr C.A. (1999) Effective diagrammatic communication: Syntactic, semantic and pragmatic issues. Journal of Visual Languages and Computing, 10(4).
Landau B., Jackendoff R. (1993) “What” and “where” in spatial language and spatial cognition. Behavioral and Brain Sciences, 16, 217-265.
Larkin J.H., Simon H.A. (1987) Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 10, 65-100.
Lesh, R., Behr, M., Post, T. (1987). Representations and translations among representations in mathematics learning and problem solving. In C. Janvier (Ed.), Problems of representation in the teaching and learning of mathematics, (pp. 33-40). Hillsdale, N.J.:Lawrence Erlbaum Associates.
Lindsay R.K. (1988) Images and inference. Cognition, 29, 229-250.
Marriott K., Meyer B.(eds) (1998) Visual language theory, Berlin, Springer-Verlag.
Mayer R.E.  (2003) The promise of multimedia learning: using the same instructional design methods across different media, Learning and Instruction 13, 125–139.
Robinson D.H. & Kievra K.A. (1995) Visual argument: graphic organizers are superior to outlines in improving learning from text, Joumal of Educational Psychology, 87, 3, 455-467.
Robinson D.H. (1998) Graphic organizers as aids to text learning, Reading Research and Instruction, 37, 2, 85-105.
Schnotz W. & Bannert M. (2003) Construction and interference in learning from multiple representation, Learning and Instruction, 13 , 141–156.
Sierpinska, A. (1992). On understanding the notion of function. In E. Dubinsky, & G. Harel (Ed.), The concept of function: Aspects of epistemology and pedagogy (pp. 25-58). USA: Mathematical Association of America.
Sweller,J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285.
Sweller,J., van Merriënboer,J.J.G., & Paas,F.G.W.C. (1998). Cognitive architecture and instructional design.Educational Psychology Review, 10, 251-296.
Ullman S. (1985) Visual routines. In Pinker S. (ed.) Visual cognition. Cambridge, Mass: MIT Press, pp.97-159.
Winn W.D. (1991) A theoretical framework for research on learning from graphics. International Journal of Educational Research, 14, 553-564.
Appendix: Some picture examples
marmaris excursions
bayan escort gursu bayan escort
escort bayan nilufer escort