What Do You See in A Face?

The test was to measure how many questions were answered before the visitor left the system, so it was important that the questionnaire should appear credible, and that it was easy to leave the system at any moment.

Initially I tested this questionnaire on seven people in three countries, working in 3 operating systems, aiming to make the questionnaire long enough so as to be a little demanding to complete but not so long that it would lose credibility.

Through trial and error I settled on 72 faces, half being of females, and of known personalities in diverse fields or cultural backgrounds.

Because I relied on six types of user audiences in the e-mail lists for my participants I chose 'faces' known in the film world, art world, media / computer worlds, education, literature, music, New Zealand and Bahai worlds.

I invited feedback on the questionnaire in the invitation I emailed out.

I recorded the length of time (from question 1 until they clicked 'ok' after 'stop'), the final question they reached, the IP address, the type of questionnaire, gender, and age range.

I received responses from ip addresses from around the world (below I give a breakdown but didn't refer to this in my thesis).

For my graduation thesis I used results from between July 7th and August 22nd [not counting the participants I'd asked to check the questionnaire for its usability]. So I used 347 responses: 85, 94, 88 and 80 for types 1 to 4 respectively.

I received feedback related to the content of the questionnaire from 18 respondents, many of whom I knew personally. Most made responses in relation to the content of the questionnaire or in relation to the results they saw in relation to their own answers.
A few made the comment that they found it difficult to make a choice, not knowing if they should respond to the face or the captions.

To see if this was a significant factor, I added types 5 and 6 on July 19th.
These had photographs (type 5) or drawings (type 6) with no captions, and were intended to see whether the mere existence of captions was enough to change participant's behaviour.
I stopped this on August 22nd and had 24 responses in both types.
These few results indicated a pattern similar to the other types with six stopping at question 9 or less, and five finishing at question 72 with the rest spread thinly in between. The number of responses was too small for the numerical analysis applied to types 1 to 4.

I also observed 8 individuals in the U.K. undertaking the questionnaire in early August and found that all read the captions before making a choice. I noted that some stopped because they had to do something else, and not because they wished to leave. I had told them that part of my test was to see when they would leave the system, so that they wouldn't feel they had to complete the questionnaire because of my presence.

However given that all of these individuals as well as any person who did the questionnaire were allotted to the types randomly, this shouldn't effect the results between the types of questionnaire.

For all the results, I also recorded the length of time each visitor spent answering the questions, but chose not to use this because the number of questions answered is a more valid proxi measure for engagement. The time on-line could be affected by various events happening around each participant.

Demographic spread of those who responded

Added here for your interest. These results not used in my thesis experiment.

The table shows a total of 371 responses, including types 5 + 6 [not counted in my final analysis]. Roughly a third were affiliated to universities and it seems that from my network, those in the "Aotearoa Digital Arts" email list, a New Zealand folk list, and Bahai circles responded or possibly passed on my questionnaire the most.
The respondents in the Netherlands were spread across the country indicating to me that Bahais most likely spread this onto their acquaintances. The city or province, business or organization was also indicated in all of these addresses.

Results

The responses for each questionnaire were ranked according to the number of questions answered.

The most striking difference between the questionnaires was that those with a sketch achieved greater engagement (measured by the final question answered) than those with a photograph.

I was surprised to discover in the questionnaire that sketches had such a significantly positive influence on the engagement, while the “socially charged” captions had a modest positive effect on median scores, and did not significantly affect the numbers who continued to the very end of the questionnaire.
And it appears that socially charged content may well be a factor in achieving greater sustained user engagement (explained below), although it is considerably less important than design features.

I broke the results into groups of nine (so questions 64-72 are the final nine questions) to make this more manageable and because I noted marked changes in responses from question 9 onwards not only in numbers but in distinctions between the types.
The table below uses 371 responses [347 for just Types 1 to 4.] from between July 7th and August 22nd 2008.

The results for 24 respondents who left the system without answering the first question were discarded, on the assumption that these were accidental participants who left because they did not mean to be there in the first place: their leaving would not relate to the type of questionnaire they had encountered.
For each questionnaire, the median of the responses was found: if for example there were 81 responses, the number of questions answered by the 40th ranking respondent was taken as the engagement score for that type of questionnaire. If there were 80 responses, the average of the questions answered by the 39th and 40th respondent would be the score. This gave the following results:

These differences are not significant, given that the number of questions answered is necessarily a whole number.
It is notable that the engagement for the photograph and sketch variants also do not differ, although the distribution of the most tenacious respondents (22 versus 29, described above) indicates that this is a large factor.

This suggested that uninterested respondents may click on a number questions before leaving, rather than just one, the number not relating to the content or design of the questionnaire. If we discard all the responses which stopped at or before question 9, the picture is quite different:

Here it appears that the engagement scores for the versions with sketches are higher than those with photographs, but also that the two scores for the versions with charged content are both higher than those for the same design but with neutral content.

I then recorded the number of responses for those stopping between question 1 and 15

There is a marked drop off in responses from Question 11 onwards, so to test the sensitivity of this analysis,
I did the same analysis as above but started with results for Question 16 onwards which gave the following results.

These results show a similar pattern as the Table showing the median for Questions 10-72.
It appears that socially charged content may well be a factor in achieving greater sustained user engagement, while neither design aspects nor captions had a detectable effect on initial engagement.


If we return to the most tenacious respondents, those who stopped at Question 64 or higher, where 22 had answered questionnaires using photographs, and 29 had answered questionnaires with sketches. The breakdown for socially charged versus neutral content for the same 51 respondents is, surprisingly, almost the same:

The higher median scores for socially charged captions are not reflected in this last grouping. Why not? One answer is that 51 respondents is a fairly small group to give results. This is the advantage of using median scores: they show the engagement of all the participants included rather than of the most engaged group.
But another answer is that socially charged content gives greater engagement in the middle range – ignoring the results of the first 9 questions – but for really sustained engagement, to the bitter end, use Art! The drawings were much more effective than the photographs (29 vs 22).

Potential Ramifications

The questionnaire showed there is some significance in relation to charged or meaningful content when it comes to (relatively) sustained engagement. This was what I was interested in: an engagement that goes beyond user interaction. I chose to test an aspect relating to content because my critique in the development of interactive media was not focused on issues of interactivity or even immersiveness per se, but on purpose, content and emotional effect. This is in part a reaction to the focus on technique that seems to be a popular trend among practitioners in digital arts worlds. It seems to be me almost an uncritical approach. It is our subjectivity that makes things interesting and sustains engagement.

My masters thesis, the context for this questionnaire, is here >> Engaging Media: issues of engagement in digital media art projects (26 page PDF, 930kb) (This opens in a new window)