Seeing Without Knowing: Potential Bias in Mobile Research
Tuesday November 1, 2016
Mobile research – specifically, research by way of smartphone technology – has become a widely used and accepted design option for conducting qualitative and survey research. The advantages of the mobile mode are many, not the least of which are: the high incidence of smartphone ownership in the U.S. (more than 60% in 2015), the ubiquitous influence smartphones have on our lives, the dependence people have on their smartphones as their go-to channel for communicating and socializing, and the features of the smartphone that offer a variety of response formats (e.g., text, video, image) and location-specific (e.g., geo-targeting, geo-fencing) capabilities.
From a research design perspective, there are also several limitations to the mobile mode, including: the small screen of the smartphone (making the design of standard scale and matrix questionnaire items – as well as the user experience overall – problematic), the relatively short attention span of the respondent or participant precipitated by frequent interruptions, the potential for errors due to the touch screen technology, and connectivity issues.
Another important yet often overlooked concern with mobile research is the potential for bias associated with the smartphone response format and location features mentioned earlier. Researchers have been quick to embrace the ability to capture video and photographs as well as location information yet they have not universally exercised caution when integrating these features into their research designs. For example, a recent webinar in which a qualitative researcher presented the virtues of mobile qualitative research – esp., for documenting in-the-moment experiences – espoused the advantages of utilizing systems that allow the researcher to identify a participant’s location. Among these advantages, according to the presenter, is the ability to gain the exact location of someone’s home address during an in-home use test (IHUT) which then, with the help of Google Earth, enables the researcher to actually see the property and surrounding neighborhood. The presenter went on to state that these location images can and should be used with the intent of evaluating some aspect of this person’s life such as their socio-economic status.
The blatant bias this introduces into the research should be obvious. Where someone chooses to livemay say something about their household income, educational achievement, and even their “social circles”; however, it is certainly not true in all cases and, indeed, such appearances can be grossly deceiving. And, even if the researcher could ascertain some idea of the individual’s demographic or social group, what would be the point or use of this information? Only to deepen the bias by creating a story of someone’s lived experience based on unsubstantiated claims built on preconceived stereotypical assumptions?
A similar bias creeps into mobile qualitative research when participants are asked to submit their responses in the form of videos and/or photographs without also being asked for accompanying commentary or follow-up questions by the researcher. By simply submitting these images without explanation, the researcher comes to his/her own conclusions which then lead to bias and error in the data which ultimately downgrades the value of the final outcomes. If the researcher conducting an IHUT study on eating habits, for example, learns from the participant that she and her family eat a “healthy” diet but sees from a submitted photograph a refrigerator containing fruits and vegetables but also donuts, Coke, and processed cheese – what is the researcher to make of that? Are the participant’s eating habits really not that “healthy”? Are there additional healthier foods hidden from view in the refrigerator’s compartments or drawers? Does the participant’s definition of “healthy eating” include donuts, Coke, and processed cheese? Without examining the whys and whereforeswith the participant, the researcher is left to form a subjective understanding of the fridge contents and may create a false yet seemingly plausible story about the participant from the image.
Mobile research gives the researcher new and convenient ways to learn about the lives of the people who matter most in our research designs. And yet, researchers are cautioned to tread carefully or risk infecting their data with an insidious and potentially destructive bias that comes from conjecturing stories of people’s lives by relying on what researchers see rather than from what they know to be true.