The challenges of using digital trace data to measure online behaviors: lessons from a study combining surveys and metered data to investigate affective polarization

Abstract: This case study describes the data collection strategy of the TRI-POL project, which represents the first attempt to collect both longitudinal survey and digital trace data, from the same individuals, to understand whether and how the Internet and social media are related to affective polarization across Southern European and Latin American countries. The most innovative part of this project is that to measure the online behaviors of participants, we used information about participants’ visited URLs obtained through tracking technologies willingly installed on their browsing devices. This type of digital trace data is often called metered data. This case study describes in detail our experience designing the metered data collection strategy, the problems faced, and how we dealt with them. Specifically, we focus on two main challenges, namely, (1) how to properly operationalize concepts of interest into valid metered data measurements; and (2) how to assess and correct for the fact that some participants might not be tracked in all the devices, browsers, or apps that they use to go online. This research shows that, although using metered data in combination with surveys is a promising approach, collecting high-quality metered data are challenging and little expertise exists yet. Hence, we provide strategies and practical recommendations on how to assess and, when possible, mitigate validity, and undercoverage issues of metered data. Check open access preprint paper here