AMS 2 - Optimizing the Use of Discrepant Results When Using, Interpreting, and Integrating Assessment Data

Researchers often administer multiple instruments designed to measure the same domain. In research, scores taken from multiple instruments may be used to assess their psychometric properties (e.g., convergent or discriminant, criterion-related validity) or to address substantive research aims (e.g., identifying risk factors for mental health concerns, predicting treatment outcomes, identifying evidence-based therapies). In applied settings, these same instruments may be used to make high-stakes decisions about individuals (e.g., diagnosing, treatment planning, response monitoring). Several decades of research indicate that in both applied and research settings, two instruments designed to measure the same domain commonly produce discrepant results when assessing individuals (De Los Reyes, 2024). By “discrepant results” we mean distinct estimates about the same domain in terms of its form, function, level, and relations to other domains. Yet, for nearly 70 years, researchers have made sense of these discrepant results using paradigms that assume that results that converge contain all the valid data (e.g., Converging Operations, multi-trait, multi-method matrix; Campbell & Fiske, 1959; Garner et al., 1956). This assumption translates to use of data integration strategies that misclassify discrepant results as measurement error when these results, in fact, contain valid data (Makol et al., 2025). Grounded in new training and coursework resources (https://bit.ly/3CySQPx), this workshop provides attendees with a falsifiable approach to “diagnosing” their discrepant results that addresses five competencies: (a) theory, (b) epistemology, (c) selection of data sources, (d) construction of analytic models, and (e) design of studies. A series of collaborative, experiential activities will allow attendees to apply these competencies to their work. The workshop will also provide attendees with syntax for the analytic procedures illustrated in the workshop, and examples of publicly available datasets to implement them. In turn, we expect these resources will allow attendees to inform the production of high-quality research and the construction of training modules, coursework, and other forms of pedagogy at their home institutions.

Recommended Reading 1: De Los Reyes, A., Wang, M., Lerner, M.D., Makol, B.A., Fitzpatrick, O., & Weisz, J.R. (2023). The Operations Triad Model and youth mental health assessments: Catalyzing a paradigm shift in measurement validation. Journal of Clinical Child and Adolescent Psychology, 52(1), 19-54. https://doi.org/10.1080/15374416.2022.2111684

Recommended Reading 2: McLeod, B. D., Porter, N., Hogue, A., Becker-Haimes, E. M., & Jensen-Doss, A. (2023). What is the status of multi-informant treatment fidelity research?. Journal of Clinical Child and Adolescent Psychology, 52(1), 74-94. https://doi.org/10.1080/15374416.2022.2151713

Recommended Reading 3: Rescorla, L.A., Ivanova, M.Y., Achenbach, T.M., Almeida, V., Anafarta-Sendag, M., Bite, I., Caldas, J.C., Capps, J.W., Chen, Y.C., Colombo, P., da Silva Oliveira, M., Dobrean, A., Erol, N., Frigerio, A., Funabiki, Y., Gedutienė, R., Guðmundsson, H.S., Heo, M.Q., Kim, Y.A., Zasępa, E. (2022). Older adult psychopathology: International comparisons of self-reports, collateral reports, and cross-informant agreement. International Psychogeriatrics, 34(5), 467-478. https://doi.org/10.1017/S1041610220001532

Recommended Reading 4: von der Embse, N., & De Los Reyes, A. (2024). Advancing equity in access to school mental health through multiple informant decision-making. Journal of School Psychology, 104, 101310. https://doi.org/10.1016/j.jsp.2024.101310

Recommended Reading 5: Makol, B.A., Youngstrom, E.A., Racz, S.J., Qasmieh, N., Glenn, L.E., & De Los Reyes, A. (2020). Integrating multiple informants’ reports: How conceptual and measurement models may address long-standing problems in clinical decision-making. Clinical Psychological Science, 8(6), 953-970. https://doi.org/10.1177/2167702620924439

Outline:
• Hour 1: Principles of Measurement and Methodology
o Workshop Overview and Brief History about Measuring Mental Health
o Data Conditions: Describing Discrepant Results in Mental Health
o Theory, Epistemology, and Assumptions: Relevance to Discrepant Results
o Think-Pair-Share Activities Throughout the Hour
• Hour 2: How Traditional Practices Conflict with Mental Health Data
o Traditional Strategies for Analyzing and Integrating Mental Health Data
o Discrepant Results Aren’t Created Equal: The Operations Triad Model
o Q and A about Principles and Practices
o Think-Pair-Share Activities Throughout the Hour
• Hour 3: A Paradigm for Capitalizing on Discrepant Results in Mental Health
o Interpreting Your Discrepant

Results: The CONTEXT Validation Paradigm
o CONTEXT Part I: Selecting Your Data Sources
o CONTEXT Part II: Selecting Your Analytic Strategies
o CONTEXT Part III: Designing Your Studies and Selecting Your Validity Criteria
o Think-Pair-Share Activities Throughout the Hour
• Hour 4: Group-Based Activities and Extended Q and A
o Group Activity: Implementing Strategies for Integrating Data
o Extended Q and A and Informal Discussion about Discrepant Results