Reciprocal Human-Machine Learning: A Theory and an Instantiation for the Case of Message Classification

Dov Te’eni; Inbal Yahav; Alexely Zagalsky; David Schwartz; Gahl Silverman; Daniel Cohen; Yossi Mann; Dafna Lewinsky

doi:10.1287/mnsc.2022.03518

Reciprocal Human-Machine Learning: A Theory and an Instantiation for the Case of Message Classification

Dov Te’eni¹; Inbal Yahav¹; Alexely Zagalsky¹; David Schwartz²; Gahl Silverman¹; Daniel Cohen³; Yossi Mann⁴; Dafna Lewinsky⁴

¹ Coller School of Management, Tel Aviv University, Tel Aviv 6997801, Israel; · ² School of Business Administration, Bar Ilan University, Ramat Gan 5290002, Israel · ³ Department of Management, Bar-Ilan University, Ramat Gan 5290002, Israel; · ⁴ Department of Middle Eastern Studies, Bar Ilan University, Ramat Gan 5290002, Israel

Management Science 2026

There is growing agreement among researchers and developers that in certain machine-learning (ML) tasks, it may be advantageous to keep a “human in the loop” rather than rely on fully autonomous systems. Continual human involvement can mitigate machine bias and performance deterioration while enabling humans to continue learning from insights derived by ML. Yet a microlevel theory that effectively facilitates joint and continual learning in both humans and machines is still lacking. To address this need, we adopt a design science approach and build on theories of human reciprocal learning to develop an abstract configuration for reciprocal human-ML (RHML) in the context of text message classification. This configuration supports learning cycles between humans and machines who repeatedly exchange feedback regarding a classification task and adjust their knowledge representations accordingly. Our configuration is instantiated in Fusion, a novel technology artifact. Fusion is developed iteratively in two case studies of cybersecurity forums (drug trafficking and hacker attacks), in which domain experts and ML models jointly learn to classify textual messages. In the final stage, we conducted two experiments of the RHML configuration to gauge both human and machine learning processes over eight learning cycles. Generalizing our insights, we provide formal design principles for the development of systems to support RHML. This paper was accepted by D. J. Wu, Special Issue on the Human-Algorithm Connection. Funding: This work was supported by the Israel’s Ministry of Defence [Grant R4441197567] and the Israel’s Ministry of Science and Technology [Grant 207076]. Supplemental Material: The data files are available at https://doi.org/10.1287/mnsc.2022.03518 .

DOI: 10.1287/mnsc.2022.03518
Volume: 72 (1)
Pages: 167-192
Language: en
Export: BibTeX
Sources: crossref