Nonstationary Bandits with Habituation and Recovery Dynamics

Yonatan Mintz; Anil Aswani; Philip Kaminsky; Elena Flowers; Yoshimi Fukuoka

doi:10.1287/opre.2019.1918

Nonstationary Bandits with Habituation and Recovery Dynamics

Yonatan Mintz¹; Anil Aswani²; Philip Kaminsky²; Elena Flowers³; Yoshimi Fukuoka⁴

¹ School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332 · ² Department of Industrial Engineering and Operations Research, University of California, Berkeley, Berkeley, California 94720 · ³ Department of Physiological Nursing, School of Nursing, University of California, San Francisco, San Francisco, California 94143; · ⁴ Department of Physiological Nursing & Institute for Health & Aging, School of Nursing, University of California, San Francisco, San Francisco, California 94143

Operations Research 2020

In many sequential decision-making settings where there is uncertainty about the reward of each action, frequent selection of specific actions may reduce expected reward while choosing less frequently selected actions could lead to an increase. These effects are commonly observed in settings ranging from personalized healthcare interventions and targeted online advertising. To address this problem, the authors propose a new class of models called ROGUE (reducing or gaining unknown efficacy) multiarmed bandits. In the paper, the authors present a maximum likelihood approach to estimate the parameters of these models, and we show that these estimates can be used to construct upper confidence bound algorithms and epsilon-greedy algorithms for optimizing these models with strong theoretical guarantees. The authors conclude with a simulation study to show that these algorithms perform better than current nonstationary bandit algorithms in terms of both cumulative regret and average reward.

DOI: 10.1287/opre.2019.1918
Volume: 68 (5)
Pages: 1493-1516
Language: en
Export: BibTeX
Sources: crossref openalex