Generalized fitted Q-iteration with clustered data

This paper focuses on reinforcement learning (RL) with clustered data, which is commonly encountered in healthcare applications. We propose a generalized fitted Q‐iteration (FQI) algorithm that incorporates generalized estimating equations into policy learning to handle the intra‐cluster correlations. Theoretically, we demonstrate (i) the optimalities of our Q‐function and policy estimators when the correlation structure is correctly specified and (ii) their consistencies when the structure is mis‐specified. Empirically, through simulations and analyses of a mobile health dataset, we find the proposed generalized FQI achieves, on average, a half reduction in regret compared to the standard FQI.