Ahmet Zahid Balcioglu: Improving decision-making by learning surrogate outcomes and instance representations

Online decision-making is a key component of many high-stakes scenarios, as it provides direct evidence for finding the optimal decision. In healthcare, for instance, personalizing treatments through sequential decision-making and quantifying the effect of novel treatments through randomized controlled trials (RCTs) are two important examples. In practice, however, learning online can be slow: online algorithms are sample-hungry, making it impractical to train them from scratch, and in RCTs the primary outcome may take a long time to observe. One option is to make use of offline (observational) data, either to improve sample efficiency or to learn a surrogate that replaces the primary outcome. Using observational data requires care, as the desired effect is generally not causally identifiable from it, which makes it hard to transfer knowledge from the observational setting to the online one. Here, we present two papers that study how novel learning algorithms can draw on observational data in a causally principled way. The first work focuses on the personalized treatment example and proposes a contrastive learning objective to learn an identifiable latent representation of patients, which is then used for downstream reward modeling and for generalizing to new patients. For the latter, we propose hybrid algorithms that adapt the learned model with online evidence. The hybrid algorithms show substantial improvement over online and offline learning baselines when identifying conditions are satisfied. The second work focuses on learning surrogate outcomes for RCTs in which the treatment and the primary outcome are not jointly observed. We propose a novel objective for learning a surrogate whose conditional average treatment effect most closely matches that of the primary outcome, and two methods for optimizing it. One of these, our surrogate sampling method, achieves the best results both in synthetic experiments with known effects and in data from a real-world experiment.