Browsing by Subject "Exploration"

Human learning and memory relies on a broad network of neural substrates, and is sensitive to a range of environmental factors and behaviors. The present studies are designed to investigate the modulation of active exploration behaviors in humans. In the current work, we operationalize exploration in two ways: participants’ spatial navigation (using a computer mouse) in environments containing rewarding and informative stimuli, and participants’ eyegaze activity while viewing images on a computer screen. The study described in Study 1 investigates the relationship between spatial exploration and reward, using participants’ reported anxiety levels to predict between-subject variability in vigor and information-seeking. The study described in Study 2 investigates the relationship between cue-outcome predictive validity and eyegaze behavior during learning; additionally, we test the extent to which differing states of expectation drive differences in eyegaze behavior to novel images. The study described in Study 3 expands on the questions raised in Study 2: using functional imaging and eyetracking, we investigate the relationship between predictive validity, gaze, and the neural systems supporting active exploration. Taken together, the findings in the present study suggest that emerging certainty in reward outcomes, rather than uncertainty, drives exploration and associative learning for events and their outcomes as well as encoding into long-term memory.

As the reinforcement learning community has shifted its focus from heuristic methods to methods that have performance guarantees, PAC-optimal exploration algorithms have received significant attention. Unfortunately, the majority of current PAC-optimal exploration algorithms are inapplicable in realistic scenarios: 1) They scale poorly to domains of realistic size. 2) They are only applicable to discrete state-action spaces. 3) They assume that experience comes from a single, continuous trajectory. 4) They assume that value function updates are instantaneous. The goal of this work is to bridge the gap between theory and practice, by introducing an efficient and customizable PAC optimal exploration algorithm, that is able to explore in multiple, continuous or discrete state MDPs simultaneously. Our algorithm does not assume that value function updates can be completed instantaneously, and maintains PAC guarantees in realtime environments. Not only do we extend the applicability of PAC optimal exploration algorithms to new, realistic settings, but even when instant value function updates are possible, our bounds present a significant improvement over previous single MDP exploration bounds, and a drastic improvement over previous concurrent PAC bounds. We also present Bellman error MDPs, a new analysis methodology for online and offline reinforcement learning algorithms, and TCE, a new, fine grained metric for the cost of exploration.