Adolescents flexibly adapt action selection based on controllability inferences
- 1Department of Psychology, New York University, New York, New York 10003, USA
- 2Center for Neural Science, New York University, New York, New York 10003, USA
- Corresponding author: cate{at}nyu.edu
-
↵3 These authors contributed equally to this work.
Abstract
From early in life, we encounter both controllable environments, in which our actions can causally influence the reward outcomes we experience, and uncontrollable environments, in which they cannot. Environmental controllability is theoretically proposed to organize our behavior. In controllable contexts, we can learn to proactively select instrumental actions that bring about desired outcomes. In uncontrollable environments, Pavlovian learning enables hard-wired, reflexive reactions to anticipated, motivationally salient events, providing “default” behavioral responses. Previous studies characterizing the balance between Pavlovian and instrumental learning systems across development have yielded divergent findings, with some studies observing heightened expression of Pavlovian learning during adolescence and others observing a reduced influence of Pavlovian learning during this developmental stage. In this study, we aimed to investigate whether a theoretical model of controllability-dependent arbitration between learning systems might explain these seemingly divergent findings in the developmental literature, with the specific hypothesis that adolescents’ action selection might be particularly sensitive to environmental controllability. To test this hypothesis, 90 participants, aged 8–27, performed a probabilistic-learning task that enables estimation of Pavlovian influence on instrumental learning, across both controllable and uncontrollable conditions. We fit participants’ data with a reinforcement-learning model in which controllability inferences adaptively modulate the dominance of Pavlovian versus instrumental control. Relative to children and adults, adolescents exhibited greater flexibility in calibrating the expression of Pavlovian bias to the degree of environmental controllability. These findings suggest that sensitivity to environmental reward statistics that organize motivated behavior may be heightened during adolescence.
From a young age, the positive and negative consequences of our actions guide our behavior. Adaptive action selection reflects a dynamic balance between instrumental and Pavlovian evaluative systems that learn from rewards and punishments in different ways. Instrumental learning promotes the selection of actions that effectively lead to reward or avoid punishment. In contrast, the Pavlovian system learns the positive or negative values of stimuli (Pavlov 2010). These stimuli can then elicit reflexive, evolutionarily hard-wired behavioral responses that couple valence and action, with expectations of reward promoting active, approach behaviors and expectations of punishment inhibiting action (Williams and Williams 1969; Bolles 1970; Hershberger 1986; Gray and McNaughton 2003). Flexible arbitration between Pavlovian and instrumental behavioral control may be particularly important for navigating the environments that individuals encounter during adolescence—a period associated with greater exploration and increased autonomy (Spear 2000). Instrumental control can support the discovery of actions that yield rewarding outcomes across novel social and environmental contexts, whereas Pavlovian control may enable greater safety when exploring environments in which there is a potential threat (Kavaliers and Choleris 2001; Moscarello and Hartley 2017).
Importantly, Pavlovian and instrumental learning systems can cooperate or compete (O'Doherty 2016). Studies of interactive dynamics between Pavlovian and instrumental learning systems in adult humans and animals have yielded convergent findings across species. When Pavlovian reactions are aligned with action-outcome contingencies in the environment, instrumental actions are typically invigorated. For example, the presentation of a food-predictive cue typically causes animals to lever-press more vigorously for an instrumentally obtained food reward. However, default Pavlovian reactions that conflict with action-outcome contingencies can hinder instrumental learning. For example, across species, individuals exhibit difficulty learning to make active motor responses to avoid shock following threat-predictive cues (Estes 1943; Holland 1979; Talmi et al. 2008; Galatzer-Levy et al. 2014; Guitart-Masip et al. 2014; Hartley et al. 2014).
Work examining developmental changes in the expression of Pavlovian responses and their interaction with instrumental learning have yielded conflicting findings. Studies in humans suggest that Pavlovian interference with instrumental learning decreases from childhood to adolescence (Raab and Hartley 2020), stabilizes from adolescence to early adulthood (Moutoussis et al. 2018), and then increases again with aging into older adulthood (Betts et al. 2020). Rodent studies are somewhat consistent with these observations, demonstrating that compared to juveniles or adults, adolescent animals exhibit better learning of active instrumental responses to avoid shock delivery (Stavnes and Sprott 1975; Bauer 1978). However, multiple studies have also demonstrated that adolescents readily acquire Pavlovian conditioned threat responses and in fact, show heightened expression of these responses during extinction (McCallum et al. 2010; Kim et al. 2011; Pattwell et al. 2012), relative to both preadolescents and adults.
A parsimonious account for these seemingly conflicting findings might be that adolescents are particularly adept at calibrating their expression of Pavlovian responding to the degree of controllability of the learning environment. Theoretical proposals have suggested that an “optimal” learner should arbitrate between the use of Pavlovian and instrumental learning systems based on their assessment of environmental controllability (Moscarello and Hartley 2017; Dorfman and Gershman 2019). In high-control environments, instrumental learning can enable the discovery and exploitation of beneficial responses and should be prioritized (e.g., learning an action to avoid an anticipated shock). However, in uncontrollable environments, where actions have no causal influence on experienced events, the additional computational complexity involved in trying to learn action-outcome relations is unnecessary (Dorfman and Gershman 2019), and simpler Pavlovian reactions (e.g., freezing in a state of threat) can serve as “default” behavioral responses. Consistent with this theoretical account, empirical studies in adult humans and animals have found that Pavlovian responding is attenuated in controllable environments and increases in uncontrollable environments (Overmier and Seligman 1967; Maier and Seligman 1976; Baratta et al. 2007; Hartley et al. 2014; Dorfman and Gershman 2019; Csifcsák et al. 2020; Gershman et al. 2021). However, while controllability-dependent arbitration between learning systems has been observed in adults, to date, the developmental trajectory of this ability remains uncharacterized.
In this study, we examined how environmental controllability affects the balance between Pavlovian and instrumental learning across development. We hypothesized that adolescents might exhibit heightened sensitivity to environmental controllability. Such a hypothesis could account for the varied patterns of Pavlovian behavioral expression observed in prior studies, in which adolescents exhibited heightened expression of extinction-resistant Pavlovian responding in uncontrollable conditioning paradigms (McCallum et al. 2010; Kim et al. 2011; Pattwell et al. 2012), but reduced Pavlovian interference in controllable, instrumental learning paradigms (Stavnes and Sprott 1975; Bauer 1978). To test this hypothesis, we manipulated the degree of outcome controllability by adding an uncontrollable condition to a child-friendly probabilistic-learning task, in which valence and action were orthogonalized (Raab and Hartley 2020, adapted from Guitart-Masip et al. 2011), leveraging a computational model to quantify controllability-dependent arbitration between learning systems. We expected that participants across ages would show greater expression of Pavlovian bias in the uncontrollable versus controllable condition, and that relative to children and young adults, adolescents would show greater flexibility in calibrating their expression of Pavlovian bias to the controllability of the environment.
Results
Approach
Ninety participants, ages 8–27 yr (N = 90; mean age = 16.34 yr, standard deviation [SD] age = 5.52 yr, 45 females, 45 males) (see Materials and Methods; Supplemental Fig. S1) performed a child-friendly adaptation of a probabilistic Go/No-Go reward learning task in which valence and action were orthogonalized (Raab and Hartley 2020, adapted from Guitart-Masip et al. 2011). The goal of the task was to earn as many tickets as possible by choosing whether to “press” or “not to press” a virtual button in response to a stimulus (robot) (Fig. 1A). Each robot was either a “Ticket Giver” or “Ticket Taker.” Ticket Givers could either give one ticket or do nothing. Ticket Takers could either take one ticket or do nothing.
(A) Example trial sequence. On each trial, one of four different colored robots appeared on the screen (750 ms). Then a fixation cross was shown (250 ms), followed by the robot's “button” (1500 ms). For “Go” responses, the border of the button appeared bold for the remainder of the 1500 ms. When the button disappeared, an outcome appeared on the screen (1000 ms). Each trial was followed by a fixation intertrial interval (750 ms). (B) Each robot was associated with the potential to either win or lose a ticket. Greater Pavlovian bias is reflected in a heightened tendency to take action in anticipation of reward or withhold action in anticipation of punishment (bolded diagonal). (C) Reward contingencies across conditions. In the controllable condition, a correct action for a given robot resulted in the desirable outcome 80% of the time (a ticket for Ticket Givers and nothing for Ticket Takers), whereas in the uncontrollable condition, outcomes were not contingent upon participant's actions.
Valence and action were orthogonalized such that each of the four robots was associated with a distinct valence-action pairing, leading to four trial types (i.e., Go to Win, Go to Avoid Losing, No-Go to Win, and No-Go to Avoid Losing) (Fig. 1B). In the controllable condition, a correct action resulted in the better outcome 80% of the time (a ticket for Ticket Givers and nothing for Ticket Takers) and the worse outcome 20% of the time (nothing for Ticket Givers and the loss of a ticket for Ticket Takers), whereas incorrect actions led to the better and worse outcomes on 20% and 80% of trials, respectively. In the uncontrollable condition, four new colored robots were presented. Two were Ticket Givers and two Ticket Takers, but there was no longer a correct action. Instead, the better and worse outcomes each occurred in 50% of trials, regardless of which action was taken (Fig. 1C). As previous studies suggest that controllability inferences often generalize to subsequent learning environments (Moscarello and Hartley 2017), condition order was counterbalanced across participants. In each condition, each robot was encountered 45 times, for a total of 360 trials.
Behavioral results
In the controllable but not the uncontrollable condition, participants could learn to take action to win or avoid the loss of tickets. Thus, we first investigated whether the number of tickets won varied by condition and age. Participants earned more tickets in the controllable condition (t(174) = −11.2, P < 0.001) (Fig. 2) and the number of tickets won in the controllable condition increased nonlinearly with age (age-by-condition: t(174) = −2.58, P = 0.011; age-squared-by-condition: t(174) = 2.34, P = 0.021). To clarify the nature of this nonlinear age effect, we implemented a piecewise linear regression (Muggeo 2003), which identified a single change point in the relationship between the number of tickets won in the controllable condition and age at 16.5 yr, with the number of tickets won significantly increasing from age 8 to 16.5 (t(86) = 2.02, P < 0.05) but not from age 16.5 to 25 (t(86) = −0.88, P = 0.38).
Participants won more tickets in the controllable condition, and tickets won increased nonlinearly from childhood to adulthood. Each point represents the sum of tickets won in the task per participant and condition. The lines represent the nonlinear effect of age on tickets won per condition. The dashed line represents the expected score for random responding. Shaded areas depict 95% confidence intervals.
Next, we quantified age-related differences in the expression of Pavlovian bias. We computed Pavlovian performance bias scores separately for the controllable and uncontrollable condition by calculating the proportion of reward-invigorated actions for Ticket Givers (number of Go responses to Win cues/total number of Go responses) and punishment-suppressed actions for Ticket Takers (number of No-Go responses to Loss cues/total number of No-Go responses). Bias scores closer to 1 reflect a greater Pavlovian bias, whereas scores of 0.5 indicate an absence of bias. A linear regression model with age and condition as interacting predictors of Pavlovian performance bias revealed greater bias in the uncontrollable condition (F(1,88) = 6.75, P = 0.01). No other effects reached significance (all other P’s > 0.7). Including an additional age-squared term did not improve model fit.
Given our a priori hypothesis that sensitivity to environmental controllability would be greatest during adolescence, we split participants into three categorical age groups: children (8–12), adolescents (13–17), and adults (18–27), and tested whether Pavlovian performance bias in each group differed across task conditions. Pavlovian biases differed significantly between the controllable and uncontrollable condition only in adolescents (children: t(87) = −0.52, P = 0.6, adolescents: t(87) = −3.54, P < 0.001, adults: t(87) = −0.57, P = 0.57, critical α after Bonferroni correction = 0.017). When comparing the magnitude of difference between conditions across age groups, adolescents showed a greater difference across conditions compared to children (t(87) = 2.13, P = 0.036) and adults (t(87) = 2.1, P = 0.039), with no difference between children and adults (t(87) = 0.03, P = 0.97). However, these between-group comparisons did not exceed the significance threshold following Bonferroni correction (critical α = 0.017) (see Fig. 3).
Pavlovian performance bias was greater in the uncontrollable than in the controllable condition, particularly for adolescents. Each pair of points connected by a line represents the Pavlovian bias across the controllable (CON) and uncontrollable (UNCON) conditions for a given participant in that age group (children: 8–12, adolescents: 13–17, adults: 18–27). Thicker black lines depict mean Pavlovian bias scores for that age group. Error bars depict SEM.
Computational modeling
To understand the mechanisms underlying age-related variation in task performance, we fit participants’ choices with a computational model that formalizes both the process of inferring environmental controllability as well as using those controllability inferences to determine the extent to which state (Pavlovian) versus state-action (instrumental), the computational statistics that inform Pavlovian and instrumental responding, respectively, govern one's choices (Dorfman and Gershman 2019). The model yields a Pavlovian weight coefficient w, which governs the relative weighting of state (Pavlovian) versus state-action (instrumental) values and reflects dynamic changes in controllability inferences across the blocks of the task. The model has four free parameters: an initial learning rate, a single initial state and action value, and a free parameter governing the initial value of w at the start of the second block (w2), which can account for the potential carryover of Pavlovian bias levels from the first to the second block, and an inverse temperature. For details regarding model specification, model fitting procedures, model comparison, parameter recovery, and posterior predictive checks, see Materials and Methods, Supplemental Material, and Supplemental Figures S2 and S3. The primary measure of interest derived from the model is the Pavlovian weight coefficient w, which governs the relative weighting of state versus state-action values and reflects individual differences in the exploration of Go and No-Go responses and resulting dynamic controllability inferences across the blocks of the task.
To test our hypothesis regarding adolescents’ sensitivity to environmental controllability, we used a linear mixed-effects model that included age group, task condition, trial number, the order in which task conditions were experienced, and their interactions as predictors. We observed a significant effect of condition (F(1,84) = 23.64, P < 0.001) such that Pavlovian weights were higher in the uncontrollable relative to the controllable condition. We also observed a significant interaction between age group and condition (F(2,84) = 3.43, P < 0.05), which reflected a difference in Pavlovian weight values across task conditions in the adolescent age group that was not evident in children or adults (children: t(84) = −2.31, P = 0.024, adolescents: t(84) = −4.86, P < 0.001, adults: t(84) = −1.26, P = <0.21, critical α Bonferroni correction = 0.017). In addition, the order in which task conditions were experienced affected Pavlovian weight values. Weights were higher for participants who encountered the uncontrollable condition first than those who encountered the controllable condition first (F(1,84) = 17.24, P < 0.001). Moreover, we observed a condition-by-order interaction (F(1,84) = 20.78, P < 0.001), such that participants who first experienced the controllable environment showed little change in Pavlovian weight in the subsequent uncontrollable condition (t(84) = −0.21, P = 0.83), while those initially exposed to the uncontrollable condition significantly reduced their Pavlovian weights in the subsequent controllable condition (t(84) = −6.82, P < 0.001) (see Fig. 4). For a full description of the model's output and post hoc analyses, see Supplemental Tables S1 and S2.
Model-derived Pavlovian weight for each condition (controllable or uncontrollable) across blocks, plotted by age group (children: 8–12, adolescents: 13–17, adults: 18–27). Adolescents exhibit the greatest controllability-dependent adjustment of weights across conditions. Shaded areas depict 95% confidence intervals.
None of the model-derived parameter estimates (i.e., initial stimulus and action values, initial learning rate, and w2) change with age (all P’s > 0.1, where each best-fitting model only contained a linear age term; see Supplemental Fig. S4A–D). Collectively, this suggests that adolescent-specific outperformance in the task (and their corresponding flexibility in model-derived w values) reflects the interactive effects of these learning parameters on action sampling and inference.
Discussion
In this study, we examined whether calibration of the expression of Pavlovian and instrumental learning to the degree of environmental controllability changed with age. We formalized this process of calibration within a computational framework (Dorfman and Gershman 2019) in which controllability inferences directly modulate reliance on the state or state-action values that respectively inform Pavlovian or instrumental responses. Extending past work in adults (Dorfman and Gershman 2019; Csifcsák et al. 2020; Gershman et al. 2021), participants spanning middle childhood to early adulthood exhibited greater expression of Pavlovian bias in the uncontrollable, relative to the controllable, task environment. Moreover, we found evidence in support of our hypothesis that the ability to flexibly arbitrate between these learning processes is greatest in adolescence.
Our finding that adolescents exhibit heightened sensitivity to environmental controllability may reconcile apparently inconsistent findings from past studies. Previous investigations of aversive conditioning have observed that adolescents exhibit particularly persistent Pavlovian responses during extinction (McCallum et al. 2010; Kim et al. 2011; Pattwell et al. 2012). In such experiments, the relation between stimuli and aversive outcomes is fundamentally uncontrollable—they are predetermined by the experimenter and cannot be influenced by participants’ actions. An opposite pattern of developmental differences has been observed when the environment is controllable. In our previous study (Raab and Hartley 2020), mirroring the present study's controllable condition, adolescents exhibited the best performance, reflecting the reduced influence of Pavlovian bias on their instrumental learning. Consistent with this finding, adolescent rodents in active avoidance tasks have been shown to more readily learn to shuttle across a conditioning chamber to prevent a shock, whereas reflexive freezing hinders such learning in older and younger animals (Stavnes and Sprott 1975; Bauer 1978). Collectively, both our present results and these previous studies suggest that when learning in uncontrollable environments, adolescents display more robust Pavlovian reactive behavior than other age groups. However, when the environment is controllable, adolescents more effectively diminish their expression of Pavlovian bias, enabling better instrumental learning.
The observed nonlinear age differences in the sensitivity of action selection to environmental controllability diverge from past studies documenting age-linear improvements in inferences of environmental controllability (Raab et al. 2022), and in the influence of diverse forms of task structure knowledge on choice behavior (Decker et al. 2016; Potter et al. 2017; Cohen et al. 2020; Nussenbaum et al. 2020b; Smid et al. 2023). Differences in the manner in which task structure knowledge was acquired and used across these studies may underpin the divergent developmental patterns. In the current study, participants’ estimates of controllability could be derived from their direct experiences of rewards and punishments, without any need to explicitly represent beliefs about the structure of the task. In contrast, the studies observing linear age-related variation assessed whether mental models of task structure, which needed to be derived from either explicitly instructed rules or observed state transitions, modulated participants’ choices. This suggests that while the ability to use mental models of task structure may improve linearly from childhood to adulthood, adolescents may be particularly skilled at learning from their direct experiences with reward outcomes, and using this derived information to organize their behavior.
Prior beliefs about environmental controllability have been shown to generalize, promoting less reactive behavioral tendencies in subsequent learning contexts (Maier and Seligman 2016; Moscarello and Hartley 2017). In the present study, we observed a condition-by-order interaction effect consistent with such generalization—experiencing the controllable condition first prevented increases in Pavlovian weights in the subsequent uncontrollable condition. Whereas we observed these generalization effects across task blocks, experiments examining the influence of controllable or uncontrollable stressors have found evidence of generalization at longer timescales, with the controllability of initial learning environments modulating performance on novel tasks the following day (Boeke et al. 2017), up to a week later (Baratta et al. 2007; Hartley et al. 2014), and strikingly, even over longer developmental timescales. Controllable stress exposure during adolescence has been shown to yield less reactive behavioral phenotypes in adulthood (Kubala et al. 2012; Sanchís-Ollé et al. 2019). Collectively, one speculative interpretation of these findings is that adolescence might be a “sensitive” developmental period during which individuals derive global estimates of environmental controllability that alter their tendencies to express reactive versus proactive behaviors on developmental timescales. Such a process would be consistent with theoretical accounts of developmental specialization, which proposed that organisms sample their early environments to derive the most accurate estimates of behaviorally relevant statistics and adapt their behavior in a manner that is increasingly optimized to their idiosyncratic early environments into adulthood (Frankenhuis and Panchanathan 2011). In this manner, heightened sensitivity to reward controllability during adolescence, when sampled across diverse real-world environments, may generalize to influence behavior on longer timescales. While speculative, such an account could represent a developmental mechanism modulating risk for psychopathology, as numerous disorders associated with heightened reactive responding and diminished perceptions of control (e.g., addiction, affective disorders, PTSD, and OCD) (Wasserman et al. 1974; Poulos et al. 1981; Belin et al. 2009; Waters et al. 2009; Hammack et al. 2012; Cartoni et al. 2016; Huys et al. 2016; Apergis-Schoute et al. 2017; Mkrtchian et al. 2017; Cooper and Dunsmoor 2021) commonly emerge during adolescence (Maier and Seligman 1976; Lee et al. 2014; Pauls et al. 2014; Cousijn et al. 2018; Volkow and Boyle 2018).
Individuals tend to perceive control over events, even when they are uncontrollable (Langer 1975; Fontaine et al. 1993; Taylor and Brown 1994; Fein 1995; Fiscella and Franks 1997; Morgan and Tromborg 2007; see Na et al. 2023 for a review). The aforementioned effects of the order in which the task conditions were encountered are consistent with such a bias toward a perception of controllability. Following an initial controllable condition, Pavlovian weights remained low, indicating that prior control experiences are difficult to override. Conversely, when an uncontrollable condition was followed by a controllable one, there was a greater reduction in the Pavlovian weights, indicating that initial experiences of lack of control could readily be counteracted by subsequent experiences of control. Such a bias toward inferences of controllability could have positive effects: beliefs of uncontrollability can lead to reactive avoidance biases that inhibit exploration and lead to learning traps (Rich and Gureckis 2018), while a bias toward control can facilitate exploration and identification of affordances for action (Huys and Dayan 2009). Indeed, we tend to learn more from actions that are freely chosen (Cockburn et al. 2014; Palminteri et al. 2017; Katzman and Hartley 2020). The bias toward inferences of control may be facilitated by the affective consequences of perceived control: having control over choices is often preferred, even when it does not necessarily lead to better gains (Bown et al. 2003; Cockburn et al. 2014; Nussenbaum et al. 2023), and perceived controllability promotes positive emotions (Véronneau et al. 2005; Weinstein and Mermelstein 2007; Stolz et al. 2020). Thus, while a controllability bias does not enhance performance within our task, it may foster adaptive behavior in the diverse real-world environments that are increasingly encountered across adolescence (Saragosa-Harris et al. 2022).
This study sought to determine whether adolescents might exhibit the greatest flexibility in adapting their expression of Pavlovian bias to the controllability of the environment. Our findings supported this hypothesis when we considered adolescents as a categorical group, but not when age was treated as a continuous variable. Adolescence is a developmental stage characterized by profound environmental and biological transformations. These include shifts in social dynamics, exposure to novel environmental stressors, significant neurobiological restructuring, and surges in hormones such as testosterone and estrogen (Blakemore and Choudhury 2006; Burnett and Blakemore 2009; Somerville and Casey 2010; Schulz and Sisk 2016). Individual variation arising from these multifaceted changes may contribute to corresponding changes in behavioral phenotypes. As such maturational processes have significant individual variability in their timing (Mendle et al. 2010; Marceau et al. 2011), it is possible that numerical age may not be tightly correlated with the underlying causal mechanisms that inform controllability-dependent action selection across adolescence. Despite these complexities, our data suggest that adolescence is a developmental stage characterized by heightened sensitivity to environmental controllability. This sensitivity may be adaptive, facilitating the discovery of actions that are beneficial in the novel environments typically encountered during the transition to independence, and enabling long-term generalization of expectations for the environments one might encounter in adulthood. However, this heightened sensitivity may also confer vulnerability, as exposure to uncontrollable environments during this period may foster reactive behaviors that may prove maladaptive in future controllable environments.
Materials and Methods
Participants
Ninety individuals, ages 8–27 yr, from the New York City area, took part in the study. Two additional children were tested but excluded due to technical errors in the task. Our final sample comprised 30 children (8–12 yr old, mean = 10.48, SD = 1.56, n = 15 female), 30 adolescents (13–17 yr old, mean = 15.59, SD = 1.36, n = 15 female), and 30 adults (18–27 yr old, mean = 22.94, SD = 2.85, n = 15 female) (see Supplemental Fig. S1). Our target sample size of 90 was determined a priori based on recent studies that used computational modeling to investigate developmental changes in learning (Cohen et al. 2020; Nussenbaum et al. 2020a; Raab and Hartley 2020). All participants reported no color blindness, mood or anxiety disorders, learning disabilities, or current use of β-blockers or psychoactive medication. Forty percent of participants self-identified as Asian, 35.6% as Caucasian, 14.4% as more than one race, and 10% as Black. In addition, 16.7% of participants self-identified as Hispanic.
Participants were paid $15/h and were told that their performance determined their bonus payment. In reality, all participants received a $5 bonus. The study was conducted according to the procedures approved by the New York University Committee on Activities Involving Human Subjects. Adult participants and parents of minors provided written informed consent and minors provided assent before the study.
Procedure
Participants completed both controllable and uncontrollable conditions of the orthogonalized Go/No-Go task in counterbalanced order. Before the task, participants completed extensive, interactive instructions and practice during which they learned about the task's probabilistic reward structure and how to press or not press the buttons. After participants completed the first block of the task, they were instructed that they would play again with a new set of robots. They were not informed of the different reward probabilities across blocks.
Following the learning task, we tested participants’ explicit knowledge about the action and valence of each robot. Participants saw a given robot and were asked to indicate whether it was better to press the button or not for this robot and whether the robot was a “Ticket Giver” or “Ticket Taker.” Both the order for which the condition was probed first and the order in which the robots appeared were randomized. The task was coded using Cogent 2000, a MATLAB toolbox.
Model-free analysis methods
All analysis codes and anonymized data are publicly available online at https://osf.io/e49ua/. We used R version 4.1.0 and MATLAB R2021a for statistical analyses. All continuous variables (e.g., age) were z-scored before inclusion as predictors in any regression models. In all analyses, to test for potential quadratic effects of age (e.g., adolescent-specific effects), we assessed whether the addition of an age-squared term improved the model fit (Somerville et al. 2013; Raab and Hartley 2020). Age-squared was computed by squaring the z-scored age term. Mixed-effects regression models were conducted using the optimizer “bobyqa” with one million model iterations in the afex package version 1.3-0 (Singmann et al. 2016). Except where noted, models included the maximal random-effects structure (i.e., random intercepts, slopes, and their correlations across fixed effects for each subject) to minimize Type I error (Barr et al. 2013). If a model did not converge, we reduced the random-effects structure. For all linear models, the significance of fixed effects was determined by an ANOVA using the Kenward–Roger method to calculate degrees of freedom. Post hoc tests were implemented with emmeans package version 1.8.7. Critical α value for multiple comparisons was determined using the Bonferroni method.
Computational modeling
We fit participants’ choices with a computational model that yields a Pavlovian weight coefficient w, which governs the relative weighting of state (Pavlovian) versus state-action (instrumental) values and reflects dynamic
changes in controllability inferences across the blocks of the task. In this model, the Pavlovian learning process estimates
the mean reward value for a given stimulus 
, whereas the instrumental learning process estimates the mean reward value for a given stimulus and action 
. The value of both estimates is updated on each trial through an error-driven learning process (here, described for 
, but which also applies to 
):
(1)
where δ is the prediction error (r − 
), which reflects how much better or worse an outcome (r) was than expected. η is a dynamic learning rate that is incremented by 1 after each encounter with the stimulus, yielding smaller value updates
on each trial. Additionally, a single initial value for both stimulus and action values and an initial learning rate are free
parameters. Initial stimulus and action values reflect individual reward expectations at the beginning of the task, while
the initial learning rate determines the degree to which these values should be updated following each encounter with a stimulus.
Instrumental values are equal to action value estimates 
, whereas the Pavlovian value is equal to the stimulus value estimate 
for Go actions, or 0 for No-Go actions. This results in positive state value estimates promoting “Go” responses, and negative
value estimates discouraging “Go” responses, thus promoting “No-Go” responses.
In controllable environments, using instrumental values to inform action selection will yield more rewards. In contrast, in
uncontrollable environments, where actions do not influence reward outcomes, Pavlovian and instrumental values will predict
reward equally well. Thus, the differential reward-predictive ability of instrumental versus Pavlovian values provides evidence
for controllability or uncontrollability. Within the model, the degree of environmental controllability (L) is estimated using a log-odds convention, with the prior log-odds given by:
(2)
The posterior log-odds are updated according to Equation 3, which assesses the predictive accuracy of state versus state-action values. If the state-action (instrumental) values fail
to forecast a more favorable outcome than the state (Pavlovian) values, L will rise, signaling an increasing perception that the environment is uncontrollable.
(3)
Using the relation 
, the Pavlovian weight parameter (w) is updated on each trial. A larger Pavlovian weight (i.e., a greater posterior probability that the learner is in the uncontrollable
environment) drives reflexive Pavlovian behavior; whereas a smaller Pavlovian weight yields a greater reliance on instrumental
actions.
(4)
The Pavlovian weight was initialized at 0.5 at the start of the first block, reflecting unbiased beliefs about the controllability
of the environment, and its value at the beginning of the second block was a free parameter (w2) estimated for each individual. The value of L at the start of each half of the task was initialized with respect to each of these initial weights as:
(5)
Weighted action values were converted into probabilities using a softmax choice function with an inverse temperature parameter
(β) governing action stochasticity:
(6)
Previous studies introducing this model included separate initial values (stimulus and action values and learning rate) for each task condition; controllable or uncontrollable (Dorfman and Gershman 2019; Gershman et al. 2021). Here, we tested two additional variants of the original model: the first was a simpler version, with a single initial reward and learning rate parameter used across both blocks. The second variant introduced a single initial reward and learning rate parameter as well as an additional free parameter w2 to account for the potential carryover of Pavlovian bias levels from the first to the second block. While the original model was favored in model comparison (Supplemental Fig. S2), it had low parameter recoverability. Hence, we adopted a simpler version of this model with a single initial reward and learning rate parameter used across both and the additional free parameter w2. This model had the second-best model fitting results, but better parameter recovery compared to the original model. For additional details regarding model fitting procedures, model comparison, parameter recovery, and posterior predictive checks, see Supplemental Material and Supplemental Figures S2 and S3.
Data access
The original data and analysis code are available on Open Science Framework (https://osf.io/e49ua/).
Competing interest statement
The authors report no biomedical financial interests or potential conflicts of interest.
Acknowledgments
This work was supported by National Science Foundation grant no. 1654393 to C.A.H. We thank Kate Nussenbaum for helpful feedback on the manuscript. This work was also supported by the NYU Vulnerable Brain Project.
Footnotes
-
[Supplemental material is available for this article.]
-
Article is online at http://www.learnmem.org/cgi/doi/10.1101/lm.053901.123.
- Received November 13, 2023.
- Accepted February 19, 2024.
This article is distributed exclusively by Cold Spring Harbor Laboratory Press for the first 12 months after the full-issue publication date (see http://learnmem.cshlp.org/site/misc/terms.xhtml). After 12 months, it is available under a Creative Commons License (Attribution-NonCommercial 4.0 International), as described at http://creativecommons.org/licenses/by-nc/4.0/.
