RL-DDMs. (A) Model comparison. LOO-IC favors model M12, implementing separate drift rate intercepts for Win and Avoid cues and separate non-decision times for low stakes, congruent cues under high stakes, and incongruent cues under high stakes. (B) Densities of best-fitting parameters for model M12 per participant. Drift rate intercepts for Win cues are consistently higher than drift rate intercepts for Avoid cues. Note that, although the winning model implements separate non-decision times for high/low stakes and congruent/incongruent cues, the parameter values for these different conditions are not significantly different from each other. (C) Posterior predictive checks for the winning model M12. (Left panel) Simulated proportion of Go responses per required action and cue valence averaged over simulations and participants. The winning model M12 reproduces Pavlovian biases in responses and RTs (see Supplemental Material S07). (Right panel) Simulated RTs per cue congruency per stakes level averaged over simulations and participants. The winning model M12 reproduces the overall slowing under high stakes as well as differences in slowing between congruent and incongruent cues, but somewhat underestimates this difference compared to the empirical data. For further plots, see Supplemental Material S07. (D) Parameter recovery for the winning model M12. Correlations between generative parameters used for simulating 1000 data sets based on M12 and parameters obtained when fitting M12 to simulated data. All correlations between generative and fitted parameters (on-diagonal correlations) are significantly above chance (Mr = 0.83, SDr = 0.14, range 0.62–0.98; 95th percentile of permutation null distribution: r = 0.08; see Supplemental Material S07 for scatter plots of on-diagonal correlations). Besides correlations between generative parameters with their corresponding fitted parameters, there were two notable cases of off-diagonal correlations: first, the different non-decision times (under low stakes, under high stakes for congruent cues, and under high stakes for incongruent cues) were correlated (r = 0.71 and r = 0.77), reflecting an overall tendency toward faster/slower responses that is naturally shared across all three parameters. Second, learning rates and drift rate slopes were negatively correlated across parameter settings (r = −0.56), which mimics the frequently observed trade-off between learning rate and inverse temperature parameters in more classic RL models of choices (Ballard and McClure 2019). In RL-DDMs, the drift rate slope is multiplied with the Q-value difference, such that steeper slopes lead to more deterministic choices and shallower slopes lead to more stochastic choices, similar to an inverse temperature parameter. (E) Model recovery for models M1–M12. The forward confusion matrix displays the conditional probabilities that model Y is identified as the best-fitting model (columns) if model X (rows) is the underlying generative model used to simulate a given data set. On-diagonal probabilities indicate the probability of reidentifying the generative model. All on-diagonal probabilities are significantly above chance (Mp = 0.31, SDp = 0.32, range 0.13–0.98; 95th percentile of permutation null distribution: P = 0.10). Model recovery was particularly high for the winning model M12, which was the best-fitting model for 98% of data sets for which it was the generative model. Recovery for the other models was not quite as high, though still significantly above chance for all models. Hence, while our model selection procedure will likely identify M12 if it is the data-generating process, it might struggle in other scenarios in which other DDM parameters are affected by experimental manipulations. For the inverse confusion matrix and matrices on subsets of models, see Supplemental Material S07.
