Neural coding of reward magnitude in the orbitofrontal cortex of the rat during a five-odor olfactory discrimination task

Results

Behavior

For the analysis, data were used from 24 recording sessions, obtained from seven rats. Animals needed on average ∼17 trials for each of the different positively reinforced trial types to reach the criterion of 15 successful trials per reward size; there was no difference in performance level between the different amounts of reward (number of trials needed for 0.05 mL: 16.6 ± 0.4; 0.15 mL: 16.9 ± 0.5; 0.30 mL: 17.4 ± 0.6). Rarely the number of 15 trials per reward size was not reached during a session. For the nonreinforced and quinine condition, animals made on average 5.8 ± 0.7 and 4.5 ± 0.7 “go” responses, respectively (sucrose versus quinine or versus nonreinforced: P = 0.000; paired sampled t-test). In the course of a session these erroneous “go” responses (false alarms) were followed by trials in which animals, after sampling odors predictive for these outcomes, began to withhold responses toward or at the fluid well. Withholding was typically followed by odor pokes with durations shorter than the required 2 sec, thus yielding invalid trial types. This indicates that the animals were actually capable of associating an odor with either a positive or negative outcome. Since the minimal number of trials accepted for the analysis of the electrophysiological data was six, neural responses to the various stimuli applied during the nonrewarded trial type were not examined any further. However, sessions in which six or more trials for quinine were present (n = 9) were used to determine whether neural responses showing reward magnitude differences were compatible with response patterns for quinine (see below). This was done by comparing neural responses during the quinine trials with responses during the trial type with the same amount of sucrose (0.15 mL).

Examination of the movement time showed no significant differences between the positively reinforced trial types (0.05 mL: 1.43 ± 0.04 sec; 0.15 mL: 1.35 ± 0.03 sec; and 0.30 mL: 1.37 ± 0.04 sec). A comparison between these trial types and the nonreinforced and quinine trials revealed a significant difference: movement time during positively reinforced trials taken together (1.38 ± 0.03 sec; sample sizes per reward magnitude: 0.05 mL: n = 336; 0.15 mL: n = 343; 0.30 mL: n = 356) was significantly shorter compared to nonrewarded and quinine trial types (nonrewarded “go” responses: 1.50 ± 0.06 sec, n = 138; quinine “go” responses: 1.74 ± 0.13 sec, n = 107). However, when the entire sequence of odor sampling and moving to the fluid well was taken into consideration (overall response time), rats performed significantly faster for the 0.15 mL and 0.30 mL trial types (3.46 ± 0.06 sec and 3.51 ± 0.05 sec, respectively) as compared to the 0.05 mL sucrose trials (3.68 ± 0.06 sec).

Histology

Histological verification of the tetrode positions (Fig. 1) showed that the recording sites ranged between 2.7 mm and 4.2 mm anterior to bregma, and were limited to the ventral and lateral orbital regions of the OBFc. Recording depth ranged from ∼3 mm to 5.5 mm below cortical surface (Paxinos and Watson 1996).

Electrophysiology

General overview

During the 24 recording sessions that were mainly performed on consecutive days, a total of 894 single units were recorded in the OBFc. The number of single units recorded per session ranged from 24 to 61 (mean ± SEM: 38.9 ± 2.2) with mean firing rates ranging from 0.06 to 48.4 spikes/sec. Of these 894 neurons, 141 (16%) showed 176 statistically significant changes in firing rate correlated to one or more main events in the task (Table 1). These neural correlates consisted of responses during the following four events or phases: odor sampling, behavioral activity preceding the nose entry into the fluid well, the waiting period, and the delivery of reinforcement (Fig. 2). The remaining neurons failed to demonstrate any significant task-related modulation as revealed by the statistically assessed histograms and were not examined further. For the analysis of neural correlates, only positively rewarded trial types were taken into consideration.

During odor sampling, a total of 44 neural responses were found (25%), all consisting of an increase in firing (Fig. 2A). Many of the cells showing this correlate (45.5%, n = 20) started firing when the nose was already in the sampling port but before odor was presented, activity that might reflect the anticipation of odor delivery or the onset of behaviors related to odor sampling. All responses during odor sampling typically peaked within 1500 msec after odor onset. Because the task was not designed to determine whether differential responses during odor sampling were due to different sensory inputs (odor identity) or expectancy for different reward magnitudes, the results are inconclusive regarding the coding of expected reward magnitude during this phase. Therefore the responses observed during odor sampling were not analyzed further.

In “go” trials, the presentation of the odor stimulus was followed by the animal’s movement from the sampling port toward the fluid well and the subsequent waiting period in the fluid well before reinforcement delivery. In both phases, neurons demonstrated significant changes in firing. In the task phase, where the animal moved from the sampling port to the fluid well, 33 correlates (19%) consisting of an enhancement in firing rate were found (Fig. 2B). During the waiting period, 47 responses (27%) were found, one of which showing a decrease in firing activity, whereas the other 46 consisted of an enhancement in firing (Fig. 2C).

During the reinforcement delivery phase, all animals displayed similar behavior at the fluid well: for every amount of sucrose solution, their snouts remained in the fluid well until the end of trial. In response to positive reinforcement, 52 correlates were found (29%), consisting of five different response types. The first type of response consisted of a transient increase in firing rate, peaking within 500 msec after reinforcement delivery (27%, n = 14) (Fig. 2D), and the second type consisted of an increase in firing rate that peaked between 500 and 1500 msec after delivery (31%, n = 16). The third type of response consisted of an increase in firing rate starting almost immediately after the delivery of the reward, remaining during the entire period the animal visited the fluid well (15%, n = 8). The fourth type was a rarely encountered transient decrement in firing rate (8%, n = 4), whereas the fifth type comprised all other responses showing enhancements in firing rate starting at least 3 sec after reward delivery and with variable duration (19%, n = 10).

Neural activity during the movement and waiting period

After determining the various neural responses to task events, the question arose whether expectancy for the upcoming reinforcement was represented within these different types of neural correlates. Given the absence of changes in sensory input and the animal’s overall immobility, neurons that showed responses within the waiting period in the fluid well may code the predicted outcome. However, to exclude the possibility that these responses were due to anticipatory licking of the animal instead of expected outcome per se, neural responses (n = 22) recorded during sessions in which lick detection was available were examined in relation to the extensions and retractions of the tongue in the fluid well. This showed that these responses did not covary with licking behavior since in these cases anticipatory licking was either completely absent, started after the neural response, or, when overlapping in time with the neural response, continued during reward consumption, when the neural response had already ended. Furthermore, a comparison between neural responses in this period between (incorrect) “go” responses during quinine trials versus “go” responses during sucrose trials revealed that 10 out of 11 neurons recorded showed differential firing for the sign of the response outcome during this period. Therefore, the present data are consistent with neural coding of reward expectancy in a subset of OBFc neurons during the waiting period.

Expectancy for reward, however, might also occur earlier in the task sequence, for example, before or during the movement period of the “go” response. To examine whether neural responses observed during the movement phase were related to the “go” movements of the animal per se or whether they reflected a truly goal-directed action in the context of the task, neural responses obtained during the correct “go” trials were compared with those during performance of the same behavior within the intertrial interval. This showed that these correlates were task dependent for 30 out of the 33 neurons exhibiting significant responses in this period: they were not observed when the behavioral sequence (“leave odor port and go to reward site”) was performed during the intertrial interval (Fig. 3). In three units, a significant response occurred during the intertrial interval at a similar time point as during correct “go” trials, but these activations were not as strong as the responses observed during trials. In addition, a comparison between neural responses during (incorrect) “go” responses during quinine trials versus “go” responses during sucrose trials revealed that during the movement period, 12 out of 14 neurons recorded demonstrated differential firing depending on the response outcome.

Neural activity related to different magnitudes of reward

Whether neurons showed significant differences in response to the three different reward magnitudes was examined for three phases, namely, the responses occurring during the movement period preceding the fluid poke, the waiting period, and the period starting with reward delivery. A total number of 67 (55%) of the 122 correlates found during these three task phases demonstrated statistically significant differences between different amounts of reward, which was either between two (66%) or all three (34%) different reward sizes. The proportion of neurons showing differential activity varied for the three task periods and ranged from 45% to 69% (Table 1).

Within the group of 33 neurons demonstrating task-specific neural correlates during the movement period preceding the fluid poke, 17 neurons (52%) demonstrated statistically significant differential firing across different reward sizes, with seven and 10 neurons discriminating between two and three different sizes, respectively (Fig. 3). Some neurons displayed firing activity that increased with incremental reward size (18%, n = 3 out of 17), whereas other neurons displayed the inverse relationship (29%, n = 5) or demonstrated the largest or smallest response to the middle reward size (53%, n = 9) (Fig. 4A). During the waiting period, in which 47 neurons showed significant responses, 21 units (45%) discriminated significantly between different magnitudes of reward (Fig. 5). A total number of 18 of these 21 neurons showed differential activity between two different reward sizes, three neurons between three sizes. A similar heterogeneity within the tuning of the neurons was found as during the movement period: units showed a rise in firing activity with increasing reward size (29%, n = 6 out of 21), whereas other neurons displayed the inverse relationship (24%, n = 5) or showed the largest or smallest activation to the middle reward size (48%, n = 10) (Fig. 4B).

Besides expressing neural correlates related to reward magnitude in the periods preceding reinforcement delivery, OBFc neurons may code aspects related to the actual reward magnitude. After examining whether neural responses (n = 10) in the period of reward delivery were related to the tongue movements of the animal during licking, none of the responses could be ascribed to licking behavior. Furthermore, due to the different nature of the responses observed in this period, determining whether neurons demonstrated discriminatory responses to the different magnitudes of reward was done using different criteria for the various response types. For the transient response types, only the time periods in which the response occurred were compared; for the sustained responses, the entire period with enhanced firing was tested. Responses of the fifth type were left out of the analysis due to the variability of the responses within this group of neurons. Of the 42 neurons showing responses of the remaining four types, 29 neurons (69%) showed significant differential firing across reward sizes, with 19 and 10 neurons discriminating between two and three different reward sizes, respectively (Fig. 6). Again, all types of response profiles were demonstrated: increasing (21%, n = 6 out of 29) or decreasing (14%, n = 4) firing activity with increasing reward size, or the largest or smallest response to the middle reward size (66%, n = 19) (Fig. 4C).

An additional Kolmogorov-Smirnov test (P < 0.05) was performed for the responses that peaked between 500 and 1500 msec (Fig. 6A, second response type), since this subgroup often showed a response time course varying with reward magnitude. Taking firing rate values for successive bins as variable, this test revealed that for all neurons of this second type (n = 16), the time course of the response differed either between two (53%) or three (47%) reward sizes as well.

Discussion

The present experiment was performed to examine whether and how information concerning reward magnitude is coded in rat OBFc. A total number of 141 out of 894 neurons recorded (16%) demonstrated a behavioral correlate of firing activity during several events in the task, that is, odor sampling, the movement period from odor port to the fluid delivery well, the waiting period in the fluid well, and the reward delivery period. During the movement and waiting period, respectively, 52% and 45% of the neurons showing a significant behavioral correlate demonstrated differential firing across varying reward sizes. After reward delivery, 69% of the neurons showed differential responses toward the various reward sizes.

Overall, the percentage of neurons showing significant correlations during this olfactory discrimination task (16%) was lower than in a previous study with an eight-odor olfactory discrimination task (Schoenbaum and Eichenbaum 1995). Apart from differences in the behavioral paradigms, this might be due to the conservative statistical assessment in the present study, to methodological differences in unit recording and isolation, or to the fact that recording sites in the latter study were located more laterally in the OBFc and also comprised the ventral part of the agranular insular cortex, whereas in the present study locations of recordings were limited to the ventral and lateral orbital regions.

Testing the ability of animals in a T-maze task to discriminate between the different reward sizes used in this behavioral task demonstrated that animals were able to discriminate the various reward sizes. In the five-odor discrimination task, the scores on overall response time revealed that animals responded significantly slower for the smallest amount of reward, indicating that animals learned to discriminate at least between the smallest and the two larger amounts of reward. The finding of this difference not being present within the movement time might be explained by the limited number of trials. The differences could have turned out to be significant when the number of trials per session had been larger. However, other possible explanations include a higher degree of stereotypy or habit-based performance of “go” responses, once initiated, than applies to the odor sampling phase. That the movement time was highly similar across different reward sizes makes it likely that the discriminatory neural responses during the movement period were not due to differences in motor-related aspects, including the speed of responding.

Neural responses during the reward delivery phase: Magnitude effects

As concerns actual reward magnitude, the results indicate that neurons in rat OBFc may, indeed, code information related to this reinforcement parameter, showing differential firing across various reward sizes during reward consumption. Although this discriminative firing appears not to be due to licking behavior, there is the possibility of modulation of these responses by sensory-motor processes (e.g., tasting or ingestion), insofar as this is distinguishable from reward processing per se. The observation that the OBFc is involved in the processing of information related to the magnitude of the actual reinforcement is in line with the fMRI result in humans that distinct regions within the OBFc represent the magnitude of monetary reward (O’Doherty et al. 2001). Furthermore, it fits in with earlier findings concerning neural coding of other aspects of reinforcers in the OBFc, such as qualitative properties or motivational value (Thorpe et al. 1983; Tremblay and Schultz 1999; Hikosaka and Watanabe 2000; Wallis and Miller 2003; Roesch and Olson 2004; Ichihara-Takeda and Funahashi 2006; Padoa-Schioppa and Assad 2006). Together with the present result, this implies that at least some major aspects of reinforcers that might be influential for the selection of the most appropriate response strategy are represented within the OBFc.

Neural coding of reward expectancy

In a previous study by Schoenbaum et al. (1998), neurons in the OBFc were suggested to code expected outcomes, since they demonstrated discriminatory firing during the delay period in which animals awaited either positive or negative reinforcement, which is a result confirmed in this study. However, a confounding factor in the study by Schoenbaum et al. (1998) was the comparison between “go” responses for both positive and negative reinforcers, the latter being erroneously made. The possibility of such a confounding factor is underscored by a study of Ramus and Eichenbaum (2000) on olfactory recognition memory, who reported units in rat orbitofrontal cortex showing firing rate changes during odor sampling that occurred specifically prior to erroneous “go” responses. The advantage of the parametric design in the present study for the examination of reward expectancy is that very similar “go” responses can be compared, since the impending outcomes are all appetitive. Indeed, we found no significant difference in movement time across the three positively rewarded trial types, while the waiting period was fixed in duration (1.5 sec). Thus, the present design may provide more certainty about the nature of the observed neural correlate, since no change or switch in response strategy is required during these trials.

Nonetheless, it remains possible that other cognitive processes besides reward expectancy, like working memory or attention, although not obviously required for correct task performance, are reflected within the waiting or movement period. An additional remark is that, although learning of the discrimination problem by the animals was obvious from both “go”/“no-go” decisions, movement and overall response time for all five odors, it would be difficult to actually show learning-related changes in single-unit firing. Animals learned to discriminate positively versus negatively reinforced odors within a few trials (average for quinine trials: 4.5 ± 0.7), and the number of spikes during the waiting period per trial appears insufficient to permit a robust analysis of the evolution of the neural representation of stimulus–reward contingencies across these few trials. Resolving this issue will necessitate a new study in which learning is temporally spread out across more trials. Despite this, it does not seem likely that the reward-size differences in neural activity would be attributable to carryover effects from previous sessions. First, new odors were used in every new session, and care was taken to avoid constant reinforcement contingencies of the odor families used here. Second, interference by previously established contingencies would be expected to prevent the emergence of a consistently slower response to the odor predicting the lowest amount of reward as compared to the high-reward contingencies.

Reward expectancy: Timing and magnitude effects

Discriminatory activations toward different magnitudes of reward were found to occur during the waiting period as well as during the movement period, the latter likely being an expectancy-related correlate during the “go” response that to our knowledge has not been described earlier in rat or primate OBFc. In addition to the rat OBFc, expectancy-related activity was previously found in rat basolateral amygdala (BLA), with neurons demonstrating selective firing during a delay period prior to reinforcement or during predictive odor cues after learning (Schoenbaum et al. 1998, 1999; for review, see Holland and Gallagher 2004). In addition to these two types of expectancy-related activity, the results of the present study indicate that at least in the OBFc, reward expectancy may also be coded during the movement phase of the “go” response. Although this neural correlate occurred during motor behavior, it was absent when the animal executed the same movement outside the context of the trials, thus excluding a pure motor confound. The presence of this expectancy-related signal matches the results from lesion studies indicating the involvement of the OBFc in the flexible guidance of goal-directed behavior (Dias et al. 1996; Meunier et al. 1997; Hornak et al. 2004; Izquierdo et al. 2004) and is also consistent with a role for OBFc in encoding motor set as well as action-outcome relationships (Baxter et al. 2000). Especially when reward contingencies are changing, information representing these relationships should be available during task performance, particularly before and during the execution of a behavioral response. The idea that a reward expectancy signal is available during ongoing behavior is supported furthermore by the finding that neurons in the primate OBFc may code both long- and short-range reward expectancy (Hikosaka and Watanabe 2004). It is suggested by these investigators that during behavior these different OBFc signals serve to adjust motivational levels across different temporal ranges, namely, toward an immediate or a more distant outcome. Altogether, the available results support the notion that reinforcement parameters such as magnitude and quality are coded across different temporal phases along task trials, corroborating the hypothesis that OBFc neurons collectively code a matrix of reward parameters as a function of the delay toward the moment of outcome. Future recordings will be needed to study the function of neurons with expectancy-related firing during movement periods in more detail, for example, during different motor behaviors or reversal learning.

A previous study by Roesch et al. (2006) already demonstrated neurons within rat OBFc discriminating between two different reward sizes. However, apart from the fact that reward size was not parametrically varied (only two different reward sizes were used), their results are not directly comparable with the results from the present study. Additional differences concern the method of reward application and location of reward delivery. Roesch et al. (2006) applied a single bolus of 0.05 mL as a small reward and an additional drop 500 msec after the first bolus as the large reward, with different reward sizes administered at different locations. In contrast, in the present study all reward sizes were delivered in a single dose and at a common location.

The finding of the present study that information concerning the magnitude of reward is represented in rat OBFc is in line with one of the predictions from the HSAT (Hebbian Synapses with Adaptive Thresholds) model of RL (Pennartz 1997), in which the OBFc, together with other related limbic areas, functions as a brain center for processing and predicting reinforcement, as well as directing Hebbian changes in synaptic strength in brain areas targeted by the OBFc. The presence of information comparing the actual versus the expected magnitude of reinforcers within the OBFc was modeled to direct behavior of a neural network performing parallel distributed processing toward the most profitable outcome in an efficient and flexible manner. In particular, the observation that about one-half of the OBFc neurons (23 out of 52) demonstrating responses during reward delivery also showed expectancy-related activity during the waiting period is in line with the model, since in the course of learning, neurons within the reinforcement processing module of the HSAT network that are initially activated by the reward will become activated during the expectancy period preceding reward as well. Besides the OBFc, other brain areas emitting glutamatergic fibers may serve as candidate regions for the reinforcement processing posited by this model, such as medial prefrontal cortex (Pratt and Mizumori 2001), anterior cingulate cortex, and dorsolateral prefrontal cortex (Watanabe 1996; Leon and Shadlen 1999; Tremblay and Schultz 1999; Shidara and Richmond 2002). That the OBFc neurons in the present study exhibited a variety of different tuning curves to reward magnitude—either during the expectancy phase or actual reward delivery—constitutes a result that was not directly predicted by the computational model. This finding is in line with results of Wallis and Miller (2003) demonstrating that neurons within monkey OBFc code expected reward magnitude both parametrically and nonparametrically. Such a diversity of tuning is as compatible with coding in a parallel-distributed network as is a monotonic relationship between firing activity and reward magnitude. Adopting a wider perspective on RL than prescribed by any particular model, it should be mentioned that reward variables, including magnitude, are also represented in the striatum (Hollerman et al. 1998; Hassani et al. 2001; Cromwell and Schultz 2003). Although recent results by Pasupathy and Miller (2005) showed that learning-related changes in firing activity in a cue-saccade association task occurred earlier in striatum than prefrontal cortex, the actual anatomical loci mediating stimulus–response and action–outcome learning remain to be firmly established. The distribution of the information across several cortical and subcortical areas suggests a broader computational system by which RL can be mediated than a basic actor–critic architecture trained by temporal difference learning that is composed of the striatum and mesencephalic dopamine system (for review, see Montague et al. 2004).