A simple neural network model of the hippocampus suggesting its pathfinding role in episodic memory retrieval

The phenomenon of hippocampal spatial representations and the hippocampal role in episodic memory retrieval remain two of the most puzzling mysteries in cognitive neuroscience that intuitively seem to be connected to each other. Since the finding that the hippocampus plays a pivotal role in long-term memory consolidation (Scoville and Milner 1957; Zola-Morgan and Squire 1986), many proposals have been made regarding its specific role (Teyler and Discenna 1985; Squire 1987; O'Reilly and McClelland 1994; McClelland et al. 1995). A prominent view of the mechanisms underlying consolidation of episodic memories involves fast formation (e.g., via Hebbian mechanisms) of strong associations between hippocampal sparse patterns of activity and distributed neocortical representations. As a result, the former subsequently serve as pointers to the latter. This memory-indexing theory that goes back to Teyler and Discenna (1985, 1986) and underlies several subsequent major theoretical contributions to the field (Nadel and Moscovitch 1997, 2001; Wheeler et al. 1997; Tulving 2002) assumes that a memory of an episode is retrieved by reactivating a hippocampal pointer to it. Consistent with this view, recent clinical and fMRI studies indicate that the hippocampus in humans is involved in and required for retrieval of all autobiographical, but not semantic memories (Vargha-Khadem et al. 1997; Ryan et al. 2001; Westmacot et al. 2001; Maguire and Frith 2003).

At the same time, details of this picture remain unclear. In order to retrieve (and make sense of) an episodic memory, it is necessary to reconstruct contextual information that was present at the time of encoding. For some episodic memories, the vital contextual information may not be explicitly encoded in memory traces or their associations. Consider the following example. An idea of driving under a building may never come to your mind while planning a way home, because you know that this is impossible. Your decision making is influenced by this knowledge, while on the other hand, you are not aware of this knowledge at the time of decision making, and therefore, you will not remember it in association with the episode. This knowledge, however, may not be in effect in the future when a tunnel is built. From the future context point of view, your episodic memory of driving home acquired before the construction began may seem irrational, and therefore, hard to retrieve without reinstating the context of the past. According to Moscovitch and Melo (1997), episodic memory traces lack their historical context (e.g., time stamps), but may include associative context (which refers to the multimodal, e.g., spatial background that comprises the event). Therefore, strategic retrieval (as opposed to cued recall) of an episodic memory may require reconstruction of the original context of experience step by step via effortful, guided memory search that is controlled by the prefrontal cortex (PFC) (Moscovitch 1989, 1992; Moscovitch and Melo 1997; Fletcher and Henson 2001). The role of PFC in this process amounts to cue specification, strategic search, maintenance, monitoring, and verification of retrieved information, while medial temporal lobe cortices (MTL) and the hippocampus may play the role of pattern completion and comparison (Simons and Spiers 2003). Experimental data confirm the idea that the hippocampus is essentially involved in the process of contextual reinstatement (e.g., Dobbins et al. 2003). Its particular functional role in this process is currently hotly debated in computational modeling studies (e.g., Howard and Kahana 2002; Becker and Lim 2003).

On the other hand, hippocampal memory function must be reconciled with the spatial semantics of hippocampal activity patterns, which have been intensively studied for more than three decades. The rodent hippocampus is known to represent spatial maps of environments. Hippocampal pyramidal cells, called place cells in rodents, each selectively fire at a high rate when the animal is located in a particular spatial domain, called a place field of the place cell. This type of firing is observed during active maze running (O'Keefe and Dostrovsky 1971), and to some extent, during slow-wave sleep states (Wilson and McNaughton 1994; Jarosiewicz et al. 2002; Lee and Wilson 2002; Jarosiewicz and Skaggs 2004). Equivalent or similar phenomena were observed in primates (Rolls and O'Mara 1995; Robertson et al. 1998), including humans (Ekstrom et al. 2003). The system of place fields forms a cognitive map of an environment (O'Keefe and Nadel 1978, 1979). Although hippocampal cognitive maps have been implicated in navigation and in formation of episodic memories (for review, see Redish 1999), many basic questions about underlying mechanisms and place field semantics remain unanswered. For example, what are the general principles of place field formation? Why do they come in various shapes and sizes and typically have broad tails? How can a cognitive map be used for navigation if its activity during maze running only provides information about the current location? Alternatively, what exactly do place fields encode? Several theoretical frameworks have been proposed to answer these questions, including local view models (e.g., Burgess and O'Keefe 1996), path integration models (McNaughton 1996; McNaughton et al. 1996; Redish and Touretzky 1997), and trajectory learning models (e.g., Blum and Abbott 1996). None of the proposed solutions seem to answer all questions.

It has long been noticed that the combination of the two apparently different functions in one brain structure poses a great challenge in theoretical neuroscience. The metaphoric analogy between linking remembered episodes by context and linking nearby locations in an environment (e.g., Wallenstein et al. 1998) characterizes the present level of understanding of this issue, although several more specific proposals were put forward (Levy 1996; Recce and Harris 1996; Gaffan 1998; Robertson et al. 1998; Burgess 2002). A common ground in these proposals is the notion of a hippocampal pointer providing necessary associations. In the present study, we construct a model of episodic memory retrieval, departing from a traditional assumption that reactivating a hippocampal pointer is sufficient for retrieval of the associated episodic memory. Our assumption is that an episodic memory trace may not be retrievable without reinstatement of a missing contextual key—a key that is not associated with the pointer. We propose a theory explaining how this key is found by the hippocampus, why finding it may be impossible without the hippocampus, and how this relates to spatial navigation. Specifically, we show, using numerical simulations, that one and the same mechanism can work for episodic memory retrieval and for spatial navigation. We take this observation as the basis for a conceptual unification of the two hippocampal functions.

The central idea of the proposed approach can be unfolded as follows. Because the context of the retrieval conditions may be substantially different from the context of the remembered episode, the system must “prepare itself” for retrieval by reinstating vital contextual information that is not referenced in the target episode. This information needs to be acquired from a set of other episodic memories that serve as a key for understanding the target episode. This acquisition can be subserved by the system taking a quick path through the key set, briefly reactivating necessary memories one by one. Finding a correct path may be crucial for a successful retrieval. How can one decide which path is “correct”? In this study, we say that there is a connection from episodic memory A to episodic memory B, if, given an explicit awareness and an understanding of A, the system is capable of immediate retrieval and understanding of B without consulting any additional episodic knowledge (in contrast, available semantic knowledge may be used). Then, a rule that ensures the retrieval of an episodic memory X, given an initial state of awareness Y, is to follow a connected path from Y to X, i.e., a continuous path composed of relevant remembered episodes and connections among them.

One can think of a set of episodic memories and their connections as an abstract graph. Most of the content of episodic memories can be excluded from this model; nodes of the graph can be considered as pointers that merely label the essential context of remembered episodes. The connections of this graph may involve acts of imagination or assumption, perspective shifting, dropping or accepting various rules and conditions, etc. This interpretation is consistent with the view that episodic memories are reconstructed based on schemas rather than retrieved as images (e.g., Watkins and Gardner 1979; Parkin 1993; Moscovitch and Melo 1997; Schacter 1999; Rusted et al. 2000). The generalized notion of a cognitive map (O'Keefe and Nadel 1978) viewed as an abstract cognitive space, or a cognitive graph (Muller et al. 1996), or an internal cognitive model (Samsonovich and McNaughton 1997) can be understood as a set containing all previously explored contexts (no matter how abstract or specific they are) along with connections among them (understood in the above sense). Here, by an abstract context, we refer to a set of rules and facts that apply to a class of possible situations (McCarthy and Buvac 1997). Therefore, a given abstract context may specify a large set of episodic memories, yet it may be represented by one node in the graph (and accordingly, by one hippocampal pointer). Looking ahead, in our model, each CA1 unit will represent one such abstract context.

With this setup, considerations of efficiency of an episodic memory system inspire several conjectures. (1) It is reasonable to memorize available connections, at least a sufficient subset of them, in association with each remembered context or episode. (2) The resultant graph must be highly interconnected, in the sense that any shortest path in it has a limited, reasonably small length (measured as the number of connections), despite the huge size of the whole memory space (Alon 2003; Barabasi and Bonabeau 2003). An abstract example demonstrating this possibility is the space of 4.3·10¹⁹ states of Rubik's cube, where any shortest path involves no more than 20 moves, and each state offers only 12 possible moves. (3) A selected path in the graph must be among the shortest or nearly shortest paths. (4) Episodic memory system must be able to quickly find a nearly shortest path, given available connections, which makes its function closely related to that of a spatial navigation device. Our model assigns the pathfinding function to the hippocampus in addition to its other, better-known cognitive map functions (including multimodal associations, episodic memory indexing, and spatial self-representation). Validation of this proposal involves (1) analysis of experimental data supporting the mathematical model; (2) formalization of the concept outlined above in terms of the model; and (3) numerical application of the model to spatial, problem-solving, and episodic memory-retrieval paradigms selected to demonstrate the concept in action.

Connectivity in Cornu Ammonis

Cornu Ammonis (CA), or the hippocampus proper, consists of areas CA1–CA3. The principal neurons in CA areas are known as pyramidal cells (based on the shape of their somata), complex spike cells (based on their activity pattern), or place cells (based on their behavioral correlates). There are ∼3 × 10⁵ pyramidal cells in CA3 and 4 × 10⁵ in CA1 (Amaral et al. 1990). Multiunit recordings in freely behaving rats (Wilson and McNaughton 1993) show the following essential details. (1) Relative anatomical location of place cells does not correlate with their spatial preferences (Redish et al. 2001). (2) Approximately 0.1% of all CA3 pyramidal cells and ∼1% of all CA1 pyramidal cells are active at any given location in an environment (McNaughton et al. 1996). Accordingly, place fields in CA1 are typically bigger than in CA3. (3) Any given CA3 or CA1 pyramidal cell has a probability of about 30% to exhibit activity within a given environment (McNaughton et al. 1996). Typically, this activity is spatially selective (compact place field); nevertheless, it can be spontaneously exhibited at any location in the environment (broad “tails” and background noise). The other 70% of cells, however, may remain silent for the whole recording session.

A typical CA3 pyramidal cell is innervated by ∼12,000 other CA3 pyramidal cells (Ishizuka et al. 1990; Amaral and Witter 1995) via wide-spread projections (the axonal field of each cell covers ∼1/3 of the hippocampus) (Miles et al. 1988). A typical CA1 pyramidal cell is innervated by ∼16,000 CA3 pyramidal cells via Schaffer collaterals and commissural pathways (Bernard and Wheal 1994). These diffuse projections distributed throughout CA1 in a gradient fashion constitute the main input to CA1. In addition, both CA3 and CA1 areas receive diffuse input from the entorhinal cortex (which, in the case of CA3, is partially mediated by the dentate gyrus, DG). The main output of CA1 consists of topographically organized projections to the entorhinal cortex, both directly and through the subicular complex. The entorhinal cortex, in turn, has distributed two-way connections throughout the neocortex. Associative plasticity in CA1 appears to be crucial for memory consolidation (Shimizu et al. 2000).

Sharp waves and associative spatial learning

Sharp waves and ripples are high-frequency oscillatory events (lasting on average ∼100 msec) of nearly synchronous firing of many hippocampal place cells, accompanied by a temporary decrease or disappearance of the θ rhythm (O'Keefe and Nadel 1978; Buzsaki 1986; Sik et al. 1994). During typical behavioral experiments, rats exploring a maze do not run continuously (Fig. 1A). Rather, they intermittently pause, with a transient change in the hippocampal activity from θ mode to large-amplitude irregular activity (LIA), during which sharp waves and ripples are observed (Fig. 1A–C).

During a typical recording session of a freely behaving rat, sharp waves may occur frequently and spontaneously at arbitrary locations in the environment (Buzsaki 1986). Experimental data suggest that time-compressed replay of the recent trajectory may occur during sharp waves in sleep and wake states (Nadasdy et al. 1999; Louie and Wilson 2001; Lee and Wilson 2002; Buzsaki et al. 2003). The following similar observations can be made from Figure 1, B and C. (1) Different sharp waves are characterized by different firing patterns. Only a fraction of cells that were recently active fire in a sharp wave. (2) Cells that were active a few seconds before a sharp wave are more likely to fire in a sharp wave. (3) Cells that were not active immediately during the last few seconds before a sharp wave, but were active recently during the maze running, still can fire in the sharp wave. These observations are consistent with the findings of Csicsvari et al. (2000) showing correlations between CA3 spikes and CA3-CA1 ripples. They underlie the choice of learning rules detailed in the Materials and Methods.

Phase precession: Exploration of feasible actions?

The phenomenon called “phase precession” (PP) was discovered by O'Keefe and Recce (1993) and subsequently studied by Skaggs et al. (1996) and others. It consists of oscillations of the phase of place-cell firing with respect to the θ rhythm. According to a common description of this phenomenon (Skaggs et al. 1996), the firing phase of a place cell gradually changes from late to early phases of the hippocampal θ rhythm, as the rat traverses the place field of the given cell. An alternative, equivalent description of the same phenomenon can be given in terms of a population code (Samsonovich and McNaughton 1997). In short, one can imagine positioning all hippocampal place cells in the environment, each at the maximum of its own firing rate distribution. Over such an abstract construct (called a chart), the plot of a momentary smooth distribution of place-cell activity would result in a bell-shaped function (the activity packet) (Fig. 1 in Samsonovich and McNaughton 1997), which oscillates with the θ rhythm in the direction of the animal's head (Fig. 1D,E). These oscillations constitute PP.

Many theories have been recently proposed regarding the mechanisms and the function of PP, including its role in learning associations among sequentially visited places (Skaggs and McNaughton 1996) and its possible origin from the modulation of the sensory input (Burgess et al. 1994; Mehta et al. 2002). Several interesting features of PP have been established experimentally, including the following (Skaggs et al. 1996). PP is not sensitive to visual or other sensory input, e.g., it is observed in complete darkness. PP is observed during linear one-dimensional unidirectional motion as well as during random two-dimensional foraging. In both cases, its direction is determined by the direction of the head of the animal, but the spatial amplitude of one-dimensional PP is approximately twice as large as that of two-dimensional PP (Fig. 1D,E). In addition, while one-dimensional PP is mostly focused in the direction of motion, two-dimensional PP fans out at a wide angle with respect to the current head direction (Fig. 1D,E). Because the latter fact is established by averaging the spike data over an entire recording session (usually lasting for 20 min or more), it may not characterize a typical momentary snapshot of the phenomenon. One possible interpretation of the fan-out average pattern (Fig. 1F) could be that the direction of PP reflects current or future turns of the rat (Muller et al. 1996; Lisman 1999); however, to the best of our knowledge, this idea is so far not supported by any direct experimental evidence.

In this work, we make an alternative assumption that PP explores possible directions of motion (more generally, feasible actions) in consecutive θ cycles (see Materials and Methods for details), thereby producing, on average, a fan-out pattern (Fig. 1F). This assumption is consistent with an observation by Harris et al. (2003) that different groups of CA1 cells fire in consecutive θ-cycles (Fig. 1G). We assume spontaneous alternation of active “cell assemblies” representing different directions of motion or different feasible actions (this assumption can be tested directly; see Discussion).

Materials and Methods

Model principles, organization, and dynamics

We start by defining a simple model of spatial learning and navigation. Essentially the same model is used to describe episodic memory retrieval. The central component of the model is the hippocampal network (Fig. 2). This component can be thought of as interacting with other functional modules to exchange, interpret, and use relevant information (Fig. 2A). While these other modules are only implemented algorithmically, the hippocampal network consists of CA3 and CA1 model units, each representing groups of neurons (Fig. 2B). The input–output interaction between the hippocampal network and other model components can be approximately mapped to the hippocampus proper and related brain areas (Fig. 2C). This modular architecture and its mapping onto the brain are further explained in the next subsection.

In the application of the model to spatial navigation, each small neighborhood in the environment has a unique representation in the system being associated with a particular CA3 unit and a particular CA1 unit. This is consistent with the (now widely accepted) view of rodent hippocampal pyramidal cells as place cells. Place cells have preferred locations in an environment characterized by a high firing rate of the cell. Any of these locations may happen to be a goal location under certain circumstances, and as such, we refer to them as “potential goals.” However, we do not assume that place cells specifically serve to represent goals.

We interpret the gradient of a firing-rate distribution of each place cell as encoded “driving directions” to the point of a maximal firing rate for this cell, which can be an arbitrarily selected point of the environment. In order to find a way to a goal, the system “listens” to the background activity of goal-related cells, while exploring alternative feasible actions during sequential θ cycles.

The model presupposes that, during exploration, the environment is navigated randomly, but extensively. However, as illustrated by Figure 3A, the system is not required to visit every location during exploration, and the frequency of visits (occupancy) during the exploration phase may not be evenly distributed among visited locations. Each visit to a particular location results in a strong activation of the corresponding CA3 and CA1 model units (Fig. 2). At various arbitrary moments during exploration, corresponding to the occurrence of sharp waves, the system pauses, and the current location is taken as a potential future goal. At this point, the activity mode changes from regular, θ-modulated place-cell firing to LIA with “sharp wave” reactivation of recently active CA3 place cells, with a firing rate proportional to the recency of their last strong activation. In addition, CA1 cells representing the current location are also reactivated (Buzsaki 1989). As a result, CA3 place cells become associated with the selected CA1 cell whose place field represents a potential future goal. The strength of the associations in this model is proportional to the recency of a place-cell firing during exploration: where W_ij is the efficacy (weight) of the synaptic connection from CA3 unit j to CA1 unit i, t is the current moment of time, and t_j is the time stamp of the last visit to the location associated with the unit j during the running session. The initial conditions for W's are arbitrary small values below 1/L_max, where L_max is the maximal (allowed) trajectory length that can be associated with the goal during the training session. The weights of reciprocal connections are set to a constant (W_ii = 2). Generally, the exact functional dependence (equation 1) of W on t-t_j is not essential for the results of simulations, as long as the function remains strictly monotonic and noticeably decreasing with the distance.

After the conclusion of an associative learning event, the system resumes random exploration of the environment, until another arbitrary potential goal is selected, and the associative process (a sharp wave) repeats. Thus, if the exploration phase continues until a preselected goal location is reached, the CA3 model units acquire potentiated, weighted connections to the CA1 representation of the goal. At this point, the network of place cells actually provides a capacity for finding a short path to this goal location, starting from any given location within the environment. This is accomplished by a simple algorithm; at each location, several randomly generated possible local moves are explored via PP, and the move that produces the strongest excitation of the place cell associated with the goal is selected and performed. This process is repeated until the goal is reached. In the case of two-dimensional spatial navigation, the PP direction fluctuates within a standard set (“right,” “left,” etc.) In the memory search paradigm (see below), PP explores those connections that are stored in the CA3-to-CA3 synaptic matrix. For the sake of parsimony, CA3-to-CA3 synapses are only included in the model of memory retrieval, and not in the model of spatial navigation.

Model architecture and its mapping onto the brain

In addition to the hippocampal module described above (Fig. 2B), the implementation of the spatial navigation algorithm implies the presence of an external “goal module” to interpret CA1 activity (Fig. 2A). The goal module is tuned to detect a maximal response during PP in the CA1 unit associated with the goal location. We assume that the goal module is also modulated by the θ frequency. In partial support of this assumption, is the observation that any “voluntary” movement of a rat, e.g., pressing of the “gas” pedal while driving a toy car, is phase-locked to the θ rhythm (Terrazas et al. 1997). The goal module is connected to a control module, which detects the PP direction that maximally excites the CA1 goal-related cell and selects the corresponding movement. A motor module connected to the control module executes this selection. A sensory module delivers the updated sensory and/or proprioceptive input translated into a new perceived context directly to the hippocampal module.

This model architecture can be mapped onto brain structures (Fig. 2C; see also Samsonovich and McNaughton 1997). Here, we emphasize one difference between the mapping of the model applied to spatial navigation and the mapping of the model applied to episodic memory retrieval. While the hippocampal module is mapped onto the hippocampus proper and plays the same functional role in both cases, the role of the environment (Fig. 2A) in a spatial navigation case is delegated to the extrastriate sensory neocortex (the neocortical component) (Fig. 2C) in the case of episodic memory retrieval. In particular, all episodic memory contents are assumed to be stored in this neocortical component. In the model applied to episodic memory retrieval, the control module maps onto PFC, while the functions of the goal, motor, and sensory modules are mediated by MTL and may also be partially attributed to PFC.

Computational implementation details

In order to apply the above general scheme (equation 1 and Fig. 2) to model a rat running in a typical experimental environment (e.g., Fig. 3A), a few more details need to be specified as follows: (1) the rat coordinates change continuously, (2) the animal possesses inertia, e.g., it cannot instantly stop or turn at a sharp angle, and (3) it is the acceleration rather than the location in space that the nervous system can control immediately. These features are captured by the following simplified equations of motion written in the discrete time t: where x is the two-dimensional location of the rat, v is its velocity, and the Gaussian random variable a is its acceleration, which has a zero mean and a standard deviation σ. During initial exploration, any randomly generated value is accepted. During goal-directed search, the “voluntary control” is introduced algorithmically as the selection (by the control module) of one among several (10) randomly generated values of a with the strongest PP effect on the goal unit (therefore, not all possible values of a, which constitute a continuum, are explored). The motion is dissipative; the model rat velocity decays over time with the exponent log α. The CA3 and CA1 units to be activated are selected at each discrete time step via rounding of x toward the nearest integer vector.

The values of parameters used in the simulations of Figure 3 were α = 0.875, σ² = 0.5. In addition, v was reset to zero at each bump into a wall. The numerical values of α and σ determine the average speed of motion and were adjusted to yield realistic rat-like trajectories. Primarily, however, the realism of the trajectories achieved by the model is due to the choice of equations of motion (equation 2) that describe the control of acceleration rather than position. Within the range of values for the parameters of equation 2 that provided realistic trajectories, the qualitative simulation results were robust and not sensitive to particular parameter combinations.

Nonspatial paradigms

In the case of the space of states of the three-disc Tower of Hanoi problem, only discrete transitions between neighboring nodes of the graph (Fig. 4A) constitute possible movements. Therefore, equations of motion (equation 2) were not used in this case. Instead, 10 randomly generated (possibly repeating) moves were probed at each node before selecting the best of them based on the associated weights, as described above. All other implementation details were the same as in the previous case.

To simulate an example of episodic memory retrieval event (Fig. 5), we use essentially the same model architecture (Fig. 2A) and dynamical rules as in the previous case, implementing only the following modifications. CA3 layer in the hippocampal module has now internal synapses (Fig. 2B; horizontal links), which store connections of the cognitive graph. Environment is replaced with a separate neural network representing the neocortical component. The implementational details of this component are described in the next subsection. The stable patterns in this neocortical network are attractors taken to represent “episodic memories.” Each of these patterns is assumed to be previously explored and associated with the hippocampal module units, thus binding together the dynamics of the two components. During the random memory exploration phase, the weights W are adjusted according to equation 1, and local CA3- to-CA3 connections are created corresponding to the transitions among attractors in the neocortical component. The memory-search phase proceeds exactly as the goal-search phase in the Hanoi Tower simulations, except that, now, two neural networks, hippocampal and neocortical, are working in parallel.

Attractor model of the neocortical component

A simple model of the neocortical component was simulated explicitly to illustrate the role of the hippocampus in an example of memory retrieval (Figs. 5, 6, 7). The neocortical component was implemented as an attractor network of neuronal units characterized by their activity rates v_i (equation 3): Here, r_i are abstract, randomly generated two-dimensional coordinates assigned to each model unit (Fig. 6), I is the strength of the external input to the network from a hippocampal pointer (via MTL), the parameter σ_o and the “location” determine the input pattern. The state of the neocortical component at any moment of time is regarded as nearly stationary (i.e., variables v_i are considered fast) at the scale of the hippocampal module dynamics. The (asymmetric) synaptic weights w are randomly generated according to the formula where ω_ij are independent random numbers uniformly distributed in the interval (0,1), ν is a constant inhibitory parameter (which stabilizes connectivity), and S is a standard logistic sigmoid function: The firing rate in equation 3 was solved iteratively with values of model parameters n = 300, I = 3, σ = 0.15, σ_o = 0.2, ν =0.1, θ = 3. Given a specific realization of random numbers (produced by Matlab code included in Supplemental online material at http://www.krasnow.gmu.edu/ascoli/sa_lm05_som/), this network has six attractor states. Five of these are characterized by focused activity patterns represented in Figure 5, A–E. The sixth state has a near-zero, almost uniform distribution of activity (v = 0.032 + 0.005) and it can only be reached after a “cold” start.

The implementation of this model corresponds to a fairly standard artificial neural network, with Gaussian input and Gaussian distribution of synaptic weights. The goal of this neocortical component simulation is not to embody a biologically plausible representation of the neocortex, nor to offer a mechanistic insight into its function. Instead, this model is one of the simplest network implementations yielding nontrivial dynamics among stable activity patterns. We take the transitions among these activity patterns to illustrate sequential episodes in the space of memories. Thus, this neocortical network was only used to analyze the role of the hippocampal module in a simple memory retrieval paradigm. Other network models with dynamical activity patterns that could represent episodic memories could serve the same purpose.

Simulation of the hippocampal module as a large two-layer neural network

In order to address the biological plausibility of the hippocampal model for episodic memory retrieval under realistic conditions (e.g., in terms of the size of memory space, number of visited episodes during memory exploration, number of episodes necessary to reach an arbitrary target during retrieval, etc.), we implemented a larger scale hippocampal module model. The network was composed of N CA3 plus N CA1 model neuronal units (N = 11, 20, 30, 50, 100, 200, 300, 500, 1000, 2000, 3000, 5000, 10,000, 20,000, 30,000). Each unit was considered as a group of principal hippocampal cells representing a unique, previously explored abstract context, with one-to-one correspondence between CA3 and CA1 units. Selected random connections between abstract contexts were stored in the CA3-to-CA3 synapses (matrix J). Each CA3 unit was associated with 10 other randomly selected CA3 units (with the uniform probability, but avoiding repetitions). An interpretation of these stored connections is that they result from associative learning during initial experience and during subsequent off-line replays. One epoch of memory exploration was simulated for each afferent connection of each CA3 unit. Each epoch included M steps connecting M + 1 units (M ranging from 1 to 100 in different simulations). Technically, these exploratory epochs were simulated by sequentially activating the stored connections J backward. All CA3 units that were active during a given epoch were associated with the CA1 unit that was active at the end of the epoch, with the weight given by equation 1, where t_j is the time of the last reactivation of unit j during the epoch. Associations across different epochs were not created.

During retrieval, the activity in CA3 was centered on one unit and distributed with the weight 0.2 among all 10 efferent neighbors of the central unit. The resultant profile of the CA3 activity packet is represented in Figure 8A. The corresponding distribution of activity in CA1 (Fig. 8B) was calculated as the distribution of activity in CA3 multiplied by the matrix W. When stated explicitly, a Gaussian, uncorrelated noise was added to the resulting CA1 activity. The standard deviation of noise ranged from 0 to 0.0015, i.e., 2.5% of the peak CA1 activity value (this level of noise is attributed to individual model units, each of which may be related to a large population of neurons). In addition, both CA3 and CA1 activity packets performed PP oscillations in synchrony with each other. The direction of PP was alternated, so that all 10 stored efferent connections were explored at each visited location. As before, a PP oscillation that produced the strongest effect on the goal-related CA1 cell was selected as the next step. If there was no effect, a random efferent connection of the current CA3 unit was selected as the next step. The process of retrieval was terminated when the goal-related CA1 cell became the center of the CA1 activity packet. In each retrieval simulation session, both the starting and the target units were randomly selected anew. The Matlab code used in all simulations is included in Supplemental online material at http://www.krasnow.gmu.edu/ascoli/sa_lm05_som/.

Results

Simulated navigation in a continuous two-dimensional space

We start by describing simulation results for a two-dimensional maze shown in Figure 3. After random exploration of the environment (that starts at the bottom and ends in the upper right corner; Fig. 3A), the model rat is given the task to reach a preselected goal. The typical resulting trajectory steers clear of dead ends and does not includes loops (Fig. 3B). Most importantly, when starting from any of the previously explored locations, the model reaches the goal following a path with length of the same order of that of the shortest possible path.

Figure 3C shows a histogram of 50 trajectory lengths for the learning session represented in Figure 3A, when the starting location is selected at random for each trajectory, and the goal location is fixed. The long tail on the right side is due to the selection of starting locations within the unexplored area (the upper left part in Fig. 3A). In this case, the model rat proceeds randomly until it reaches a previously explored location. The average ratio of the length of the trajectory leading to the goal before and after learning was 60:1.

Simulated solution of the Tower of Hanoi problem

The qualitative results obtained with this model in simulated mazes are not contingent on the geometrical and topological properties of the search space (such as continuity and planarity of the environment, symmetry of connections, etc.) and can be reproduced in nonspatial problem solving. The classic Tower of Hanoi puzzle (analyzed, e.g., in Russell and Norvig 1995) is an example of a nonspatial problem that can be solved by the same algorithm used above for two-dimensional navigation. Figure 4A represents the entire space of states of the three-disc Tower of Hanoi problem.

One hundred independent sessions of the exploration-and-solving paradigm were run. The system had no a priori knowledge of the goal during the exploration phase, in which associations were learned according to equation 1. The exploration time was doubled every 10 cycles. During each of the exploratory epochs (all started with a randomly generated configuration) the goal could be reached 0, 1, or several times. The resulting distribution of the solving time vs. exploration time is presented in Figure 4B. Instead of a gradual decrease in the average solving time, a sharp transition is observed at ∼100 exploratory moves, when the probability that the goal is not reached during the exploration phase becomes negligible. The system always (and immediately) finds a nearly optimal path in the discrete search space, provided that every location is previously explored.

Episodic retrieval involving navigation of memory

Here, we present a simple model description of a realistic example of episodic memory retrieval. A critical element of this example is that the episode to be retrieved requires other episodic knowledge for its understanding. This case is crucially distinct from retrieval of typical episodes. In the retrieval of a typical episode (“What did you have for breakfast last Monday?”), PFC can be used to immediately access facts from semantic memory in MTL that are highly accessible insofar as they have been stamped in by multiple experiences as follows: “What do I normally do on Mondays? I go to the National Institute of Health (NIH). What do I normally have for breakfast? Orange juice. Where do I normally eat breakfast? At a cafeteria.” As a result of this process, the system may produce a rather trivial answer; “I had an orange juice at the NIH cafeteria.” This account, however, may not apply to memories of unique, unusual episodes. Imagine that the above answers are not characteristic of you at all. You do not go to NIH regularly. You eat breakfast at home, and you never drink orange juice in the morning. Yet someone tells you the following: “I saw you last Monday morning at the NIH cafeteria with orange juice!” In order to be able to retrieve and to understand a memory of this episode (assuming that it did take place), you may need to retrieve several related episodes first that contain key episodic (rather than semantic) memory data. Those related key episodes may be involved with completely different semantic features, and in this case, PFC may not be able to find them without the help of the hippocampus (Fig. 5). Our model allows for the following account of this case.

1. At the beginning, PFC tunes to receiving signals from MTL related to specific semantic features of the target episode, namely, “orange juice,”“NIH cafeteria,” and the notion of last Monday. Suppose that pointer E (Fig. 5) is the only perfect match for these features, and it is associated with the right memory. Nevertheless, activating this pointer immediately may not help in retrieval of the episode.

2. Next, PFC attempts to activate any semantic features (schemas) associated with connections from the current context (at the beginning, this is the context of talking to a friend) to other contexts, while “listening” to the background activity in E via the two selected features. Suppose that the strongest response in E at the first step of retrieval is associated with simultaneous activation of two schemas, “morning hurricanes” and “seminars at NIH” (Fig. 5).

3. Therefore, the system activates these features in MTL, thereby causing a transition along the connection from A to B in the hippocampal module. The corresponding alteration of the activity pattern in the neocortical component is induced (Fig. 6); as the pointer B becomes active, the episode B is retrieved in the neocortical component, corresponding to the arrival to NIH for a morning seminar, when a strong hurricane is beginning.

4. The pattern of action (2 and 3) is repeated (e.g., the next retrieved key episode would be the moment of power and water outage at NIH), until the system reaches the target pointer E in the hippocampal module and the associated target episode E in the neocortical component. This is the moment of ordering orange juice at the NIH cafeteria, because no other drink is available. The resultant path (fat gray lines inside the hippocampal module, Fig. 5) is shorter than the original sequence of experience (the chain of thin black arrows outside of the hippocampal module, Fig. 5).

In order to illustrate the possible role of the hippocampus in the above process, we apply another instance of the the hippocampal module network (Fig. 2) to this problem. The role of the environment in this case is played by the attractor network model of the neocortical component. In this model, neuronal units are randomly allocated on a plane, with potentially all-to-all connections limited by a Gaussian function of the distance (see Materials and Methods section above). In the particular realization of the network we used, five attractor states are compact patterns of activity (Fig. 6), which can be taken to correspond to episodes A–E in Figure 5. The model neocortical network was simulated in parallel with the original hippocampus model (Fig. 2B), described by equation 1, and taken in the form of the graph represented in Figure 5 (hippocampal module). The output from an active hippocampal module unit was distributed among inputs of the neocortical component units using a Gaussian function centered at a location (a, b, c, d, or e) associated with the the hippocampal module unit. During an interval between stimulations, the neocortical component relaxed to an attractor state. Thus, possible control actions over the neocortical component were limited to activation of one (or none) of the “pointers” that project to neighborhoods a, b, c, d, and e in Figure 6, while the role of the hippocampal module and related structures (Fig. 2) was to find the right sequence of activation of the neocortical component.

The neocortical network's dynamics were not governed by geometrical distances between attractor centers. Transitions between well-separated, distant patterns (e.g., from E to C) were readily induced by the stimulus. At the same time, some transitions between nearby overlapping patterns did not occur. For example, when the network was in state A, and the stimulus was applied at the label e, attractor state A was slightly distorted (Fig. 6A1), but it did not jump to the pattern E. All connections in the model network (i.e., transitions that can be caused by a stimulus applied at a, b, c, d, or e) are represented by thin black arrows connecting pointers A–E inside of the hippocampal component in Figure 5. In particular, the network can be led to E from A via B and D, but not directly from A to E. This result illustrates the idea that in some neuronal networks, contrary to the traditional theory going back to Teyler and Discenna (1985, 1986), a premature reactivation of a hippocampal pointer associated with the target episode may not cause retrieval of the episodic memory, while at the same time reactivation of a connected path successfully results in retrieval.

During one selected typical training session in simulations of this example (captured in the movie “exploration” included in Supplemental online material at http://www.krasnow.gmu.edu/ascoli/sa_lm05_som/), the system randomly explored possible transitions between pointers in the hippocampal module. The system followed (and stored in CA3-to-CA3 synapses J) only those transitions that successfully induced the correspondent pattern switching in the neocortical component. In parallel, the matrix W was modified according to equation 1. This memory exploration phase lasted for 10 steps. The resultant matrices J and W are represented in Figure 7 by the thickness and the style of lines (see Fig. 7 caption). The reader can easily reconstruct the path from A to E that was selected by the model in the retrieval phase, when the pointer E was given as a goal (see the movie “retrieval” included in Supplemental online material at http://www.krasnow.gmu.edu/ascoli/sa_lm05_som/). To reconstruct the path, the reader may follow at each step a solid CA3-to-CA3 arrow leading to a CA3 node with the strongest connection to CA1 node E (this path is A → B → D → E).

The function of this rather simple example simulation is to illustrate the possible involvement of the hippocampal network in the retrieval of an unusual episodic memory, as in the example presented in Figure 5. In particular, the attractor network model of the neocortical component allows the implementation of episodic memory dynamics in parallel with the hippocampal network. The particular choice of the neocortical component model does not affect the plausibility of the hippocampal module. Nevertheless, several critical questions remain open regarding the applicability of the hippocampal network model in conditions closer to real episodic memory retrieval. How does the model perform in the case of a large, realistic memory load? How many sequential episodes in this case should be associated with a CA1 unit in a sharp wave for the model to succeed in retrieval? Can the model remain efficient, if more realistically spread distributions of activity are used in CA3 and in CA1 instead of a single winner-take-all unit? Finally, how robust is the model performance with respect to the intrinsic noise that is due to the random and discrete nature of neuronal spikes?

Episodic memory retrieval with a substantially large memory load

In order to address the above questions, we increased the size of the hippocampal module network up to 30,000 units representing abstract contexts. To the best of our knowledge, the number of distinct contexts remembered by a typical, normal adult human has not yet been reliably estimated. At the same time, assuming that the pointers to those contexts are stored in the hippocampus, one may come up with an estimate based on the anatomical data (see Connectivity in Cornu Ammonis). CA3 network is frequently considered as an autoassociative attractor network. In many theoretical models of autoassociative attractor neural networks (Amit 1989; Hertz et al. 1991) the decimal order of the maximal number of stored patterns can be roughly estimated as the decimal order of the number of synapses per neuron. Based on this logic, we take the number 10⁴ as a “realistic” estimate of a typical episodic memory load. While this number intuitively appears low, most long-term memories may be of semantic nature, and therefore, their retrieval may not primarily depend on the hippocampus.

In this simulation, we implemented the hippocampal module as a large, randomly generated two-layer network with the total number of neuronal units up to 6 × 10⁴ and the total number of modified synapses up to 3 × 10⁶ (the corresponding maximal total number of synapses was of the order of 10⁹). The neocortical component was not explicitly simulated in this case. Instead, random connections were generated (and stored in J) during training sessions. We used a more realistic model activity distribution in CA3 (Fig. 8A), which resulted in a more realistic activity packet shape in CA1 (Fig. 8B). For each randomly generated and explored network, 10⁴ retrieval sessions were run, with a random selection of both the starting and the goal locations in each session. Among the varied parameters of the model were the number of episodes (or abstract concepts) associated with CA1 units in sharp waves, and the level of dynamical noise in CA1. Results are presented in Figure 8. The main finding is the robust ability of this model system to find a nearly optimal path toward a given target.

For example, consider the memory load N = 10⁴, with n = 10 efferent connections per hippocampal module unit, with M = 5 exploration steps of sequential episodes to be associated with a CA1 unit, in the absence of noise. In these realistic conditions, the retrieval path length between two randomly selected units (calculated over 10⁴ retrieval sessions, each with random start and goal selection) is 6.75 ± 3.10sd, while the shortest path length (calculated in parallel for the same set of cases) is 4.22 ± 0.68. The ratio of the mean values is 1.6, and the second interval is almost entirely contained within the first. The histogram of retrieval path lengths at these values of parameters is compact (Fig. 8C), with all of the 10⁴ trajectories shorter than 60 steps, and 99.5% of them shorter than 20 steps. To compare, a typical random trajectory connecting two randomly selected units in this model network consists of the order of 7000 steps. This estimate also holds for the case when the two selected units coincide, meaning that a random traveler in this huge space would be lost practically forever after the very first step. Nevertheless, our hippocampal network model was never lost.

This result is robust with respect to alteration of many model parameters within and beyond their realistic range (Fig. 8). For example, introducing Gaussian dynamic noise at the level of 2.5% of the peak CA1 activity only increases the mean retrieval path length by 9%. Doubling the number of units N increases the mean retrieval path length by 30% (while at the same time, the mean shortest path length increases by 7%). Shortening the number of associated episodes M from five to two increases the mean retrieval path length by 11% (Fig. 7D). Therefore, unlike the case of spatial navigation of large two-dimensional environments, this paradigm of memory retrieval does not require long exploration epochs to be associated with potential target units. On the contrary, association of very short epochs may suffice for a reasonably good performance. At the same time, increasing M up to 100 and beyond does not significantly improve the performance of the model.

Discussion

The last simulation constitutes a proof-by-example of the viability of the proposed concept in the case of an abstract model system with a realistically large memory load. Can the same phenomenon be demonstrated in a network with more realistic architecture and dynamics? One way to improve the model is by using a more precise description of the distribution of activity in CA3 and in CA1. For example, extending the rapidly descending tails of the activity packet in CA3 (Fig. 7B) up to the next-to-nearest neighbors of the central unit should improve the performance of the system, in particular, in the case of a very large network (10⁵ units and above). This idea is consistent with the broad spatial tuning of place cells in CA3 (in addition to CA1). Another interesting possible modification of the model concerns the architecture of the random network. Here, we assumed that the probability of connection is uniformly distributed among units (with the only constraint that there are 10 efferent connections for each unit). Instead, a subset of units that are connected to the same parent could have a higher probability to be connected to each other than to any randomly selected unit. This bias, if present, would result in a certain degree of clustering in the network, consistent with the idea of cell assemblies (Harris et al. 2003), as well as with the hierarchical organization of remembered contexts.

Understanding hippocampal function in episodic memory retrieval

According to the view of the hippocampus as a context-indexing device (e.g., Wheeler et al. 1997), all real or imagined general contexts of experiences (rather than experiences per se) are represented by distinct hippocampal activity states. Neural activity in each of these states is focused on a relatively small, uniquely selected subpopulation of principal cells. Each state of this sort could possess certain (meta-) stability typical of a latent attractor state (Samsonovich and McNaughton 1997; Doboli et al. 2000). The most commonly held account of the neural basis of contextual reinstatement is that PFC is used to access information in semantic memory (Shimamura 1995; Moscovitch 1992; Dobbins et al. 2002) that is stored in MTL, and not in the hippocampus (Vargha-Khadem et al. 1997) nor in PFC (e.g., Sylvester and Shimamura 2002). PFC then maintains this information in working memory and uses it to cue the hippocampus. The goal of the contextual reinstatement process would be to provide enough details that match the hippocampal memory trace, so that it is triggered. Specific information retrieved from the hippocampus can be used to further refine the retrieval cue (e.g., Howard and Kahana 2002). Finally, according to this view, when the correct hippocampal pointer is reactivated, the entire episodic memory content, distributed across modalities in the neocortical component, is retrieved. This traditional view makes two essential (if implicit) assumptions, that we choose to drop.

1. All contextual information necessary for understanding an episode is associated with the memory trace of experience (possibly, via the hippocampal pointer).