The ANCCR (Jeong et al., 2022) model, which stands for adjusted net contingency for causal relations, proposes that mesolimbic dopaminergic conveys an adjusted net contingency for causal relationships (to biologically meaningful targets). The mathematics (and logic) behind the model go well beyond what I can cover here, but for now, it will suffice to say that the model:
The degree to which a stimulus \(i\) at time \(t\) is “active” in memory is denoted by:
\[ \tag{Eq.1} E_i(t) = \Sigma_{t_i \leq t}e^{-(\frac{t-t_i}{t\_constant})} \]
where \(t_i\) are all the time steps up to time \(t\), and \(t\_constant\) is a time constant (usually meant to be the inter-reward rate)1
The model learns retrospective associations after meaningful causal targets occur. Whether event \(j\) is a meaningful causal target is given by:
\[ \tag{Eq.2} \Phi_j = \begin{cases} 1,& \text{if } \Phi_j(t) = 1\\ 1,& \text{if }DA_j + \beta_j > \theta\\ 0,& \text{otherwise} \end{cases} \]
where \(\Phi\) plays the role of an indicator function, \(DA_j\) is the total dopamine activity at the time of event \(j\), \(\beta_j\) is the unconditioned value of event \(j\) and \(\theta\) is a global threshold parameter.2 Note that the indicator function is self-preserving: once a stimulus becomes a meaningful causal target, it does not stop being so.
After stimulus \(j\) is observed, the predecessor representation contingency, or PRC, for each stimulus \(i\) is updated via:
\[ \tag{Eq.3} PRC_{i \leftarrow j} = M_{i \leftarrow j} - M_{i} \]
where \(M_{i \leftarrow j}\) is the predecessor representation of \(i\) given \(j\) has occurred, and \(M_{i}\) is the base rate with which \(i\) occurs. Both of these quantities are given by:
\[ \tag{Eq.4a} M_{i \leftarrow j} = M_{i \leftarrow j}' + \Phi_j\alpha(E_{i \leftarrow j} - M_{i \leftarrow j}') \]
and
\[ \tag{Eq.4b} M_{i} = M_{i}' + k\alpha(E_{i} - M_{i}') \]
where \(M_{i \leftarrow j}'\) and \(M_{i}'\) are the quantities before \(j\) was observed, \(k\) and \(\alpha\) are learning rate parameters, and \(E_{i \leftarrow j}\) is the eligibility trace of stimulus \(i\) at the time \(j\) occurs (see Eq. 1).
Then, the PRC can be used to derive the prospective association, aptly named the successor representation contingency, or SRC via Bayes rule:
\[ \tag{Eq.5} SRC_{i \rightarrow j} = PRC_{i \leftarrow j} \frac{M_j}{M_i} \]
The base rate for \(j\), \(M_j\) is calculated via Eq.4b.
The model postulates that dopaminergic signaling encodes the adjusted net contingencies for causal relations between stimuli, or ANCCRs. The total dopaminergic activity at the time of event \(i\) is equal to:
\[ \tag{Eq.6} DA_i = \Sigma_j (ANCCR_{i \rightarrow j}\Phi_j) \]
And the ANCCR from stimulus \(i\) to stimulus \(j\) is given by:
\[ \tag{Eq.7} ANCCR_{i \rightarrow j} = NC_{i \leftrightarrow j} CW_{i \rightarrow j} - \sum_{k \neq i}(ANCCR_{k \leftrightarrow j}\Delta_{k \leftarrow i}\Phi_{k \leftrightarrow i}) \]
where \(NC_{i \leftrightarrow j}\) is the net contingency between stimuli \(i\) and \(j\), \(CW_{i \rightarrow j}\) is the causal weight that \(i\) has with \(j\), \(\Delta_{k \leftarrow i}\) is the recency of stimulus \(k\) with respect to stimulus \(i\), and \(\Phi_{k \leftrightarrow i}\) is an indicator function denoting whether \(k\) and \(i\) have a putative causal relationship with each other.
The net contingency between stimuli \(i\) and \(j\), \(NC_{i \leftrightarrow j}\), is given by:
\[ \tag{Eq.8} NC_{i \leftrightarrow j} = wSRC_{i \rightarrow j} + (1-w)PRC_{i \leftarrow j} \]
or a weighted sum of successor and predecessor representation contingencies.
The net contingency is used to calculate the indicator function above, as:
\[ \tag{Eq.9} \Phi_{k \leftrightarrow i} = \begin{cases} 1,& \text{if } NC_{i \leftrightarrow j} > \theta\\ 0,& \text{otherwise} \end{cases} \]
where \(\theta\) is the same threshold parameter used in Eq.23, and the indicator function for a stimulus and itself, \(\Phi_{i \leftrightarrow i}\), is 0.
The recency term, \(\Delta_{k \leftarrow i}\), is given by:
\[ \tag{Eq.10} \Delta_{k \leftarrow i} = e^{-(\frac{t_j-t_i}{t\_constant})} \]
where \(t\_constant\) is the same parameter used in Eq.1. Note however that Eq.9 does not include the sum term in Eq. 1. Finally, the causal weight from stimulus \(i\) to stimulus \(j\) is given by:
\[ \tag{Eq.11} CW_{i \rightarrow j} = CW_{i \rightarrow j}' + \alpha_{reward}\delta_{i \rightarrow j} \]
where \(CW_{i \rightarrow j}'\) is the previous causal weight, \(\alpha_{reward}\) is a learning rate parameter exclusive for causal weights, and \(\delta_{i \rightarrow j}\) is a delta term depending on the sign of the total dopaminergic activity, given by:
\[ \tag{Eq.12} \delta_{i \rightarrow j} = \begin{cases} CW_{j \rightarrow j} - CW_{i \rightarrow j}, & \text{if } DA_j \ge 0\\ (0-CW_{i \rightarrow j})\frac{n_i^{-1}\Delta{i \leftarrow j} \Phi_{i \leftrightarrow j}}{\Sigma_{k \neq j}(n_k^{-1}\Delta_{k \leftarrow j} \Phi_{k \leftrightarrow j})},& \text{otherwise} \end{cases} \]
where \(CW_{j \rightarrow j}\) above is the reward magnitude of stimulus \(j\). In plain words, when dopaminergic activity is positive, causal weights (from all present and absent stimuli) strengthen. Conversely, when dopaminergic activity is negative, causal weights (from all present and absent stimuli) weaken, proportional to their normalized frequency and recency (as long as they have putative causal relations with \(j\)).
Responding in ANCCR is lightly specified. The value of responding upon presentation of stimulus \(i\) is given by:
\[ \tag{Eq.13} Q_i = \Sigma_k(SRC_{i \rightarrow k} CW_{i \rightarrow k}) \]
which can then be mapped onto probabilities via a softmax function4.
The diagram below shows the dependencies in the model. I am excluding the indicator functions and parameters for simplicity.5
The implementation of this model is a port from the MATLAB code that
Jeong et al. shared in the GitHub repository
associated with their paper. The output of the R model was checked
against the outputs of the MATLAB model, using training routines
(“eventlogs” in their parlance) generated using their MATLAB code. The
training routines generated in calmr differ somewhat, to
accommodate generality. For example, as of version 0.6.1,
it is not possible to specify probabilistic relations between cues and
rewards. Instead, it is left to the user to specify an exact probability
via trial numbers (e.g., an 80% reward probability can be specified as
“80A>(US)/20A”). The naming of parameters also differs between
codebases.
The \(t\_constant\)
defaults to NA, and if it remains like that, is calculated
internally as the mean of the mean ITI for each trial. You should set
this by hand if only some of your trials contain rewards.↩︎
The first case in Eq. 2 was not written in the supplementary information of the Jeong et al. paper, but it reflects their MATLAB implementation.↩︎
Notably, although Eq.2 is self-perpetuating, Eq.9 is not, meaning causal relations between stimuli can flip between 0 and 1↩︎
The implementation of ANCCR in calmr
returns both action values and probability matrices. The latter is
parametrized via the global parameters \(cost\) (the cost of responding) and \(temperature\), a multiplier, not
divider on action values↩︎
If nothing appears below, try rebuilding this vignette
after install DiagrammeR↩︎