A @phase
block¶
The environment that the subject interacts with is specified using one or more @phase
blocks.
A @phase
block defines the sequence of stimuli that the subject is exposed to, and how the next stimulus
in the sequence depends on the response to the previous stimulus.
If the environment is an experiment setup, say a maze, one may use one @phase
block per phase in the experiment,
for example one @phase
for the training phase and one for the test phase.
The structure of a phase block is outlined in the below example:
@phase training stop: reward==100
TRIAL_START lever | pull_lever: REWARD | NO_REWARD
REWARD reward | TRIAL_START
NO_REWARD background | TRIAL_START
The simulation will start with the first phase line below the title line (except for inherited phase blocks, see Inheriting a phase).
The @phase
title line¶
In general, a @phase
block starts with a title line of the format:
@phase phase_name stop:stop_condition
where
phase_name
is the name of the phase (see The phase name), andstop_condition
is the stop condition (see The phase stop condition) specifying when to exit the phase, usually after a prededermined number of exposures to a certain stimulus element, or after a predetermined number of responses with a certain behavior.
Note
@phase
and stop:
are keywords, and cannot be changed. The name of the phase (phase_name
) and the stop condition (stop_condition
) are user defined.
For example, in a script with the following parameter definitions:
stimulus_elements = lever, reward
behaviors = pull_lever, ignore
the title line of a @phase block may look like this:
@phase training stop:reward==100
or:
@phase training stop:pull_lever==100
The phase name¶
The phase name must be a valid name.
The phase stop condition¶
A phase stop condition is a boolean expression that may include the following variables:
Any stimulus element (evaluates to the number of exposures to this stimulus element)
Any behavior (evaluates to the number of responses with this behavior)
Any phase line label (evaluates to the number of times this phase line has been visited)
Any local phase variable (evaluates to the value of this variable)
For example, the first line of a @phase block may look like this:
@phase training stop: pull_lever>=10**2 or reward==10
which means that the phase stops when either
the number of responses with the behavior
pull_lever
is greater than or equal to \(10^2=100\).the number of exposures to the stimulus element
reward
is 100, or
Boolean expression¶
A boolean expression is a mathematical expression that evaluates to true or false. It may include number literals, variables, parentheses, and the below operators.
The supported operators in Learning Simulator are the arithmetic operators
+
(addition)-
(subtraction)*
(multiplication)/
(division)**
(power)
and the comparison operators
==
(equals)>
(greater than)>=
(greater than or equal to)<
(less than)<=
(less than or equal to)
and the logical operators
and
or
and the functions
For example, x>2 and y<=5
is a boolean expression of the
variables x
and y
, the constants 2 and 5, and
the operators >
and <=
. It evaluates to true
whenever x
is greater than two and y
is smaller
than or equal to 5, and false otherwise.
If x
is a number, the expression 2x-5
is not a boolean expression since it
does not evaluate to true or false, but to a number.
A phase line¶
Below the phase title line follows a number of phase lines. Each phase line
presents a stimulus to the subject (optional),
performs a number of (optional) actions (see Actions), and
informs which of the phase lines to go to next
A phase line has the general format:
LINE_LABEL s | d11,d12,...,COND1:a11,a12,... | d21,d22,...,COND2: a21,a22,... | an1,an2,...,ann
where s is a stimulus, d11
, d12
, a11
, a12
etc. are actions, and COND1
, COND2
etc. are phase line conditions (see Phase line conditions).
The actions are performed sequentially, in the order they appear. For example, the line:
LINE_LABEL s | d11,d12, COND1:a11,a12 | d21,d22, COND2:a21,a22 | d31,d32, COND3:a31,a32 | a41,a42
is interpreted by the simulator as follows:
Present the stimulus s
Perform actions d11 and d12
If COND1 is fulfilled:
Perform action a11
Perform action a12
Otherwise:
Perform action d21
Perform action d22
If COND2 is fulfilled:
Perform action a21
Perform action a22
Otherwise:
Perform action d31
Perform action d32
If COND3 is fulfilled:
Perform action a31
Perform action a32
Otherwise:
Perform action a41
Perform action a42
Note that one of the action types is “go to phase line with label X”.
During the simulation, if the above logic does not determine which phase line to go to next, an error message will appear.
For example, if COND1
is fulfilled, a12
cannot be any other action than a phase line label:
LINE_LABEL1 stimulus | d11,d12, COND1:a11,LINE_LABEL2
LINE_LABEL2 ...
Since the actions are performed sequentially, an error will appear if the action LINE_LABEL2
is followed by additional actions
as these additional actions will never be performed.
Phase line conditions¶
A phase line condition is a boolean expression that may depend on the following variables:
Any behavior (evaluates to true for the behavior that is the response to the stimulus on the line, and false for all other behaviors)
Any phase line label (evaluates to the number of times this phase line has been visited)
Any local phase variable (evaluates to the value of this variable)
and the following functions:
count
(See The function count)count_line
(See The function count_line)
The function count
¶
The function count
counts occurrences of each behavior, each stimulus element, and
number of visits to each phase line (using the phase line label as identifier). For example:
count(lever)
evaluates to the number of times the stimulus element lever
has been exposed
to the subject (since the beginning of the phase, or since it was last reset
using count_reset
, see below). Similarly,:
count(pull_lever)
evaluates to the number of times the agent has responded with the behavior pull_lever
(to any stimulus) since the beginning of the phase, or since it was last reset.
To return the number of visits to a phase line with label LBL
, use:
count(LBL)
The counter for a specific stimulus element, behavior, or phase line may be
reset to zero using the action count_reset
(see The action count_reset).
The function count_line
¶
The function count_line
counts consecutive occurrences of each
behavior, each stimulus element, and number of consecutive visits
to the phase line where this function is used. It is perhaps most useful for behaviors, to count
the number of consecutive responses to the stimulus on a phase line. For example:
stimulus_element = s1, s2
behaviors = b1, b2, b3
@phase ...
LBL1 s1 | count_line(b1)==3: LBL2 | LBL1
LBL2 s2 | LBL1
will expose the subject to the stimulus s1
repeatedly until the response
to s1
is the behavior b
three times in a row. Then s2
will be
presented. A sequence of stimulus-response pairs may be:
s1 -> b1 -> s1 -> b2 -> s1 -> b1 -> b1 -> b1 -> s2 -> b3
If instead the phase lines looked like this:
LBL1 s1 | count_line(s1)==3: LBL2 | LBL1
LBL2 s2 | LBL1
the subject will be presented with s1
three times, and regardless of
the responses, will then be presented with s2
:
s1 -> * -> s1 -> * -> s1 -> * -> s2
Note that the above example is equivalent to:
LBL1 s1 | count_line(LBL1)==3: LBL2 | LBL1
LBL2 s2 | LBL1
and that count_line(LBL1)
may be abbreviated count_line()
as the
line label is redundant (since count_line
count consecutive occurrences
on the very line where it is used):
LBL1 s1 | count_line()==3: LBL2 | LBL1
LBL2 s2 | LBL1
Local phase variables¶
It is possible to use custom local variables within a phase, whose values may change during the phase. They may be assigned a value in an action, and may be referred to in a condition. Local phase variables must be initialized (have an assigned value) before they are used. A phase variable must have a valid name.
An action that assigns the value 0.1 to the local variable x
looks as follows:
x=0.1
A phase with an action that sets x
and with a
condition that uses x
may look like this:
@phase ...
L0 x=0.1 | L1 # Set x to 0.1 and go to L1
L1 s1 | x=x+0.1, x>=0.5: L2 | L1 # Present s1, increase x with 0.1. If x>=0.5 go to L2, otherwise L1
L2 s2 | L1 # Present s2 and go to L1
Using local variables in a phase stop condition¶
It is possible to use a local phase variable in a phase stop condition. For example:
@phase phase_name stop:x>2
L0 x=0.1 | L1
L1 s1 | x=x+0.1, x>=0.5: L2 | L1
L2 s2 | L1
The stop condition is not checked until the first stimulus is presented. If the phase stop condition depends on local variables, these local variables must therefore be initialized before the first stimulus is presented (like in the example above), otherwise an “Unknown variable” error will occur.
Actions¶
An action is one of the following:
Set a local phase variable (for example,
x=2
)Go to a phase line (by simply stating the corresponding phase line label)
Go to one of several phase lines, with a probability for each (for example,
L1(0.25),L2(0.75)
). See Probabilistic go-tos.Tell the simulation to omit the state variable update in the next step with
@omit_learn
. (See The action @omit_learn.)Reset the counting of an event with
count_reset
(See The action count_reset.)
Note that global variables (defined using @variables
) cannot be set in an action.
Probabilistic go-tos¶
A probabilistic go-to is an action of the type “go to line L1 with probability p1, go to line L2 with probability p2, etc.” and has the syntax:
L1(p1),L2(p2),...
where p1
, p2
etc. are expressions of global variables, local phase variables and numbers.
For example, to go to line L1
with probability 0.4 and to L2
with probability 0.6, use:
L1(0.4),L2(0.6)
Note that the probabilities may be global variables or local phase variables. For example:
START | p1=0.1, p2=0.7, p3=0.2 | L0
L0 s | L1(p1),L2(p2),L3(p3)
L1 ...
L2 ...
L3 ...
The probabilities within parentheses do not need to add up to 1:
L0 s | L1(0.1),L2(0.7) | L3
means that the probability to go to L3
is 0.2, so this is the same as:
L0 s | L1(0.1),L2(0.7),L3(0.2)
The action @omit_learn
¶
The action @omit_learn
tells the simulator to omit the updating of the subject’s state
variables in the next step. Typically this is done at the end of a trial, when the
phase represents a number of trials. In that case, it is often not desired to carry
over the reinforcement value of, say, a reward at the and of a trial to the
first stimulus in the next.
For example, in mechanisms with conditioned reinforcement values (w-values), and we have the following stimulus-response history:
s -> response -> reward -> eat -> s
one typically wants to avoid that the w-value for the first stimulus in the
trial (s
) affects
the reinforcement value of
reward
, andthe v-value (associative strength) between
reward
andeat
The action count_reset
¶
The counter for a specific stimulus element, behavior, or phase line may be
reset to zero using the action count_reset
.
For example:
count_reset(pull)
resets the counter of pull
to zero, so that count(pull)
will return zero. This action may be used like this:
stimulus_elements = lever, reward
behaviors = pull, ignore
@phase training stop:reward=100
START | count_reset(pull), LEVER_OFF
LEVER_OFF lever | count(pull)=3: LEVER_ON | LEVER_OFF
LEVER_ON lever | pull: REWARD | LEVER_ON
BACKGROUND background | START
REWARD reward | START
After three pull
of the lever
, the lever will “activate” and give reward when pulled.
Then it is inactivated and three more pull
are required to activate it.
To reset the counter for the stimulus element e1
, use:
count_reset(e1)
To reset the counter for the phase line with label LBL
, use:
count_reset(LBL)
Note that resetting one stimulus element (or behavior or phase line) will not affect any other.
Inheriting a phase¶
If you have several @phase-blocks that are very similar to each other, there is a way to avoid
duplicating each phase line, and instead only state the lines that differ between the phases.
This is done through phase inheritance. Do create a phase with name phase2
that inherits
from from a phase with label phase1
, use:
@phase phase2(phase1) ...
For example, suppose we have the following phase block in a script:
@PHASE training stop: stimulus=50
new_trial stimulus | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD
REWARD1 reward1 | new_trial
REWARD2 reward2 | new_trial
REWARD3 reward3 | new_trial
NO_REWARD background | new_trial
and we want to create a new phase training2
that is the same as the above but where the line with label REWARD3
should read:
REWARD3 reward4 | new_trial
This can be accomplished by simply re-stating the entire phase block training
, and make the change. However, this requires
a lot of lines and a change in a part of the phase block that should be the same must be performed in each
phase block. Therefore, using phase inheritance is better:
@PHASE training2(training) stop: stimulus=50
REWARD3 reward4 | new_trial
The above phase block represents the same phase as:
@PHASE training2 stop: stimulus=50
new_trial stimulus | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD
REWARD1 reward1 | new_trial
REWARD2 reward2 | new_trial
REWARD3 reward4 | new_trial
NO_REWARD background | new_trial