A @phase block

The environment that the subject interacts with is specified using one or more @phase blocks. A @phase block defines the sequence of stimuli that the subject is exposed to, and how the next stimulus in the sequence depends on the response to the previous stimulus.

If the environment is an experiment setup, say a maze, one may use one @phase block per phase in the experiment, for example one @phase for the training phase and one for the test phase.

The structure of a phase block is outlined in the below example:

@phase training stop: reward==100
TRIAL_START  lever      | pull_lever: REWARD | NO_REWARD
REWARD       reward     | TRIAL_START
NO_REWARD    background | TRIAL_START

The simulation will start with the first phase line below the title line (except for inherited phase blocks, see Inheriting a phase).

The @phase title line

In general, a @phase block starts with a title line of the format:

@phase phase_name stop:stop_condition

where

  • phase_name is the name of the phase (see The phase name), and

  • stop_condition is the stop condition (see The phase stop condition) specifying when to exit the phase, usually after a prededermined number of exposures to a certain stimulus element, or after a predetermined number of responses with a certain behavior.

Note

@phase and stop: are keywords, and cannot be changed. The name of the phase (phase_name) and the stop condition (stop_condition) are user defined.

For example, in a script with the following parameter definitions:

stimulus_elements = lever, reward
behaviors = pull_lever, ignore

the title line of a @phase block may look like this:

@phase training stop:reward==100

or:

@phase training stop:pull_lever==100

The phase name

The phase name must be a valid name.

The phase stop condition

A phase stop condition is a boolean expression that may include the following variables:

  • Any stimulus element (evaluates to the number of exposures to this stimulus element)

  • Any behavior (evaluates to the number of responses with this behavior)

  • Any phase line label (evaluates to the number of times this phase line has been visited)

  • Any local phase variable (evaluates to the value of this variable)

For example, the first line of a @phase block may look like this:

@phase training stop: pull_lever>=10**2 or reward==10

which means that the phase stops when either

  • the number of responses with the behavior pull_lever is greater than or equal to \(10^2=100\).

  • the number of exposures to the stimulus element reward is 100, or

See also Using local variables in a phase stop condition.

Boolean expression

A boolean expression is a mathematical expression that evaluates to true or false. It may include number literals, variables, parentheses, and the below operators.

The supported operators in Learning Simulator are the arithmetic operators

  • + (addition)

  • - (subtraction)

  • * (multiplication)

  • / (division)

  • ** (power)

and the comparison operators

  • == (equals)

  • > (greater than)

  • >= (greater than or equal to)

  • < (less than)

  • <= (less than or equal to)

and the logical operators

  • and

  • or

and the functions

For example, x>2 and y<=5 is a boolean expression of the variables x and y, the constants 2 and 5, and the operators > and <=. It evaluates to true whenever x is greater than two and y is smaller than or equal to 5, and false otherwise.

If x is a number, the expression 2x-5 is not a boolean expression since it does not evaluate to true or false, but to a number.

A phase line

Below the phase title line follows a number of phase lines. Each phase line

  • presents a stimulus to the subject (optional),

  • performs a number of (optional) actions (see Actions), and

  • informs which of the phase lines to go to next

A phase line has the general format:

LINE_LABEL s | d11,d12,...,COND1:a11,a12,... | d21,d22,...,COND2: a21,a22,... | an1,an2,...,ann

where s is a stimulus, d11, d12, a11, a12 etc. are actions, and COND1, COND2 etc. are phase line conditions (see Phase line conditions). The actions are performed sequentially, in the order they appear. For example, the line:

LINE_LABEL s | d11,d12, COND1:a11,a12 | d21,d22, COND2:a21,a22 | d31,d32, COND3:a31,a32 | a41,a42

is interpreted by the simulator as follows:

Present the stimulus s
Perform actions d11 and d12
If COND1 is fulfilled:
    Perform action a11
    Perform action a12
Otherwise:
    Perform action d21
    Perform action d22
    If COND2 is fulfilled:
        Perform action a21
        Perform action a22
    Otherwise:
        Perform action d31
        Perform action d32
        If COND3 is fulfilled:
            Perform action a31
            Perform action a32
        Otherwise:
            Perform action a41
            Perform action a42

Note that one of the action types is “go to phase line with label X”. During the simulation, if the above logic does not determine which phase line to go to next, an error message will appear. For example, if COND1 is fulfilled, a12 cannot be any other action than a phase line label:

LINE_LABEL1 stimulus | d11,d12, COND1:a11,LINE_LABEL2
LINE_LABEL2 ...

Since the actions are performed sequentially, an error will appear if the action LINE_LABEL2 is followed by additional actions as these additional actions will never be performed.

Phase line conditions

A phase line condition is a boolean expression that may depend on the following variables:

  • Any behavior (evaluates to true for the behavior that is the response to the stimulus on the line, and false for all other behaviors)

  • Any phase line label (evaluates to the number of times this phase line has been visited)

  • Any local phase variable (evaluates to the value of this variable)

and the following functions:

The function count

The function count counts occurrences of each behavior, each stimulus element, and number of visits to each phase line (using the phase line label as identifier). For example:

count(lever)

evaluates to the number of times the stimulus element lever has been exposed to the subject (since the beginning of the phase, or since it was last reset using count_reset, see below). Similarly,:

count(pull_lever)

evaluates to the number of times the agent has responded with the behavior pull_lever (to any stimulus) since the beginning of the phase, or since it was last reset.

To return the number of visits to a phase line with label LBL, use:

count(LBL)

The counter for a specific stimulus element, behavior, or phase line may be reset to zero using the action count_reset (see The action count_reset).

The function count_line

The function count_line counts consecutive occurrences of each behavior, each stimulus element, and number of consecutive visits to the phase line where this function is used. It is perhaps most useful for behaviors, to count the number of consecutive responses to the stimulus on a phase line. For example:

stimulus_element = s1, s2
behaviors = b1, b2, b3

@phase ...
LBL1 s1 | count_line(b1)==3: LBL2 | LBL1
LBL2 s2 | LBL1

will expose the subject to the stimulus s1 repeatedly until the response to s1 is the behavior b three times in a row. Then s2 will be presented. A sequence of stimulus-response pairs may be:

s1 -> b1 -> s1 -> b2 -> s1 -> b1 -> b1 -> b1 -> s2 -> b3

If instead the phase lines looked like this:

LBL1 s1 | count_line(s1)==3: LBL2 | LBL1
LBL2 s2 | LBL1

the subject will be presented with s1 three times, and regardless of the responses, will then be presented with s2:

s1 -> * -> s1 -> * -> s1 -> * -> s2

Note that the above example is equivalent to:

LBL1 s1 | count_line(LBL1)==3: LBL2 | LBL1
LBL2 s2 | LBL1

and that count_line(LBL1) may be abbreviated count_line() as the line label is redundant (since count_line count consecutive occurrences on the very line where it is used):

LBL1 s1 | count_line()==3: LBL2 | LBL1
LBL2 s2 | LBL1

Local phase variables

It is possible to use custom local variables within a phase, whose values may change during the phase. They may be assigned a value in an action, and may be referred to in a condition. Local phase variables must be initialized (have an assigned value) before they are used. A phase variable must have a valid name.

An action that assigns the value 0.1 to the local variable x looks as follows:

x=0.1

A phase with an action that sets x and with a condition that uses x may look like this:

@phase ...
L0 x=0.1 | L1                        # Set x to 0.1 and go to L1
L1 s1    | x=x+0.1, x>=0.5: L2 | L1  # Present s1, increase x with 0.1. If x>=0.5 go to L2, otherwise L1
L2 s2    | L1                        # Present s2 and go to L1

Using local variables in a phase stop condition

It is possible to use a local phase variable in a phase stop condition. For example:

@phase phase_name stop:x>2
L0 x=0.1 | L1
L1 s1    | x=x+0.1, x>=0.5: L2 | L1
L2 s2    | L1

The stop condition is not checked until the first stimulus is presented. If the phase stop condition depends on local variables, these local variables must therefore be initialized before the first stimulus is presented (like in the example above), otherwise an “Unknown variable” error will occur.

Actions

An action is one of the following:

  • Set a local phase variable (for example, x=2)

  • Go to a phase line (by simply stating the corresponding phase line label)

  • Go to one of several phase lines, with a probability for each (for example, L1(0.25),L2(0.75)). See Probabilistic go-tos.

  • Tell the simulation to omit the state variable update in the next step with @omit_learn. (See The action @omit_learn.)

  • Reset the counting of an event with count_reset (See The action count_reset.)

Note that global variables (defined using @variables) cannot be set in an action.

Probabilistic go-tos

A probabilistic go-to is an action of the type “go to line L1 with probability p1, go to line L2 with probability p2, etc.” and has the syntax:

L1(p1),L2(p2),...

where p1, p2 etc. are expressions of global variables, local phase variables and numbers.

For example, to go to line L1 with probability 0.4 and to L2 with probability 0.6, use:

L1(0.4),L2(0.6)

Note that the probabilities may be global variables or local phase variables. For example:

START | p1=0.1, p2=0.7, p3=0.2 | L0
L0 s  | L1(p1),L2(p2),L3(p3)
L1 ...
L2 ...
L3 ...

The probabilities within parentheses do not need to add up to 1:

L0 s | L1(0.1),L2(0.7) | L3

means that the probability to go to L3 is 0.2, so this is the same as:

L0 s | L1(0.1),L2(0.7),L3(0.2)

The action @omit_learn

The action @omit_learn tells the simulator to omit the updating of the subject’s state variables in the next step. Typically this is done at the end of a trial, when the phase represents a number of trials. In that case, it is often not desired to carry over the reinforcement value of, say, a reward at the and of a trial to the first stimulus in the next.

For example, in mechanisms with conditioned reinforcement values (w-values), and we have the following stimulus-response history:

s -> response -> reward -> eat -> s

one typically wants to avoid that the w-value for the first stimulus in the trial (s) affects

  • the reinforcement value of reward, and

  • the v-value (associative strength) between reward and eat

The action count_reset

The counter for a specific stimulus element, behavior, or phase line may be reset to zero using the action count_reset.

For example:

count_reset(pull)

resets the counter of pull to zero, so that count(pull) will return zero. This action may be used like this:

stimulus_elements = lever, reward
behaviors = pull, ignore

@phase training stop:reward=100
START                 | count_reset(pull), LEVER_OFF
LEVER_OFF lever       | count(pull)=3: LEVER_ON | LEVER_OFF
LEVER_ON  lever       | pull: REWARD | LEVER_ON
BACKGROUND background | START
REWARD reward         | START

After three pull of the lever, the lever will “activate” and give reward when pulled. Then it is inactivated and three more pull are required to activate it.

To reset the counter for the stimulus element e1, use:

count_reset(e1)

To reset the counter for the phase line with label LBL, use:

count_reset(LBL)

Note that resetting one stimulus element (or behavior or phase line) will not affect any other.

Inheriting a phase

If you have several @phase-blocks that are very similar to each other, there is a way to avoid duplicating each phase line, and instead only state the lines that differ between the phases. This is done through phase inheritance. Do create a phase with name phase2 that inherits from from a phase with label phase1, use:

@phase phase2(phase1) ...

For example, suppose we have the following phase block in a script:

@PHASE training stop: stimulus=50
new_trial   stimulus   | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD
REWARD1     reward1    | new_trial
REWARD2     reward2    | new_trial
REWARD3     reward3    | new_trial
NO_REWARD   background | new_trial

and we want to create a new phase training2 that is the same as the above but where the line with label REWARD3 should read:

REWARD3     reward4    | new_trial

This can be accomplished by simply re-stating the entire phase block training, and make the change. However, this requires a lot of lines and a change in a part of the phase block that should be the same must be performed in each phase block. Therefore, using phase inheritance is better:

@PHASE training2(training) stop: stimulus=50
REWARD3     reward4    | new_trial

The above phase block represents the same phase as:

@PHASE training2 stop: stimulus=50
new_trial   stimulus   | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD
REWARD1     reward1    | new_trial
REWARD2     reward2    | new_trial
REWARD3     reward4    | new_trial
NO_REWARD   background | new_trial