:orphan: ****************** A ``@phase`` block ****************** The environment that the subject interacts with is specified using one or more ``@phase`` blocks. A ``@phase`` block defines the sequence of stimuli that the subject is exposed to, and how the next stimulus in the sequence depends on the response to the previous stimulus. If the environment is an experiment setup, say a maze, one may use one ``@phase`` block per phase in the experiment, for example one ``@phase`` for the training phase and one for the test phase. The structure of a phase block is outlined in the below example:: @phase training stop: reward==100 TRIAL_START lever | pull_lever: REWARD | NO_REWARD REWARD reward | TRIAL_START NO_REWARD background | TRIAL_START The simulation will start with the first phase line below the title line (except for inherited phase blocks, see :ref:`inheriting-a-phase`). The ``@phase`` title line ************************* In general, a ``@phase`` block starts with a *title line* of the format:: @phase phase_name stop:stop_condition where - ``phase_name`` is the name of the phase (see :ref:`the-phase-name`), and - ``stop_condition`` is the stop condition (see :ref:`the-stop-condition`) specifying when to exit the phase, usually after a prededermined number of exposures to a certain stimulus element, or after a predetermined number of responses with a certain behavior. .. note:: ``@phase`` and ``stop:`` are keywords, and cannot be changed. The name of the phase (``phase_name``) and the stop condition (``stop_condition``) are user defined. For example, in a script with the following parameter definitions:: stimulus_elements = lever, reward behaviors = pull_lever, ignore the title line of a @phase block may look like this:: @phase training stop:reward==100 or:: @phase training stop:pull_lever==100 .. _the-phase-name: The phase name -------------- The phase name must be a :ref:`valid name`. .. _the-stop-condition: The phase stop condition ------------------------ A phase stop condition is a `boolean expression `_ that may include the following variables: - Any stimulus element (evaluates to the number of exposures to this stimulus element) - Any behavior (evaluates to the number of responses with this behavior) - Any phase line label (evaluates to the number of times this phase line has been visited) - Any local phase variable (evaluates to the value of this variable) For example, the first line of a @phase block may look like this:: @phase training stop: pull_lever>=10**2 or reward==10 which means that the phase stops when either - the number of responses with the behavior ``pull_lever`` is greater than or equal to :math:`10^2=100`. - the number of exposures to the stimulus element ``reward`` is 100, or See also :ref:`using-local-phase-variables-in-phase-stop-condition`. Boolean expression ------------------ A boolean expression is a mathematical expression that evaluates to *true* or *false*. It may include number literals, variables, parentheses, and the below operators. The supported operators in *Learning Simulator* are the arithmetic operators - ``+`` (addition) - ``-`` (subtraction) - ``*`` (multiplication) - ``/`` (division) - ``**`` (power) and the comparison operators - ``==`` (equals) - ``>`` (greater than) - ``>=`` (greater than or equal to) - ``<`` (less than) - ``<=`` (less than or equal to) and the logical operators - ``and`` - ``or`` and the functions - ``choice`` (See :ref:`choice-function`) - ``rand`` (See :ref:`rand-function`) For example, ``x>2 and y<=5`` is a boolean expression of the variables ``x`` and ``y``, the constants 2 and 5, and the operators ``>`` and ``<=``. It evaluates to *true* whenever ``x`` is greater than two and ``y`` is smaller than or equal to 5, and *false* otherwise. If ``x`` is a number, the expression ``2x-5`` is not a boolean expression since it does not evaluate to *true* or *false*, but to a number. .. _phase-line: A phase line ************ Below the phase title line follows a number of *phase lines*. Each phase line - presents a :ref:`stimulus ` to the subject (optional), - performs a number of (optional) *actions* (see `Actions `_), and - informs which of the phase lines to go to next A phase line has the general format:: LINE_LABEL s | d11,d12,...,COND1:a11,a12,... | d21,d22,...,COND2: a21,a22,... | an1,an2,...,ann where s is a :ref:`stimulus `, ``d11``, ``d12``, ``a11``, ``a12`` etc. are `actions `_, and ``COND1``, ``COND2`` etc. are *phase line conditions* (see :ref:`phase-line-conditions`). The actions are performed sequentially, in the order they appear. For example, the line:: LINE_LABEL s | d11,d12, COND1:a11,a12 | d21,d22, COND2:a21,a22 | d31,d32, COND3:a31,a32 | a41,a42 is interpreted by the simulator as follows:: Present the stimulus s Perform actions d11 and d12 If COND1 is fulfilled: Perform action a11 Perform action a12 Otherwise: Perform action d21 Perform action d22 If COND2 is fulfilled: Perform action a21 Perform action a22 Otherwise: Perform action d31 Perform action d32 If COND3 is fulfilled: Perform action a31 Perform action a32 Otherwise: Perform action a41 Perform action a42 Note that one of the action types is "go to phase line with label X". During the simulation, if the above logic does not determine which phase line to go to next, an error message will appear. For example, if ``COND1`` is fulfilled, ``a12`` cannot be any other action than a phase line label:: LINE_LABEL1 stimulus | d11,d12, COND1:a11,LINE_LABEL2 LINE_LABEL2 ... Since the actions are performed sequentially, an error will appear if the action ``LINE_LABEL2`` is followed by additional actions as these additional actions will never be performed. .. _phase-line-conditions: Phase line conditions ********************* A phase line condition is a `boolean expression `_ that may depend on the following variables: - Any behavior (evaluates to *true* for the behavior that is the response to the stimulus on the line, and *false* for all other behaviors) - Any phase line label (evaluates to the number of times this phase line has been visited) - Any local phase variable (evaluates to the value of this variable) and the following functions: - ``count`` (See :ref:`the-function-count`) - ``count_line`` (See :ref:`the-function-count_line`) .. _the-function-count: The function ``count`` ---------------------- The function ``count`` counts occurrences of each behavior, each stimulus element, and number of visits to each phase line (using the phase line label as identifier). For example:: count(lever) evaluates to the number of times the stimulus element ``lever`` has been exposed to the subject (since the beginning of the phase, or since it was last reset using ``count_reset``, see below). Similarly,:: count(pull_lever) evaluates to the number of times the agent has responded with the behavior ``pull_lever`` (to any stimulus) since the beginning of the phase, or since it was last reset. To return the number of visits to a phase line with label ``LBL``, use:: count(LBL) The counter for a specific stimulus element, behavior, or phase line may be reset to zero using the action ``count_reset`` (see :ref:`the-action-count_reset`). .. _the-function-count_line: The function ``count_line`` --------------------------- The function ``count_line`` counts consecutive occurrences of each behavior, each stimulus element, and number of consecutive visits to the phase line where this function is used. It is perhaps most useful for behaviors, to count the number of consecutive responses to the stimulus on a phase line. For example:: stimulus_element = s1, s2 behaviors = b1, b2, b3 @phase ... LBL1 s1 | count_line(b1)==3: LBL2 | LBL1 LBL2 s2 | LBL1 will expose the subject to the stimulus ``s1`` repeatedly until the response to ``s1`` is the behavior ``b`` three times in a row. Then ``s2`` will be presented. A sequence of stimulus-response pairs may be:: s1 -> b1 -> s1 -> b2 -> s1 -> b1 -> b1 -> b1 -> s2 -> b3 If instead the phase lines looked like this:: LBL1 s1 | count_line(s1)==3: LBL2 | LBL1 LBL2 s2 | LBL1 the subject will be presented with ``s1`` three times, and regardless of the responses, will then be presented with ``s2``:: s1 -> * -> s1 -> * -> s1 -> * -> s2 Note that the above example is equivalent to:: LBL1 s1 | count_line(LBL1)==3: LBL2 | LBL1 LBL2 s2 | LBL1 and that ``count_line(LBL1)`` may be abbreviated ``count_line()`` as the line label is redundant (since ``count_line`` count consecutive occurrences on the very line where it is used):: LBL1 s1 | count_line()==3: LBL2 | LBL1 LBL2 s2 | LBL1 .. _local-phase-variables: Local phase variables ********************* It is possible to use custom local variables within a phase, whose values may change during the phase. They may be assigned a value in an action, and may be referred to in a condition. Local phase variables must be initialized (have an assigned value) before they are used. A phase variable must have a :ref:`valid name`. An action that assigns the value 0.1 to the local variable ``x`` looks as follows:: x=0.1 A phase with an action that sets ``x`` and with a condition that uses ``x`` may look like this:: @phase ... L0 x=0.1 | L1 # Set x to 0.1 and go to L1 L1 s1 | x=x+0.1, x>=0.5: L2 | L1 # Present s1, increase x with 0.1. If x>=0.5 go to L2, otherwise L1 L2 s2 | L1 # Present s2 and go to L1 .. _using-local-phase-variables-in-phase-stop-condition: Using local variables in a phase stop condition ----------------------------------------------- It is possible to use a local phase variable in a phase stop condition. For example:: @phase phase_name stop:x>2 L0 x=0.1 | L1 L1 s1 | x=x+0.1, x>=0.5: L2 | L1 L2 s2 | L1 The stop condition is not checked until the first stimulus is presented. If the phase stop condition depends on local variables, these local variables must therefore be initialized before the first stimulus is presented (like in the example above), otherwise an "Unknown variable" error will occur. Actions ******* An action is one of the following: - Set a local phase variable (for example, ``x=2``) - Go to a phase line (by simply stating the corresponding phase line label) - Go to one of several phase lines, with a probability for each (for example, ``L1(0.25),L2(0.75)``). See :ref:`probabilistic-go-tos`. - Tell the simulation to omit the state variable update in the next step with ``@omit_learn``. (See :ref:`omit_learn`.) - Reset the counting of an event with ``count_reset`` (See :ref:`the-action-count_reset`.) Note that global variables (defined using ``@variables``) cannot be set in an action. .. _probabilistic-go-tos: Probabilistic go-tos -------------------- A probabilistic go-to is an action of the type "go to line L1 with probability p1, go to line L2 with probability p2, etc." and has the syntax:: L1(p1),L2(p2),... where ``p1``, ``p2`` etc. are expressions of global variables, local phase variables and numbers. For example, to go to line ``L1`` with probability 0.4 and to ``L2`` with probability 0.6, use:: L1(0.4),L2(0.6) Note that the probabilities may be global variables or local phase variables. For example:: START | p1=0.1, p2=0.7, p3=0.2 | L0 L0 s | L1(p1),L2(p2),L3(p3) L1 ... L2 ... L3 ... The probabilities within parentheses do not need to add up to 1:: L0 s | L1(0.1),L2(0.7) | L3 means that the probability to go to ``L3`` is 0.2, so this is the same as:: L0 s | L1(0.1),L2(0.7),L3(0.2) .. _omit_learn: The action ``@omit_learn`` -------------------------- The action ``@omit_learn`` tells the simulator to omit the updating of the subject's state variables in the next step. Typically this is done at the end of a trial, when the phase represents a number of trials. In that case, it is often not desired to carry over the reinforcement value of, say, a reward at the and of a trial to the first stimulus in the next. For example, in mechanisms with conditioned reinforcement values (w-values), and we have the following stimulus-response history:: s -> response -> reward -> eat -> s one typically wants to avoid that the w-value for the first stimulus in the trial (``s``) affects - the reinforcement value of ``reward``, and - the v-value (associative strength) between ``reward`` and ``eat`` .. _the-action-count_reset: The action ``count_reset`` -------------------------- The counter for a specific stimulus element, behavior, or phase line may be reset to zero using the action ``count_reset``. For example:: count_reset(pull) resets the counter of ``pull`` to zero, so that ``count(pull)`` will return zero. This action may be used like this:: stimulus_elements = lever, reward behaviors = pull, ignore @phase training stop:reward=100 START | count_reset(pull), LEVER_OFF LEVER_OFF lever | count(pull)=3: LEVER_ON | LEVER_OFF LEVER_ON lever | pull: REWARD | LEVER_ON BACKGROUND background | START REWARD reward | START After three ``pull`` of the ``lever``, the lever will "activate" and give reward when pulled. Then it is inactivated and three more ``pull`` are required to activate it. To reset the counter for the stimulus element ``e1``, use:: count_reset(e1) To reset the counter for the phase line with label ``LBL``, use:: count_reset(LBL) Note that resetting one stimulus element (or behavior or phase line) will not affect any other. .. _inheriting-a-phase: Inheriting a phase ****************** If you have several @phase-blocks that are very similar to each other, there is a way to avoid duplicating each phase line, and instead only state the lines that differ between the phases. This is done through *phase inheritance*. Do create a phase with name ``phase2`` that inherits from from a phase with label ``phase1``, use:: @phase phase2(phase1) ... For example, suppose we have the following phase block in a script:: @PHASE training stop: stimulus=50 new_trial stimulus | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD REWARD1 reward1 | new_trial REWARD2 reward2 | new_trial REWARD3 reward3 | new_trial NO_REWARD background | new_trial and we want to create a new phase ``training2`` that is the same as the above but where the line with label ``REWARD3`` should read:: REWARD3 reward4 | new_trial This can be accomplished by simply re-stating the entire phase block ``training``, and make the change. However, this requires a lot of lines and a change in a part of the phase block that should be the same must be performed in each phase block. Therefore, using phase inheritance is better:: @PHASE training2(training) stop: stimulus=50 REWARD3 reward4 | new_trial The above phase block represents the same phase as:: @PHASE training2 stop: stimulus=50 new_trial stimulus | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD REWARD1 reward1 | new_trial REWARD2 reward2 | new_trial REWARD3 reward4 | new_trial NO_REWARD background | new_trial