Belief over policies changes rapidly

[matteolimoncini]

I'm using pymdp to model an agent that attends an experiment composed of 160 trials, with 2 timestep for each trial.
The agent can do only two actions, and there are only two states.

During each trial the agent should do an observation, then it infers a policy, it chooses an action, it executes that action and does another observation; at the end of the trial the agent updates its beliefs about the hidden states.

The problem is that the belief over the states and the belief over policies change rapidly between all trials, and they are often around 0 or around 1, rarely assume values in between.

I tried to modify the gamma parameters in the Agent constructor without success (with gamma = 5, 2, 0.5, 0.2)
How can I do?

This is the pseucode of my problem

my_agent = Agent(A=A, B=B, C=C, gamma=0.5)
for trial in range(trials):

    agent_stimul_obs = my_env.get_observation(...)
    obs = [agent_stimul_obs, agent_shock_obs, agent_surpr_obs)]
    
    qs = my_agent.infer_states(obs)

    policies = my_agent.infer_policies()
    agent_action = my_agent.sample_action()

    agent_surpr_obs = my_env.step(...)
    agent_shock_obs = my_env.get_observation(...)


    obs = [agent_stimul_obs, agent_shock_obs, agent_surpr_obs]
    qs = my_agent.infer_states(obs)

[conorheins]

Hi @matteolimoncini ,

Apologies for the delayed reply. It sounds like the fact that the beliefs about states and policies change rapidly from one trial to the next, is due to the construction of the generative model (your A and B arrays), and the way observations are generated from the environment.

Depending on how these are constructed, it might be the case that gamma has very little effect on the policy posterior. To be able to diagnose your problem I would need to hear more about your generative model / task specifics.