[Question] Maze Dense Reward

[llewynS]

Question

Looking at the dense reward function for Maze Env:

return np.exp(-np.linalg.norm(desired_goal - achieved_goal))

The agent seems to prefer sitting the ball as close as possible to the goal without touching it after optimisation.

This makes sense given there is no bonus for reaching the reward and the reward is positive for all time steps.

Why is the dense reward formulated this way?

[Kallinteris-Andreas]

1. Are you using continuing_task=True (which is the default)?
2. Are you resetting about termination=True?
3. Have you experimented with other reward functions?

[onnoeberhard]

Somewhat related: the description of the maze environments says the returned reward is the negative Euclidean distance between the achieved goal position and the desired goal. This is wrong (it is the exponential of the negative distance).

[Kallinteris-Andreas]

@onnoeberhard

Gymnasium-Robotics/gymnasium_robotics/envs/maze/maze_v4.py

Lines 374 to 381 in 8606192

	def compute_reward(
	self, achieved_goal: np.ndarray, desired_goal: np.ndarray, info
	) -> float:
	distance = np.linalg.norm(achieved_goal - desired_goal, axis=-1)
	if self.reward_type == "dense":
	return np.exp(-distance)
	elif self.reward_type == "sparse":
	return (distance <= 0.45).astype(np.float64)

You are correct, can you make a PR to fix it?
You can use the Gymnasium/MuJoCo as reference https://gymnasium.farama.org/main/environments/mujoco/ant/#rewards