Skip to content

Commit

Permalink
nits: change global-reward to just reward, since there's no longer a …
Browse files Browse the repository at this point in the history
…local-reward. Also, add note about moving right incurring a negative reward.
  • Loading branch information
anordin95 committed Dec 4, 2024
1 parent cbde8d9 commit 78ba747
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions pettingzoo/butterfly/pistonball/pistonball.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@
**Actions**: Every piston can be acted on at each time step. In discrete mode, the action space is 0 to move down by 4 pixels, 1 to stay still, and 2 to move up by 4 pixels. In continuous mode, the value in the range [-1, 1] is proportional to the amount that the pistons
are lowered or raised by. Continuous actions are scaled by a factor of 4, so that in both the discrete and continuous action space, the action 1 will move pistons 4 pixels up, and -1 will move pistons 4 pixels down.
**Rewards**: The same reward is provided to each agent based on how much the ball moved left in the last time-step plus a constant time-penalty. The distance component is the percentage of the initial total distance (i.e. at game-start) to the left-wall travelled in the past timestep.
For example, if the ball began the game 300 pixels away from the wall, began the time-step 180 pixels away and finished the time-step 175 pixels away, the distance reward would be 100 * 5/300 = 1.7. There is also a configurable time-penalty (default: -0.1) added to the
distance-based reward at each time-step. For example, if the ball does not move in a time-step, the reward will be -0.1 not 0. This is to incentivize solving the game faster.
**Rewards**: The same reward is provided to each agent based on how much the ball moved left in the last time-step (moving right results in a negative reward) plus a constant time-penalty. The distance component is the percentage of the initial total distance (i.e. at game-start)
to the left-wall travelled in the past timestep. For example, if the ball began the game 300 pixels away from the wall, began the time-step 180 pixels away and finished the time-step 175 pixels away, the distance reward would be 100 * 5/300 = 1.7. There is also a configurable
time-penalty (default: -0.1) added to the distance-based reward at each time-step. For example, if the ball does not move in a time-step, the reward will be -0.1 not 0. This is to incentivize solving the game faster.
Pistonball uses the chipmunk physics engine, and are thus the physics are about as realistic as in the game Angry Birds.
Expand Down Expand Up @@ -631,15 +631,15 @@ def step(self, action):
# The negative one is included since the x-axis increases from left-to-right. And, if the x
# position decreases we want the reward to be positive, since the ball would have gotten closer
# to the left-wall.
global_reward = (
reward = (
-1
* (ball_curr_pos - self.ball_prev_pos)
* (100 / self.distance_to_wall_at_game_start)
)
if not self.terminate:
global_reward += self.time_penalty
reward += self.time_penalty

self.rewards = {agent: global_reward for agent in self.agents}
self.rewards = {agent: reward for agent in self.agents}
self.ball_prev_pos = ball_curr_pos
self.frames += 1
else:
Expand Down

0 comments on commit 78ba747

Please sign in to comment.