Allow running of unmodified envs with original done signals

[garymcintire]

I try this and watch the movies

python -u rl_teacher/teach.py -p rl -e Humanoid-v1 -n base-rl -w 12

It always runs the full 1000 steps. Putting in a print statement in rollouts.py shows that the env.step never returns a 'done'

Is it supposed to be like this? If so, why?

[nottombrown]

Hey Gary, as in Deep RL from Human Preferences, we remove the done signals.

You can see the envs.py file for details.

I'd be interested in accepting PRs that make it easy to run the unmodified environments as well as the modified ones.

See the following issue:
#5

[garymcintire]

Thanks for clarifying

[nottombrown]

I'm leaving this open because it's a separate issue from #5

[nottombrown]

Ah, actually this is already an open issue. Closing in favor of #12