Writing tutorial for deep reinforcement learning implementation with pytorch from scratch

[hahas94]

Description

This PR is for a tutorial I have been working for, namely the deep reinforcement learning implementation with pytorch from scratch. There are no changes in other code/files, hence this RP only adds a new file, along with images and gifs.

Fixes # (issue)

Type of change

Screenshots

Checklist:

• I have run the pre-commit checks with pre-commit run --all-files (see CONTRIBUTING.md instructions to set it up)
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[RedTachyon]

Was this tutorial discussed with anyone, on Discord or otherwise?

[hahas94]

Yes, see Issues Tutorials #28, link.

[pseudo-rnd-thoughts]

Thanks for the tutorial @hahas94, is this a new tutorial or has this been published before?

Having a quick look it is very impressive, I haven't checked the implementation

My initial reaction is a couple of the more advanced points shouldn't be included, i.e., the parallel_training
A quick formatting point is that for code then you need to use two backticks not one (otherwise its italicized)

I'll try to make a more complete review this week

[hahas94]

Thank you!

No it has not been published before, so it is new.

Yes please have a thorough look and let me know what changes to make or how to restructure it if necessary.

[Kallinteris-Andreas]

Why does this need to be computed, can't env.action_space.shape or env.observation_space.shape be used instead

[hahas94]

I am not sure how it could be computed from only env.action_space.shape or env.observation_space.shape without going through the steps made by the Conv2d layer, that is kerner_size, stride padding etc.

[Kallinteris-Andreas]

doesn't output_size == action_space.shape[0]??

[hahas94]

If we are talking about the output of the network, then in C51 it is action_space.shape[0] * n_atoms.

However, note the two lines

in_features = self._output_size()
out_features = self._params.n_actions * self._params.n_atoms

where the method _output_size is used to compute the output size of the flatten layer. Maybe the choice of method name is not good?

[Kallinteris-Andreas]

"in_features = self._output_size()" I very confused by what I am reading here, maybe you need to rename the function _output_size to something else, or add some more comments

[hahas94]

Suspected that. Changed the method name to:

def _fc_input_size(self) -> int:
        """Compute size of input to first linear layer."""

[Kallinteris-Andreas]

isn't the case that:
in_features = env.observation_space.shape[0]

[hahas94]

Partially true.

If the vision part of the network is not utilized, then indeed in_features = env.observation_space.shape[0], and this is exactly what is returned by _fc_input_size, since the convolutional sequential is empty, so the observation is returned as is.

But if the vision part is active, then it depends on the observation shape and convolutional operations.
Assuming that env.observation_space.shape == (4, 84, 84) and the current convolutional structure, then the size is computed manually as follows:
the output shape of the first conv2d layer is (32, 20, 20),
the output shape of the second conv2d layer is (64, 9, 9) and
the output shape of the third conv2d layer is (64, 7, 7)
which will produce 64 * 7 * 7 = 3136 weights at the flatten layer.

But we can't always assume that the env.observation_space.shape == (4, 84, 84) or that the network has this structure in general. So I prefer computing it using the current method.

[Kallinteris-Andreas]

Why are obs_shape and n_actions hyperparameter? They are constants of the environments

[hahas94]

Not only these two, but several other parameters are in this class that are not hyperparameters, however this way it is easier to pass these parameters to any function/class that need them.

[Kallinteris-Andreas]

"in_features = self._output_size()" I very confused by what I am reading here, maybe you need to rename the function _output_size to something else, or add some more comments

[hahas94]

@pseudo-rnd-thoughts, @Kallinteris-Andreas

just checking in on this PR. Is there anything more I need to address or any additional feedback you can provide?

[JDRanpariya]

Is there a reason this PR is not merged but closed? would love to help out with this, seems like most of the work is done.