Regression tutorial

[hpplyt]

Anything related to the basic regression tutorial.

@pgarimidi @nb2838

[velezbeltran]

@ucl-jhr I am working on this and I was wondering:

1. How in-depth do you think we should go with respect to explanations about 1. Bayesian Neural Networks 2. Inference methods. In particular, I a writing a section about using MCMC and VI but I am unsure if I should just point to pyro tutorials or explain it there. I believe explaining a bit about what they do would be helpful (maybe a paragraph per inference method) but I wanted to check first with you.
2. Should we just ignore Pyro completely in the tutorials (meaning not mention at all) or do you think it is helpful to explain a bit whenever we use a pyro specific construct. In general, how much do you want to hide Pyro from users of TyXe.

[hpplyt]

Great, thanks a lot for your efforts. Regarding your questions.

1. I think I would keep this fairly basic. A paragraph per method sounds about right. I would expect that users either are already familiar with Bayesian modelling or just want software to quickly try things out without having to go through much theory first. For the former we don't really need an extensive introduction and for the latter there should be plenty of resources around if they want to learn more, so this is probably not the most urgent bit to spend time on. I'd be happy to have an optional tutorial later if you're interested in writing one, I just wouldn't make it a focus for the initial tutorial.
2. We should definitely mention and acknowledge Pyro, after all most of the complicated stuff is implemented in there. The aim of TyXe is, however, to be usable by people who don't know any Pyro at all. It is probably important for people to know Pyro if they want to add more functionality, so again it might make sense to add a tutorial at some stage. But for now we could perhaps just have some optional "deep dive" sections for people familiar with Pyro if you want to explain something in more detail?

Let me know if you have a draft that you want me to look at, I think most of this is somewhat context-dependent.

[velezbeltran]

1. Great. That sounds reasonable so I will try to stick to that.
2. I completely understand what you mean about not having to know Pyro in detail for using TyXe.

Yes, that would be very helpful. I just made a pull request #7 with a draft of the tutorial written in a notebook. I essentially summarized parts of the paper, parts of the documentation and reorganized the example. I added a really very short introduction to bayesian neural networks but we can remove it if you think it is not necessary. Please let me know if this is what you had in mind and if it wasn't I would be happy to change it to whatever you see fit.

Additionally, I was also not completely sure how we are planning on integrating them so I just put them in the examples but that is easily fixable.

You can also watch a rendered version of it my fork.

[hpplyt]

Thanks, this looks like a really good start already.

I added a really very short introduction to bayesian neural networks but we can remove it if you think it is not necessary.

I think it's nice to have this, but perhaps @karalets disagrees (and surely has further feedback). I would just rework the points with the disadvantages of deterministic networks/motivation for BNNs a little bit and keep them a bit simpler, in particular not mention overfitting (since that is a fairly complicated discussion in the context of NNs) and instead just link a couple of papers, e.g. on overconfidence and forgetting.

I was also not completely sure how we are planning on integrating them

My plan was having the tutorials in docs/source, so ideally you would move the contents of the notebook into a file called docs/source/tutorials.regression.rst so that it appears on the readthedocs page. That would also make it easier to discuss smaller points around phrasings, typos etc in the PR. If I set everything up correctly, you should be able (after installing the dependencies) to build the docs (make html from inside docs) and view them (python -m http.server) locally.
I hope this won't be too much of a headache, let me know if you run into any issues. One option would also be to remove all boilerplate code from the .rst tutorial and replace the regression notebook in examples/ with a fully runnable version of the tutorial, although that would of course be a little bit more work.

A couple of smaller points:

• I think it might be better to not position MCMC as one of two reasonable options for inference, but be clear that the focus at this time lies on SVI. As far as I know (although this might be outdated), pyro doesn't implement any SGMCMC methods (and neither does TyXe - this would obviously be a great addition if this is something that interests you), so it really is an option only for small datasets and networks. So I think it would be best to only introduce MCMC towards the end of the tutorial as more accurate inference method for small problems.
• I wouldn't make the call to .fit inside a local_reparameterization context straight away but only introduce it as a variance reduction method after having shown predict for the first time. If you plot the ELBOs both with and without local reparameterization the difference is quite noticeable if I remember correctly.
• When creating the optimizer, it would be good to stress that it has to be a pyro optimizer, not a standard pytorch one.

Thanks again for your work on the tutorials, it's much appreciated :-)

[karalets]

I need to block some time to have a look, will do so before Sunday! Thank you for all the movement here, meanwhile.

…

On Thu, Oct 21, 2021 at 1:51 PM ucl-jhr ***@***.***> wrote: Thanks, this looks like a really good start already. I added a really very short introduction to bayesian neural networks but we can remove it if you think it is not necessary. I think it's nice to have this, but perhaps @karalets <https://github.com/karalets> disagrees (and surely has further feedback). I would just rework the points with the disadvantages of deterministic networks/motivation for BNNs a little bit and keep them a bit simpler, in particular not mention overfitting (since that is a fairly complicated discussion in the context of NNs) and instead just link a couple of papers, e.g. on overconfidence <https://arxiv.org/abs/1706.04599> and forgetting <https://arxiv.org/abs/1312.6211>. I was also not completely sure how we are planning on integrating them My plan was having the tutorials in docs/source, so ideally you would move the contents of the notebook into a file called docs/source/tutorials.regression.rst so that it appears on the readthedocs page <https://tyxe.readthedocs.io/en/latest/tutorials.html>. That would also make it easier to discuss smaller points around phrasings, typos etc in the PR. If I set everything up correctly, you should be able (after installing the dependencies) to build the docs (make html from inside docs) and view them (python -m http.server) locally. I hope this won't be too much of a headache, let me know if you run into any issues. One option would also be to remove all boilerplate code from the .rst tutorial and replace the regression notebook in examples/ with a fully runnable version of the tutorial, although that would of course be a little bit more work. A couple of smaller points: - I think it might be better to not position MCMC as one of two reasonable options for inference, but be clear that the focus at this time lies on SVI. As far as I know (although this might be outdated), pyro doesn't implement any SGMCMC methods (and neither does TyXe - this would obviously be a great addition if this is something that interests you), so it really is an option only for small datasets and networks. So I think it would be best to only introduce MCMC towards the end of the tutorial as more accurate inference method for small problems. - I wouldn't make the call to .fit inside a local_reparameterization context straight away but only introduce it as a variance reduction method after having shown predict for the first time. If you plot the ELBOs both with and without local reparameterization the difference is quite noticeable if I remember correctly. - When creating the optimizer, it would be good to stress that it has to be a pyro optimizer, not a standard pytorch one. Thanks again for your work on the tutorials, it's much appreciated :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKWCL7K5WQRXQ6OV3GVNHTUIB4OBANCNFSM5FCCHKCQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[velezbeltran]

@ucl-jhr Thank you a lot for your comments. I will switch the document to .rst and put it under the docs. I will also try to do both versions (one without a boilerplate and the other runnable one in the examples/ notebook). I will take a look at the SGMCMC paper as I wasn't familiar with it and will definitely switch the MCMC code.

With regards to the local parametrization, I'll mention the local_reparametrization context but I also would like to explain it a bit more in a second "advanced" tutorial as we talked about.

[velezbeltran]

I added commits reflecting the changes we talked about. Let me know what you think.