Can RLHF even simpler to maximize the expectation of rewards?

[kindernerd]

GRPO simplifies advantage to (r-mean)/std, i'm wondering whether RLHF can even be simpler by directly maximum the following objective:
$\sum_o\pi_{\theta}(o|q)[r_o - E(r_o|q)]$
which can be approximated by sampling or using the N-best Lists
$\sum_{o_i\in \pi_{old}}\pi_{\theta}(o_i|q)[r_{o_i} -mean(r)]$
this is similar to sequence training (MWER) in e2e asr optimization, proposed by google in this paper https://arxiv.org/abs/1712.01818

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

[github-actions]

false