Spent some time looking into this, sadly it's not as easy as I thought it ought to be. Videojs doesn't provide a direct way of changing the buttons' labels (and most of them are dynamically changing) so for a clean way of changing this I think we'd need to extend every single (around 15) of the Videojs buttons we use.

When/if I am bored I'll get this done because I'd like to see it too ;)

Boosting this issue / keeping it in mind for any possible future shakaup.

From left to right the respective to the player...
Skipping the buttons with multiple options (like quality, captions, and playback speed)
I am having a hard time grepping for a few of them. (i.e. Play, Pause, Replay, Mute, Pic in Pic, and Full Screen)

Play (Shortcut 'Space Bar')
Pause (Shortcut 'Space Bar')
Replay (NA)
Mute (Shortcut 'M' or 'm')
Toggle Loop (N/A) #4298
Picture-in-Picture (Shortcut 'I' or 'i')
Full Window (Shortcut 'S' or 's')
Full Screen (Shortcut 'F' or 'f')

cloud@cloud:~/FreeTube$ grep -r "button.title"
grep: src/renderer/components/ft-video-player/.ft-video-player.js.swo: binary file matches
src/renderer/components/ft-video-player/ft-video-player.js:          button.title = 'Toggle Loop'
src/renderer/components/ft-video-player/ft-video-player.js:          button.title = 'Full Window' 
src/renderer/components/ft-video-player/ft-video-player.js:          button.title = 'Toggle Theatre Mode'
src/renderer/components/ft-video-player/ft-video-player.js:          button.title = 'Screenshot'
src/renderer/components/ft-video-player/ft-video-player.js:          button.title = 'Select Quality'

replaced by the following to at least show the shortcut on hover of full window and toggle loop buttons...

src/renderer/components/ft-video-player/ft-video-player.js:          button.title = 'Full Window (Shortcut S or s)'
src/renderer/components/ft-video-player/ft-video-player.js:          button.title = 'Toggle Loop (Shortcut R or r)'