On-Policy Distillation from First Principles
A few weeks ago, On-Policy Distillation (OPD) took over the RL world with this excellent Thinking machines blogpost ⬈. I thought the method was neat and effective, but I had a hard time understanding its connection to RL and just really, why it works in the first place. I decided the best way to grok it would be to see if I could implement the method myself. In the course of that, I went through all the math and intuition behind the method from first principles. It was really fun!