Can you expand a little bit on that? It seems like we are talking about very small programs here with simple specifications. What are some common problems with reinforcement learning implementations?
Is it really just that you can't easily test your program?
1) You can't easily test it - bugs can lead to semi-working thing and you may need to run thousands of experiments to average
2) Many hyperparameters - it may be critical to get them right
3) No gradual implementation - it doesn't work until you get all the (often gimmicky) parts right. Take A3C for example - its paper version is parallel, may be hard to implement and debug in your language (and hardware) of choice.
Generally, RL algos performance is underwhelming in my experience, without heavy tuning (and so-called "domain expertise" in reward function), but I'm not an expert and OpenAI guys show that you can make the working thing like Dota2 bot, so they give me hope.
Is it really just that you can't easily test your program?