3

Why don't the gym environments come with "valid actions"? The normal gym environment accepts as input any action, even if it's not even possible.

Is this a normal thing in reinforcement learning? Do the models really have to learn what valid actions are all the time? Would it not be much nicer to have a env.get_valid_actions() functions so that the model knows what actions are doable? Or is this somehow possible and I'm missing it?

Muppet
  • 777
  • 1
  • 7
  • 13

1 Answers1

3

In general, if the agent is simply not able to take non-valid actions in a given environment (e.g. due to strict rules of a game, like chess), then it is standard practice to have the environment support that by providing some kind of function or filter for $\mathcal{A}(s)$, the set of actions available in state $s$.

However, it does seem like the basic Gym interface does not support it, and has no plans to support it.

It is still possible for you to write an environment that does provide this information within the Gym API using the env.step method, by returning it as part of the info dictionary:

next_state, reward, done, info = env.step(action)

The info return value can contain custom environment-specific data, so if you are writing an environment where the valid action set changes depending upon state, you can use this to communicate with your agent. The caveat is that both your environment and agent would be working to a convention you had invented in order to do this, and the same approach would not extend to other environments or be used by developers of other agents.

Alternatively, you can support calculating what the valid actions are in any other custom way that is not covered by the Gym API - a new method on the environment, a separate "rules engine" for a game etc. Both the agent and the environment could call that in order to perform their roles correctly.

The situation without an official API to do this may be subject to change, but from the linked Github issue it seems that Open AI developers consider a generic interface for this, that accounted for all the different kinds of action space, was more effort than it was worth.

Neil Slater
  • 28,338
  • 4
  • 77
  • 100
  • Hey Neil, thanks for your answer. I understand that I can do it myself via a custom environment, wrapper, or other means, but one of the main issues is that all the available open source RL libraries cannot take that into account and I like using those as benchmarks. The existing RL libs always use all actions. It would be so much nicer if this was a feature of the base environment, such as it is in OpenSpiel. Even BaseLines has been discussing the implementation of valid moves here: https://github.com/hill-a/stable-baselines/pull/453 It's a shame that this isn't properly addressed by gym. – Muppet Oct 12 '19 at 20:31
  • @Muppet: I agree it is a shame, but that is the situation with Open AI's gym as far as I read it. It is possible there is some other initiative to standardise the API around dynamic action availablilty in gym, that I could not find. – Neil Slater Oct 13 '19 at 08:13