Jiajun Fan

ICLR 2023, Oral — Notable Top 5%

Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection

At a glance

The paper in four results.

To explore, leading Atari agents such as Agent57 train a population of policies and play one of them at a time. What if every episode could blend all of them? Learnable Behavior Control does, and learns which blend to play.

  • Blend.every behavior mixes all of its policies
  • Select.a bandit learns which blend to play
  • Efficient.the records fall within 1B training frames

The problem

Population-based agents pick one policy at a time.

Picking caps the behaviors at the number of policies, and every extra policy costs more data.

Pick one policy

as population-based agents such as Agent57 do

Its behaviors are only as many as its policies.

  • Every policy shapes each behavior
  • More behaviors without more policies

Blend all of them

as LBC does, with a new blend every episode

The same policies open a continuous space of behaviors.

  • Every policy shapes each behavior
  • More behaviors without more policies
Schematic: the action probabilities are illustrative, not measured.

More behaviors should come from the same policies, not from more of them.

The idea Blend, do not pick

LBC blends every policy, so the same policies open a continuous space of behaviors.

Any point in that space is one behavior, and a bandit learns which one to play.

1

Blend

Each policy is sharpened or flattened by its own temperature, then weighted. The weighted sum is the behavior.

2

Select

The space of blends is cut into regions. A bandit scores each by the returns it brought, plus a bonus for regions it has tried less.

3

Learn

The episodes train all three policies off-policy, and each return updates the bandit’s score for its region.

The space of behaviors grows with the blend, not with the population.

How it works

Every behavior is a blend of all its policies, and a bandit learns which blend to play.

Three policies learn side by side. Before each episode, a bandit sets a weight and a temperature for every policy. The blend plays the episode, and its return tells the bandit how good that choice was.

Three policieseach at its own temperature

1Policy 1

2Policy 2

3Policy 3

How much each one countsdrag the point inside the triangle

The behavior it playsone blend of all three

μ = ω1·softmax(τ1Φ1) + ω2·softmax(τ2Φ2) + ω3·softmax(τ3Φ3)

Interactive schematic: the probabilities are illustrative, not measured. Drag the point or move a slider.

Every policy shapes each behavior, and the returns decide the blend.

The evidence Finding 01

It breaks 24 human world records.

Not the average human player: the best score any human has posted, game by game. No agent in the comparison breaks more.

24 of 57 gamesLBC, 1B frames

  • Alien
  • Amidar
  • Assault
  • Asterix
  • Asteroids
  • Atlantis
  • Bank Heist
  • Battle Zone
  • Beam Rider
  • Berzerk
  • Bowling
  • Boxing
  • Breakout
  • Centipede
  • Chopper Command
  • Crazy Climber
  • Defender
  • Demon Attack
  • Double Dunk
  • Enduro
  • Fishing Derby
  • Freeway
  • Frostbite
  • Gopher
  • Gravitar
  • Hero
  • Ice Hockey
  • Jamesbond
  • Kangaroo
  • Krull
  • Kung Fu Master
  • Montezuma Revenge
  • Ms Pacman
  • Name This Game
  • Phoenix
  • Pitfall
  • Pong
  • Private Eye
  • Qbert
  • Riverraid
  • Road Runner
  • Robotank
  • Seaquest
  • Skiing
  • Solaris
  • Space Invaders
  • Star Gunner
  • Surround
  • Tennis
  • Time Pilot
  • Tutankham
  • Up n Down
  • Venture
  • Video Pinball
  • Wizard of Wor
  • Yars Revenge
  • Zaxxon

Highlighted: games where the agent reaches the human world record. Agent57 breaks 18 with 100B frames, MEME 16 with 1B; in the paper’s other tables, GDI-H3 breaks 22, MuZero 19 and Go-Explore 15.

Three records Agent57 did not reach in 100B frames

Enduro

Average human player860.5

Human world record9,500

Agent57, 100B frames2,367.71

LBC, 1B frames14,332.5

Gopher

Average human player2,412.5

Human world record355,040

Agent57, 100B frames117,777.08

LBC, 1B frames488,900

Kung Fu Master

Average human player22,736.3

Human world record1,000,000

Agent57, 100B frames206,845.82

LBC, 1B frames1,666,665

Raw game scores from the paper’s tables; LBC’s are averaged over five seeds.

The first agent to break 24 human world records within 1B training frames.

The evidence Finding 02

The highest mean score, on 1B frames.

Across all 57 games it scores highest, while training on as few frames as any agent here.

Human-normalized score sets each game’s score against a random agent and an average human player, then averages over the 57 games. Training frames: LBC and MEME 1B, Go-Explore 10B, MuZero 20B, Agent57 100B.

More score, from less experience.

The evidence Finding 03

A bigger behavior space scores higher.

Shrink the space step by step, first to one policy, then to one temperature, and the score falls each time.

Mean and median human-normalized score over the 57 games, each variant scaled to the full method; the labels are the drops the paper reports.

Enlarging the behavior space is a promising way to make existing methods stronger.

The evidence Finding 04

The blend has to be learned.

Keep the whole space but pick blends at random, and most of the score is gone.

Mean and median human-normalized score over the 57 games, scaled to the full method.

In most games, the learned choice starts with exploratory blends and turns to exploitative ones as training goes on, without letting exploration collapse to zero.

Why it matters

How an agent explores becomes something it learns.

Which behavior gathers the data used to be a fixed rule, or a pick from a fixed list. LBC turns it into a space to design and a choice to learn.

What this opens up

A harder yardstick

Human world records, not the average player, to claim superhuman play.

More behaviors, same policies

Blending grows the space of behaviors without training more policies.

A general recipe

It plugs into distributed off-policy agents: blend the policies, then learn the blend.

The film

The story, as a short film.

Film poster: LBC, learnable behavior control, breaking Atari human world records

Opens the film page.

Cite

Cite the paper.

@inproceedings{fan2023learnable,
  title={Learnable Behavior Control: Breaking Atari Human World Records
  via Sample-Efficient Behavior Selection},
  author={Jiajun Fan and Yuzheng Zhuang and Yuecheng Liu and Jianye HAO
  and Bin Wang and Jiangcheng Zhu and Hao Wang and Shu-Tao Xia},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023},
  url={https://openreview.net/forum?id=FeWvD0L_a4}
}