![](https://csdnimg.cn/release/download_crawler_static/12793445/bg8.jpg)
useful flag knowledge was learned prior to knowledge related to tagging or one’s teammate’s
situation. Note that agents were never explicitly trained to model this knowledge, thus these
results show the spontaneous emergence of these concepts purely through RL-based training.
A visualisation of how the agent represents knowledge was obtained by performing dimen-
sionality reduction of the agent’s activations using t-SNE (67). As can be seen from Figure 3,
internal agent state clustered in accordance with conjunctions of high-level game state features:
flag status, respawn state, and room type. We also found individual neurons whose activations
coded directly for some of these features, e.g. a neuron that was active if and only if the agent’s
teammate was holding the flag, reminiscent of concept cells (51). This knowledge was ac-
quired in a distributed manner early in training (after 45K games), but then represented by a
single, highly discriminative neuron later in training (at around 200K games). This observed
disentangling of game state is most pronounced in the FTW agent (Figure S8).
One of the most salient aspects of the CTF task is that each game takes place on a randomly
generated map, with walls, bases, and flags in new locations. We hypothesise that this requires
agents to develop rich representations of these spatial environments to deal with task demands,
and that the temporal hierarchy and explicit memory module of the FTW agent help towards
this. An analysis of the memory recall patterns of the FTW agent playing in indoor environ-
ments shows precisely that: once the agent had discovered the entrances to the two bases, it
primarily recalled memories formed at these base entrances (Figure 4, Figure S7). We also
found that the full FTW agent with temporal hierarchy learned a coordination strategy dur-
ing maze navigation that ablated versions of the agent did not, resulting in more efficient flag
capturing (Figure S3).
Analysis of temporally extended behaviours provided another view on the complexity of
behavioural strategies learned by the agent (34). We developed an unsupervised method to
automatically discover and quantitatively characterise temporally extended behaviour patterns,
inspired by models of mouse behaviour (73), which groups short game-play sequences into be-
havioural clusters (Figure S9, Supplementary Video https://youtu.be/dltN4MxV1RI).
The discovered behaviours included well known tactics observed in human play, such as waiting
in the opponents base for a flag to reappear (opponent base camping) which we only observed
in FTW agents with a temporal hierarchy. Some behaviours, such as following a flag-carrying
teammate, were discovered and discarded midway through training, while others such as per-
forming home base defence are most prominent later in training (Figure 4).
In this work, we have demonstrated that an artificial agent using only pixels and game points
as input can learn to play highly competitively in a rich multi-agent environment: a popular
multiplayer first-person video game. This was achieved by combining a number of innovations
in agent training – population based training of agents, internal reward optimisation, and tem-
porally hierarchical RL – together with scalable computational architectures. The presented
framework of training populations of agents, each with their own learnt rewards, makes mini-
mal assumptions about the game structure, and therefore should be applicable for scalable and
stable learning in a wide variety of multi-agent systems, and the temporally hierarchical agent
represents a sophisticated new architecture for problems requiring memory and temporally ex-
8