Do copies of a learned trading policy help or harm the market they trade in?

In a companion paper [1], we created a simulated stock exchange using a market maker and twenty-four rule-based traders, and trained a reinforcement-learning agent to trade in it. This study is looking at how the market changes when there are many copies of this agent active at the same time. The market was given 1, 2, 4, 8, or 16 copies of the agent. Each episode was tested against the same episode without the copies, using the same random price path. The results showed that in the deep market described in [1], adding 16 copies of the agent made virtually no difference. Stock prices followed the true values slightly better; the distribution of returns became slightly less fat-tailed; and the revenue generated by the copies came from the market maker instead of being taken from the other agents. When the market was shallow (the market maker held a quarter as many shares), the same 16 copies broke the market in 64% of episodes -- more often than any other type of agent (including a momentum agent which traded five times as much). Replaying the copies' recorded trades into the shallow market broke it as well, which shows that it is the trades themselves causing the breakdowns. The agent's strategy depletes the market maker's inventory because it purchases a large block of shares within a few steps -- roughly half of the amount purchased is retained by the agent. Whether or not copying a reinforcement-learning based policy will disrupt a market will depend more on the market than on the policy.