{"id":1019,"date":"2024-05-17T13:26:12","date_gmt":"2024-05-17T13:26:12","guid":{"rendered":"https:\/\/blogs.mathworks.com\/finance\/?p=1019"},"modified":"2024-05-30T15:15:57","modified_gmt":"2024-05-30T15:15:57","slug":"deep-learning-in-quantitative-finance-multiagent-reinforcement-learning-for-financial-trading","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/finance\/2024\/05\/17\/deep-learning-in-quantitative-finance-multiagent-reinforcement-learning-for-financial-trading\/","title":{"rendered":"Deep Learning in Quantitative Finance: Multiagent Reinforcement Learning for Financial Trading"},"content":{"rendered":"

\"\"<\/span><\/span><\/em><\/p>\n

The following blog was written by\u00a0Adam Peters<\/a>, Software Engineer<\/span><\/span> at Mathworks.<\/em><\/p>\n

Download<\/strong> the code for this example from Github here<\/a><\/p>\n

Overview: <\/strong><\/p>\n

Financial trading optimization involves developing a strategy that maximizes expected returns among a set of investments. For instance, based off key market indicators, a learned strategy may decide to reallocate, hold, or sell stocks on a day-to-day basis. Reinforcement learning is a common approach to learning an optimal strategy for any problem. In this machine learning paradigm, an agent makes decisions and receives penalties\/rewards, learning to optimize its actions with trial-and-error. Given the inherently competitive nature of the financial market, multiagent reinforcement learning, whereby multiple agents compete in a shared environment, offers a natural and exciting way to develop trading strategies.<\/p>\n

New multiagent functionality has been added to the Reinforcement Learning Toolbox in MATLAB R2023b, allowing you to create agents that can compete, cooperate, take-turns, act simultaneously, share learnings, and more. Using the rlMultiAgentFunctionEnv<\/a> function, one can easily create multiagent environments that seamlessly integrate with the Reinforcement Learning Episode Manager to view your training progress.<\/p>\n

In this blog post, we describe a working demo that uses multiagent reinforcement learning to create optimal trading strategies for three simulated stocks, and demonstrate that competing agents outperform noncompeting agents.<\/p>\n

 <\/p>\n

Agents: <\/strong><\/p>\n

We must first introduce some terminology and define the agents which will learn to trade stocks. In reinforcement learning, agents can be thought of as functions that take in some state, called an observation<\/em>, and then output an action<\/em>. Depending on how this action interacts with the environment<\/em>, they will receive a reward<\/em> for that action, and update their strategy depending on the strength of the reward. This strategy is called a policy, <\/em>and is a function that maps observations to actions. This loop, which can be seen in figure 1, repeats to iteratively update the policy and maximize expected rewards.<\/p>\n

 <\/p>\n

\"\"<\/p>\n

 <\/p>\n

In this example, we will create two Proximal Policy Optimization (PPO) agents<\/a>\u00a0that attempt to outcompete each other, and one PPO agent that acts independently, which will serve as a control. These agents use the PPO neural network as a policy, which was developed by openAI, and is useful in reinforcement learning due to the rate in which they learn\/update. These agents could be defined using the\u00a0Deep Network Designer<\/a>, although this example defines the networks programmatically. The network architecture for each agent follows:<\/p>\n

 <\/p>\n

\"\" \"\"<\/p>\n

 <\/p>\n

Environment: <\/strong><\/p>\n

With the agents defined, we must now define the environment, which consists of an observation space, action space, and reward function.<\/p>\n

 <\/p>\n

Observation Space: <\/u><\/p>\n

Our environment consists of 3 stocks simulated via geometric Brownian motion and $20,000 cash. At each time step, each agent sees 19 different values:<\/p>\n