Scalable Reinforcement Learning for Linear-Quadratic Control of Networks

Distributed optimal control is known to be challenging and can becomeintractable even for linear-quadratic regulator problems. In this work, westudy a special class of such problems where distributed state feedbackcontrollers can give near-optimal performance. More specifically, we considernetworked linear-quadratic controllers with decoupled costs and spatiallyexponentially decaying dynamics. We aim to exploit the structure in the problemto design a scalable reinforcement learning algorithm for learning adistributed controller. Recent work has shown that the optimal controller canbe well approximated only using information from a κ-neighborhood ofeach agent. Motivated by these results, we show that similar results hold forthe agents' individual value and Q-functions. We continue by designing analgorithm, based on the actor-critic framework, to learn distributedcontrollers only using local information. Specifically, the Q-function isestimated by modifying the Least Squares Temporal Difference for Q-functionsmethod to only use local information. The algorithm then updates the policyusing gradient descent. Finally, we evaluate the algorithm through simulationsthat indeed suggest near-optimal performance.