Finding the optimal association between users and base stations that maximizes the network sum-rate is a complex task. This problem is combinatorial and non-convex, and is even more challenging in millimeter-wave networks due to beamforming, blockages, and severe path loss. Despite the interest that this problem has gained over the last years, the various solutions proposed so far in the literature still fail at being flexible, computationally effective, and suitable to the dynamic nature of mobile networks. This paper addresses these issues with a novel distributed algorithm based on multi-agent reinforcement learning. More specifically, we model each user as an agent, which, at each time step, maps its observations to an action corresponding to an association request to a base station in its coverage range. Our numerical results show that the proposed solution offers near optimal performance and thanks to its flexibility, provides large sum-rate gain with respect to the state-of-art approaches.