Using a reinforcement learning multi-armed bandit (MAB) technique, we design a centralized and a semi-distributed online algorithms, for performing load balancing user association in multi-tier heterogeneous cellular networks. The proposed algorithms guarantee user association solutions that satisfy load balancing constraints among the base stations (BSs) by employing a central load balancer (CLB). At each time step, these algorithms provide real-time associations which give the best-to-date network spectral efficiency. In the centralized approach, the CLB performs base station assignments which determine the action for each user equipment (UE) to update its reward. In the semi-distributed approach, each UE proposes an association action based on its local information and communicates with the BS for an associated reward. Numerical results show that the proposed MAB-based algorithms exhibit fast convergence and reach closely a near-optimal benchmark centralized solution.