Quantum speed limit, furnishing a lower bound on the required time for the evolution of a quantum system through the state space, imposes an ultimate natural limitation to the dynamics of physical devices. Quantum absorption refrigerators, however, have attracted a great deal of attention in the past few years. In this paper, we discuss the effects of quantum speed limit on the performance of a quantum absorption refrigerator. In particular, we show that there exists a tradeoff relation between the steady cooling rate of the refrigerator and the minimum time taken to reach the steady state. Based on this, we define a figure of merit called “bounding second order cooling rate” and show that this scales linearly with the unitary interaction strength among the constituent qubits. We also study the increase of bounding second-order cooling rate with the thermalization strength. We subsequently demonstrate that coherence in the initial three qubit system can significantly increase the bounding second-order cooling rate. We study the efficiency of the refrigerator at maximum bounding second-order cooling rate and, in a limiting case, we show that the efficiency at maximum bounding second-order cooling rate is given by a simple formula resembling the Curzon-Ahlborn relation.