The effect of internal heat source on the onset of Darcy–Brinkman convection in a porous layer saturated by nanofluid is studied. The boundaries are considered to be free–free, rigid–rigid and lower-rigid and upper-free boundaries. The Brinkman–Darcy equation with fluid viscosity different from effective viscosity is used to characteristic the nanofluid motion. The model used for nanofluid includes the effects of Brownian motion and thermophoresis. The linear stability theory is employed and the resulting eigenvalue problem is solved numerically using the Galerkin technique with the Rayleigh number as the eigenvalue. The influence of internal heat source strength, nanoparticle Rayleigh number, modified particle-density increment, modified diffusivity ratio, Lewis number, Darcy number and the porosity on the stability of the system is investigated graphically. It is found that the internal heat source, nanoparticle Rayleigh number, modified diffusivity ratio and Lewis number have a destabilizing effect while Darcy number and the porosity show stabilizing effects on the system.