In this paper, we synthesize VLS-grown rough Si nanowires using Mn as a catalyst with various surface roughnesses and diameters and measured their thermal conductivities. We grew the nanowires by a combination vapor-liquid-solid and vapor-solid mechanism for longitudinal and radial growth, respectively. The surface roughness was controlled from smooth up to about 37 nm by the radial growth. Our measurements showed that the thermal conductivity of rough surface Si nanowires is significantly lower than that of smooth surface nanowires and decreased with increasing surface roughness even though the diameter of the smooth nanowire was lower than that of the rough nanowires. Considering both nanowires were grown via the same growth mechanism, these outcomes clearly demonstrate that the rough surface induces phonon scattering and reduces thermal conductivity with this nanoscale-hole-free nanowires. Control of roughness induced phonon scattering in Si nanowires holds promise for novel thermoelectric devices with high figures of merit.