In the current study, polyethylene glycol (PEG) was employed to cap super magnetite nanoparticles (Fe₃O₄ NPs) through hydrothermal preparation. The main goal of this study is to investigate the influence of physical incorporation of polyethylene glycol (PEG) loaded Fe₃O₄. The anticancer potentials of these particles were studied against breast cancer cell line (AMJ13). Syntheses bare Fe₃O_4, and Fe₃O₄-PEG were confirmed by TEM, SEM, and FTIR. The size of Fe₃O₄ nanoparticles range of 9–20 and 5–12 nm for the Fe₃O₄–PEG nanoparticles which exerted superparamagnetic properties as well as elevated saturation level of magnetization at ambient conditions. The MTT test was employed to detect the ability of the bare Fe₃O₄ and Fe3O4-PEG on the proliferative of AMJ13 cells. IC₅₀ values was 37.33 µg mL^− 1 for bare Fe₃O₄ and 18.23 µg mL⁻¹ for Fe₃O₄-PEG. AMJ13 Cells exposed to bare Fe₃O_4, and Fe₃O₄-PEG NPs demonstrated a significant cell death, which increased with PEG, loaded Fe₃O₄ NPs. The capability of Fe₃O₄-PEG to induce cellular apoptosis was tested using DAPI, Acridine orange/ Ethedium bromide stains, flow cytometry, with the assays of mitochondrial membrane potential (MMP), and the production of reactive oxygen species (ROS). RT-PCR, and immunofluorescence were performed to measure expression levels of Bax and Bcl-2 proteins. The toxicity of bare Fe₃O₄ and Fe₃O₄-PEG nanoparticles using animal model were investigated. Animal’s body weight, liver and kidney function enzymes, and histological alterations for liver, kidney, and lungs were addressed. The findings demonstrated that nanoparticles were biocompatible with liver and kidney function enzymes and no significant alterations were recorded in the liver, kidney and the lungs. Both nanoparticles revealed a proliferation inhibitory effect on AMJ13 cells, resulting in apoptosis as a novel pathway that involve the mitochondrial damage and NF-kB. Taken together the present data suggest that bare Fe₃O₄ and Fe₃O₄-PEG could be promising therapy protocol for cancer cells.