Most studies examine the effects of stress on memory for visual information test memory for entire scenes. However, arousal levels may differentially influence memory for backgrounds as opposed to items. Participants encoded scenes that included a negative-high-arousal, negative-moderate-arousal, or neutral item on a neutral background. After a 30-minute (Experiment 1) or 48-hour delay (Experiment 2), participants underwent a stressor or control task while heart rate was recorded. A recognition memory task was then given with items and backgrounds presented separately. High-arousal images had a greater detriment in background memory than moderate-arousal images. Further, though there was evidence that change in cortisol level at retrieval was associated with impaired memory for items, it was not associated with detriments in background memory. Increased heart rate was associated with impaired memory for both items and backgrounds. This suggests that the level of sympathetic and cortisol reactivity differentially affects memory for items and backgrounds.