During a shift of gaze, an object can move along with gaze or stay fixed in the world. To examine the effect of an object's reference frame on spatial working memory, we trained monkeys to memorize locations of visual stimuli as either fixed in the world or fixed to gaze. Each trial consisted of an initial reference frame instruction, followed by a peripheral visual flash, a memory-period gaze shift, and finally a memory-guided saccade to the location consistent with the instructed reference frame. The memory-period gaze shift was either rapid (a saccade) or slow (smooth pursuit or whole body rotation). This design allowed a comparison of memory-guided saccade performance under various conditions. Our data indicate that after a rotation or smooth-pursuit eye movement, saccades to memorized world-fixed targets are more variable than saccades to memorized gaze-fixed targets. In contrast, memory-guided saccades to world- and gaze-fixed targets are equally variable following a visually guided saccade. Across all conditions, accuracy, latency, and main sequence characteristics of memory-guided saccades are not influenced by the target's reference frame. Memory-guided saccades are, however, more accurate after fast compared with slow gaze shifts. These results are most consistent with an eye-centered representational system for storing the spatial locations of memorized objects but suggest that the visual system may engage different mechanisms to update the stored signal depending on how gaze is shifted.