Managing audio clips while automating tempo

I'm looking for a way to speed up my workflow, because it feels like I'm doing something wrong.

Basically, I'm working on a song where I automated the tempo. I then recorded a vocal track with the tempo automation on. I like the way it sounds when I play it back, feels very natural.

However, when I go to move the playhead around, the recorded vocal clip changes length dramatically based on the tempo automation clip. This causes the vocal to get out of sync with the rest of the song. If I want everything to stay in sync, I have to put the playhead prior to the tempo keyframe and then wait for the song to play through to the part that I want to listen to. This is really tedious, and makes song creation take much longer.

Is there some way to get the audio clip to stay lined up where it's supposed to, without having to automate the tempo after recording the vocal (and thus making it sound weird)?