Not at all, because that's where the agents are. And it's where the agents are because that's where the publishers are. And proximity to publishers is much more important, even these days, than proximity to clients. My sister worked for one in NYC back in the day and her boss represented writers from all over.
If you were writing and marketing a dozen or so screenplays, I'd recommend relocating to LA/Hollywood. For novels, there's no need. Wherever you are is within reach of NY and LA agencies by email or USPS. Make sure you thoroughly check out any agent you submit material to.
To answer your question; No, it does not matter where an agent is located, to the agent...
What does matter is experience and the agents relationship with editors...