It really depends are you seeing the spikes after the 3rd hop, if not then its not causing a problem during transit only on request, Which is common on some of BT Network Nodes
when you say latency what sort of Latency are you seeing a constant high ping or just an occational Spike?
are you running off wireless or Ethernet?
have you got a copy of the traceroute or Pathping which you can send on?
It isn't necessary to take a new router. I had problems with the delay at https://duckdice.io But it was not on the Internet, just my router settings were lost. I used the instructions and installed new firmware. This saved the situation. I think this problem often arises, but it is most often noticed by gamers.