Nfo servers use the internap network, which is effectively separate from the large majority of web traffic (which helps eliminate over-saturation from standard web traffic, like people trying to use google/facebook/reddit/etc.), but comes with the drawback that your isp needs to hand over your connection to their network in order for you to connect to the server.
You can see the issue happening with your connection at step 11 where you shift from your isp's network to nfo and gain an extra 20ms of travel time. On normal connections you shouldn't see much of an increase, but if the data center is overloaded or there are other issues affecting the site, you will get what you see happening to you.
I don't know if there is anything you can do to help your problem out, you could try complaining, but I doubt your isp has a different data center to do the hand off for you. Might also be able to complain to nfo/internap as well, maybe enough complaints from your area will get them to open more access points.
If it is your server that you maintain, maybe consider switching to an amazon or google vps, since they will have a direct connection from your isp to server.