Come to think about it, I’m confused why my ping times are so bad. I have a latency of noticeably well under 2ms to several first hops into the AT&T U-verse network (hops 2, 3*, 4* and 5), yet recently, anything anywhere other than the AT&T routers is always at least 4ms away.

What’s adding those extra 2 to 3 ms of delay between AT&T serving me pretty much in the middle of San Jose, and the other networks within the city limits of, well, you guessed it, still San Jose? If it matters, my ZIP code is 95126 (a 10-minute bike ride to downtown SJ), and, in case anyone missed the prior discussions, my U-verse is powered by Alcatel HONT-C (4 POTS, 1 Ethernet; “155.52 Mbps upstream and 622.08 Mbps downstream”, most certainly 1:32), with the 18Mbps (18/1.5) HSI package (highest available from AT&T over fibre right now, with 24/3 being limited to copper only).

From prior discussion and recollection (http://www.dslreports.com/forum/r26124366-What-has-better-latency-from-a-residential-area-), the best latency I ever got was this summer, at 2.1ms to a USPS site through akamai.net (see a link to my tumblr from the above thread, http://tu.cnst.su/post/8423170265/ping-to-usps-in-under-3ms-from-u-verse-ftth), so I know it should be entirely possible, but no longer happens in practice.

% traceroute -I www.sanjoseca.gov ; date
traceroute to www.sanjoseca.gov (156.39.1.45), 32 hops max, 60 byte packets
 2  76-220-32-3.lightspeed.sntcca.sbcglobal.net (76.220.32.3)  2.985 ms  1.667 ms  1.624 ms
 3  71.145.0.104 (71.145.0.104)  5.641 ms  1.659 ms  1.673 ms
 4  * * *
 5  12.83.39.137 (12.83.39.137)  1.685 ms  1.630 ms  1.588 ms
 6  12.122.200.9 (12.122.200.9)  2.916 ms  2.764 ms  2.830 ms
 7  208.178.58.185 (208.178.58.185)  115.682 ms 192.205.32.50 (192.205.32.50)  199.823 ms dcr2-so-3-0-0.washington.savvis.net (192.205.32.46)  183.584 ms
 8  po1-20G.ar3.SJC2.gblx.net (67.16.134.26)  128.934 ms  394.740 ms  233.524 ms
 9  Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net (64.214.174.246)  4.886 ms  4.904 ms  4.882 ms
10  10gigabitethernet1-4.core1.sjc1.he.net (72.52.92.117)  5.184 ms  5.421 ms  5.008 ms
11  city-of-san-jose.gigabitethernet2-8.core1.sjc1.he.net (64.71.176.98)  6.897 ms  6.839 ms  6.595 ms
12  156.39.0.251 (156.39.0.251)  5.316 ms  5.283 ms  4.951 ms
13  * * *
14  156.39.1.45 (156.39.1.45)  5.929 ms  5.690 ms  5.459 ms
Sat 17 Dec 2011 19:26:20 PST

% traceroute -I www.netflix.com ; date
traceroute to www.netflix.com (69.53.236.17), 32 hops max, 60 byte packets
 2  76-220-32-3.lightspeed.sntcca.sbcglobal.net (76.220.32.3)  4.704 ms  1.657 ms  1.627 ms
 3  * * *
 4  * * *
 5  12.83.39.137 (12.83.39.137)  1.612 ms  1.590 ms  1.854 ms
 6  ggr7.sffca.ip.att.net (12.122.114.13)  2.721 ms  2.562 ms  2.617 ms
 7  xe-1-2-0.mpr4.sjc7.us.above.net (64.125.12.117)  4.580 ms  4.315 ms  4.527 ms
 8  xe-3-1-0.er2.sjc2.us.above.net (64.125.27.94)  5.290 ms  5.437 ms  6.319 ms
 9  64.125.28.13.available.above.net (64.125.28.13)  5.183 ms  4.965 ms  5.081 ms
10  64.124.65.90.allocated.above.net (64.124.65.90)  5.305 ms  5.285 ms  5.162 ms
11  xe-2-3-0-945.jnrt-edge02.sv1.netflix.com (208.75.77.197)  6.222 ms  6.232 ms  6.252 ms
12  xe-2-2-0-955.jnrt-edge02.prod1.netflix.com (69.53.225.30)  7.432 ms  7.377 ms  7.253 ms
13  te1-8.csrt-agg02.prod1.netflix.com (69.53.225.10)  12.506 ms  7.671 ms  7.645 ms
14  netflix.co.uk (69.53.236.17)  7.220 ms  7.242 ms  10.942 ms
Sat 17 Dec 2011 19:27:47 PST

% traceroute -I www.facebook.com ; date
traceroute to www.facebook.com (69.171.228.13), 32 hops max, 60 byte packets
 2  76-220-32-3.lightspeed.sntcca.sbcglobal.net (76.220.32.3)  2.506 ms  10.590 ms  1.785 ms
 3  * * *
 4  * * *
 5  12.83.39.137 (12.83.39.137)  2.205 ms  1.656 ms  1.584 ms
 6  151.164.101.206 (151.164.101.206)  3.870 ms  3.810 ms  3.724 ms
 7  xe-0-2-0-5.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.121)  5.644 ms xe-0-2-0-6.r07.snjsca04.us.bb.gin.ntt.net (129.250.8.93)  5.565 ms xe-0-2-0-7.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.141)  5.723 ms
 8  ae-7.r20.snjsca04.us.bb.gin.ntt.net (129.250.5.52)  5.359 ms  5.358 ms  5.213 ms
 9  ae-4.r05.plalca01.us.bb.gin.ntt.net (129.250.5.32)  6.740 ms  6.442 ms  6.229 ms
10  140.174.21.142 (140.174.21.142)  6.838 ms  6.068 ms  6.574 ms
11  ae1.bb01.pao1.tfbnw.net (74.119.76.134)  6.689 ms  6.445 ms  6.464 ms
12  ae9.bb01.prn1.tfbnw.net (204.15.20.51)  32.184 ms  31.861 ms  31.486 ms
13  ae0.dr04.prn1.tfbnw.net (204.15.23.87)  30.840 ms  31.005 ms  30.862 ms
14  po1022.csw05a.prn1.tfbnw.net (74.119.76.243)  31.777 ms  31.721 ms  31.610 ms
15  www-12-05-prn1.facebook.com (69.171.228.13)  29.830 ms  29.854 ms  30.804 ms
Sat 17 Dec 2011 19:29:29 PST

% traceroute -I 69.22.162.122; date
traceroute to 69.22.162.122 (69.22.162.122), 32 hops max, 60 byte packets
 2  76-220-32-3.lightspeed.sntcca.sbcglobal.net (76.220.32.3)  2.984 ms  1.835 ms  2.272 ms
 3  * * *
 4  71.145.0.80 (71.145.0.80)  4.298 ms * *
 5  12.83.39.137 (12.83.39.137)  1.867 ms  1.653 ms  1.550 ms
 6  ppp-151-164-52-233.rcsntx.swbell.net (151.164.52.233)  4.542 ms  3.660 ms  3.726 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.46.70)  4.776 ms  4.858 ms  4.704 ms
 8  69.22.162.122 (69.22.162.122)  4.007 ms  4.041 ms  4.131 ms
Sat 17 Dec 2011 20:02:59 PST

% traceroute -I trkcnfrm1.smi.usps.com; date
traceroute: Warning: trkcnfrm1.smi.usps.com has multiple addresses; using 63.235.28.96
traceroute to a1834.b.akamai.net (63.235.28.96), 32 hops max, 60 byte packets
 2  76-220-32-3.lightspeed.sntcca.sbcglobal.net (76.220.32.3)  2.254 ms  1.912 ms  1.689 ms
 3  * * *
 4  * * *
 5  12.83.39.137 (12.83.39.137)  1.810 ms  1.505 ms  1.665 ms
 6  ggr7.sffca.ip.att.net (12.122.114.17)  2.846 ms  2.796 ms  2.620 ms
 7  192.205.36.2 (192.205.36.2)  4.413 ms  4.382 ms  4.236 ms
 8  * * *
 9  63-235-28-96.dia.static.qwest.net (63.235.28.96)  5.082 ms  6.201 ms  4.846 ms
Sat 17 Dec 2011 20:03:38 PST

Basically, from the original USPS example, it would seem like nlayer.net had the best latency, and turns out, it is indeed true to this day, too:

% traceroute -I nlayer.net ; date
traceroute to nlayer.net (204.93.207.190), 32 hops max, 60 byte packets
 2  76-220-32-3.lightspeed.sntcca.sbcglobal.net (76.220.32.3)  22.426 ms  3.748 ms  2.055 ms
 3  * * 71.145.0.104 (71.145.0.104)  2.677 ms
 4  * * *
 5  12.83.39.137 (12.83.39.137)  1.768 ms  1.620 ms  1.509 ms
 6  ppp-151-164-52-133.rcsntx.swbell.net (151.164.52.133)  3.638 ms  3.564 ms  3.340 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.251.34)  4.897 ms  4.993 ms  14.479 ms
 8  ae0-80g.cr1.pao1.us.nlayer.net (69.22.153.18)  3.806 ms  3.766 ms  4.081 ms
 9  ae1-60g.cr1.sfo1.us.nlayer.net (69.22.143.169)  10.091 ms  9.223 ms  9.687 ms
10  xe-0-0-3.cr1.slc1.us.nlayer.net (69.22.142.96)  24.748 ms  24.892 ms  25.809 ms
11  xe-5-2-0.cr1.ord1.us.nlayer.net (69.22.142.101)  53.535 ms  53.569 ms  53.602 ms
12  po6.ar1.ord1.us.scnet.net (69.31.111.58)  54.156 ms  53.814 ms  53.741 ms
13  61.po1.ar1.ord6.us.scnet.net (75.102.3.226)  54.864 ms  54.891 ms  54.795 ms
14  ge0-1.aggrGA139-1.ord6.us.scnet.net (75.102.0.146)  55.514 ms  56.915 ms  60.966 ms
15  web1.ord.servercentral.net (204.93.207.190)  54.756 ms  54.974 ms  54.642 ms
Sat 17 Dec 2011 20:25:13 PST

Any ideas why the latency is so bad between the AT&T U-verse network and the internet? Why am I routinely getting 4ms+ instead of that 2.1ms that was once sustainably achieved? Basically, it seems like there is an extra 2.5ms to 3ms delay (translating to 4.0ms and 4.5ms overall delay from my ONT) between the AT&T U-verse routers (hop 5 @ 1.5ms overall) and the actual internet (hop 7 or later @ 4ms+ overall). Why? Where are they losing the 2.5ms? Why did those 2.5ms used to be merely 0.5ms, but no more?

Also, where is my BPON coming from? Is it not the AT&T building at 95 S Almaden Ave, San Jose, CA 95113? I always thought that was their CO, and that BPON users are served right from the CO.

Written for discussion at http://www.dslreports.com/forum/r26669632-FTTH-4ms-ping-to-anywhere-outside-of-the-AT-T-border