Constantine A. Murenin
Posts tagged ‘HE.net’
IPv6 wall of shame: www.bsdvm.com and www.infowars.com, anyone else to nominate?

I’ve recently started noticing that some web-sites stopped working as I’ve switched to IPv6.

Apparently, some idiots think that it’s all right to (malignantly?) publish AAAA records for their WWW domains without sending any replies on the http ports of the respective IPv6 addresses whatsoever. I presume they might have a drop policy on their firewall, provided that the addresses themselves do have IPv6 connectivity.

Prime examples from today are !www.bsdvm.com! and !www.infowars.com! (exclamation marks to avoid this forum from making direct links). I’ve contacted bsdvm via twitter a month or two ago (http://twitter.com/Mcnst/status/171749296599875585), but their web-site is still unreachable from dual-stacked systems. Not sure about infowars.com, but, seriously, WTF?

I hereby nominate both !www.bsdvm.com! and !www.infowars.com! to an IPv6 wall of shame. I have a dual-stacked IPv6 access from AT&T at home and from two reputable IPv6 VPS providers in the cloud, and trying to open the http port on both of these wall-of-shame sites times out from all the 3 sites. At least with bsdvm, it’s surely not a matter of connectivity itself, since they’re actually hosted in Fremont, yet their web-site remains unreachable from within basically the very same DC.


AT&T 6rd support is long as live (a well hidden gem from at&t U-verse)

written for, and discussion at, http://www.tunnelbroker.net/forums/index.php?topic=2293.0

It would appear that AT&T has quietly added IPv6 support through 6rd for, seemingly, ALL of its customers without telling anyone (specifically, without telling any of the said customers).

All existing customers “are not affected” and don’t know about any such support, since it’s not actually supported by most CPE.

Details are here:

http://www.dslreports.com/forum/r26841639-

If you have Motorola NVG510, it’s enabled by default (yes, that’s how we, the customers, found out!). If you have 2Wire PoS, you can still play around and have it configured through your own equipment (and your equipment doesn’t even has to have 6rd support, e.g. OS X and OpenBSD would do just fine for basic 6rd IPv6 connectivity).

Basically, [b]2602:300::/28[/b] (6rdPrefix/6rdPrefixLen) and [b]12.83.49.81[/b] (6rdBRIPv4Address, which is an anycast) is all you need to get it running, IPv4MaskLen is 0 (use the whole IPv4 address within IPv6, but notice that due to 6rdPrefixLen being /28 (instead of the more conventional /32) you have to do some one-nibble shifting, but the plus side is that you do get a /60 in the end). Plus [b]74.82.42.42[/b] (ordns.he.net.), of course. :-) You can also try the standard dnsr{1,2}.sbcglobal.net, but they have a few problems: http://www.dslreports.com/forum/r26902814-IPv6-6rd-DNS-Cannot-resolve-IPv6-only-zones.

This 6rd from AT&T works great in the Bay Area, but I’ve heard that IPv4-wise, Los Angeles gets routed through San Jose, which is a bummer, since IPv6-wise, HE.net is routed through Los Angeles from San Jose AT&T, hence it would appear like people in LA would have to make three full roundtrips (LA -> SJ over IPv4 (1 round), then SJ -> LA -> SJ over IPv6 (2 rounds)) to reach HE’s FMT IPv6 resources over this 6rd. :-)

[b]This is a must-try if you’re on AT&T![/b]

Here’s a collection of my traceroute6’s:

http://www.dslreports.com/forum/r26907835-IPv6-latency-BayArea-Local-IPv6-around-San-Jose-

Just in case you don’t care for external links:
[code]
% traceroute6 www.google.com
Wed 22 Feb 2012 18:38:09 PST
traceroute6 to www.l.google.com (2001:4860:4001:801::1011) from 2602:306:37cY:YYY0::1, 30 hops max, 12 byte packets
1 2602:300:c533:1510::5 (2602:300:c533:1510::5) 1.579 ms 1.368 ms 1.309 ms
2 sj2ca404me3.ip.att.net (2001:1890:ff:ffff:12:122:119:192) 4.014 ms 4.042 ms 4.007 ms
3 2001:1890:c00:8c02::1116:e9cb (2001:1890:c00:8c02::1116:e9cb) 54.538 ms 55.262 ms 55.846 ms
4 2001:4860::1:0:21 (2001:4860::1:0:21) 5.572 ms 5.374 ms 5.154 ms
5 2001:4860:0:1::1af (2001:4860:0:1::1af) 5.425 ms 5.55 ms 5.343 ms
6 2001:4860:8000:e:92e6:baff:fe53:a202 (2001:4860:8000:e:92e6:baff:fe53:a202) 56.341 ms 57.198 ms 56.167 ms
[/code]

Here’s my traceroute / mtr to the 6rdBRIPv4Address:
[code]
% traceroute -I 12.83.49.81
Wed 22 Feb 2012 18:47:16 PST
traceroute to 12.83.49.81 (12.83.49.81), 32 hops max, 60 byte packets
5 12.83.39.137 (12.83.39.137) 3.581 ms 1.700 ms 1.568 ms
6 12.83.49.81 (12.83.49.81) 1.311 ms 1.310 ms 1.107 ms
% mtr -c60 —report{,-wide} 12.83.49.81
Wed 22 Feb 2012 18:49:30 PST
HOST: 99-124-xxx-xxx.uvs.sntcca.sbcglobal.net Loss% Snt Last Avg Best Wrst StDev
1.|— 99-124-xxx-xxx.uvs.sntcca.sbcglobal.net 0.0% 60 0.8 0.4 0.3 0.8 0.1
2.|— 76-220-32-3.lightspeed.sntcca.sbcglobal.net 76.7% 60 2.2 2.5 1.9 5.1 0.9
3.|— 71.145.0.104 98.3% 60 3.3 3.3 3.3 3.3 0.0
4.|— 71.145.0.80 96.7% 60 6.7 6.8 6.7 6.9 0.1
5.|— 12.83.39.137 0.0% 60 1.7 2.1 1.7 2.8 0.2
6.|— 12.83.49.81 0.0% 60 1.6 1.4 1.2 1.6 0.1
[/code]

Here’s traceroute6 to he.net and back (as I mentioned, it goes SJC -> LAX -> FMT, since LA is the closest point where HE and AT&T do IPv6 peering):

[code]
% traceroute6 ns2.linode.com
Wed 22 Feb 2012 19:07:19 PST
traceroute6 to ns2.linode.com (2600:3c01::a) from 2602:306:37cY:YYY0::1, 30 hops max, 12 byte packets
1 2602:300:c533:1510::5 (2602:300:c533:1510::5) 1.579 ms 1.2 ms 1.315 ms
2 la2ca02jt.ip.att.net (2001:1890:ff:ffff:12:122:127:43) 129.526 ms 13.831 ms 24.7 ms
3 10gigabitethernet5-2.core1.lax2.he.net (2001:470:0:1e6::1) 13.755 ms 13.938 ms 13.77 ms
4 10gigabitethernet2-1.core1.lax1.he.net (2001:470:0:72::1) 13.843 ms 13.829 ms 13.787 ms
5 10gigabitethernet7-4.core1.fmt2.he.net (2001:470:0:18d::1) 24.908 ms 21.798 ms 21.9 ms
6 gige-g4-18.core1.fmt1.he.net (2001:470:0:2d::1) 22.01 ms 21.99 ms
linode-llc.10gigabitethernet2-3.core1.fmt1.he.net (2001:470:1:1db::2) 22.852 ms
7 ns2.linode.com (2600:3c01::a) 22.439 ms 22.319 ms 22.335 ms
[/code]

[code]
# traceroute6 2602:306:37cY:YYY0::1
traceroute to 2602:306:37cY:YYY0::1 (2602:306:37cY:YYY0::1), 16 hops max, 80 byte packets
2 10gigabitethernet2-3.core1.fmt1.he.net (2001:470:1:1db::1) [AS6939] 5.666 ms 5.782 ms 5.760 ms
3 gige-g4-8.core1.fmt2.he.net (2001:470:0:2d::2) [AS6939] 0.454 ms 0.443 ms 0.532 ms
4 10gigabitethernet6-4.core1.lax1.he.net (2001:470:0:18d::2) [AS6939] 8.514 ms 8.500 ms 8.474 ms
5 10gigabitethernet1-3.core1.lax2.he.net (2001:470:0:72::2) [AS6939] 15.265 ms 8.905 ms 14.903 ms
6 att-internet4-as7018.10gigabitethernet5-2.core1.lax2.he.net (2001:470:0:1e6::2) [AS6939] 8.915 ms 8.988 ms 9.052 ms
7 * * *
8 * * *
9 2001:1890:ff:ffff:12:122:114:41 (2001:1890:ff:ffff:12:122:114:41) [AS7018] 22.141 ms 22.114 ms 22.078 ms
10 2602:300:c533:1510::5 (2602:300:c533:1510::5) [*] 21.171 ms 21.174 ms 21.148 ms
11 * * *
12 * * *
13 * * *
14 2602:300:c533:1510::6 (2602:300:c533:1510::6) [*] 22.154 ms 22.425 ms 22.649 ms
15 2602:300:c533:1510::5 (2602:300:c533:1510::5) [*] 22.280 ms 22.709 ms 23.197 ms
16 2602:306:37cY:YYY0::1 (2602:306:37cY:YYY0::1) [*] 22.543 ms 22.710 ms 23.209 ms
[/code]

Try the following to get your IPv6 prefix from the IPv4 address:

[code]
% printf “%02x%02x%02x%02x” 99 124 xxx xxx | awk ‘{print “2602:30” substr($1,1,1) “:” substr($1,2,4) “:” substr($1,6) “0::/60”}’
2602:306:37cY:YYY0::/60
[/code]

What would be interesting to know are the locations of where 6rdBR’s have been deployed. So far, it’s known that LA uses the one in San Jose, so there doesn’t appear to be a 6rdBR in LA. You can guesstimate the location of your 6rdBR by doing `traceroute 12.83.49.81` from your local AT&T network, or by doing a traceroute6 of your 6rd IPv6 address from a remote network, seeing the latency of the last hop of the 2602:300:c533:1510::/60 origin (which is, you’ve guessed it, the anycast 6rd prefix of the 6rdBR’s 12.83.49.81). Your total latency for a given IPv6 resource over this 6rd would be the sum of these two latencies around the 6rdBR anycast (12.83.49.81 / 2602:300:c533:1510::/60).

Happy 6rd tunnelling!

P.S. I should warn you that this whole info is based solely on user-submitted “reverse-engineered” information. I should also warn you that if you ever decide to go to AT&T’s corporate IPv6 web-site (the link for which is very short and memorable, but is intentionally omitted from this post such as to not waste people’s time), that you will not find a single useful piece of information about anything whatsoever, and the time that you will lose you’ll never get back! (-:


HE.net / AT&T connectivity woes continue
As reported earlier, HE.net / AT&T connectivity woes continue.
# mtr --report{,-wide} 76-220-32-XX.lightspeed.sntcca.sbcglobal.net ; date
HOST: liXXX-XXX                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 184.105.143.85                                0.0%    10    0.6   0.6   0.4   1.2   0.2
  2. 10gigabitethernet2-3.core1.fmt1.he.net        0.0%    10    0.6   5.2   0.5  11.9   4.8
  3. 10gigabitethernet1-1.core1.pao1.he.net        0.0%    10   11.4   4.3   0.8  11.4   3.4
  4. sjo-bb1-link.telia.net                        0.0%    10    1.3   3.1   1.1  19.8   5.9
  5. las-bb1-link.telia.net                        0.0%    10   14.1  15.0  14.1  22.7   2.7
  6. att-ic-149001-las-bb1.c.telia.net            30.0%    10   23.6  32.0  23.6  38.5   5.0
  7. cr2.la2ca.ip.att.net                         50.0%    10   35.0  34.8  30.0  39.7   4.1
  8. cr2.sffca.ip.att.net                         30.0%    10   32.5  36.0  30.2  41.3   4.0
  9. 12.122.149.141                               50.0%    10   27.3  35.9  27.3  40.4   5.3
 10. ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
 11. ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
 12. ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
 13. 76-220-32-XX.lightspeed.sntcca.sbcglobal.net 80.0%    10   31.1  34.4  31.1  37.8   4.8
Wed Dec 28 21:17:53 EST 2011
  2.|-- 76-220-32-3.lightspeed.sntcca.sbcglobal.net             10.0%    10    2.2   2.6   2.2   3.6   0.4
  3.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- 12.83.39.137                                             0.0%    10    1.8   1.9   1.7   2.3   0.2
  6.|-- 12.122.200.9                                             0.0%    10   45.4  17.5   2.9 105.5  33.7
  7.|-- 208.51.134.1                                             0.0%    10    4.9  10.5   4.9  58.4  16.8
    |  `|-- 192.205.32.46
    |   |-- 192.205.32.50
    |   |-- 208.178.58.185
  8.|-- po1-20G.ar3.SJC2.gblx.net                                0.0%    10    5.3   5.1   5.0   5.3   0.1
  9.|-- Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 10.0%    10   40.7  42.3  36.2  49.4   4.4
 10.|-- 10gigabitethernet1-1.core1.fmt1.he.net                  50.0%    10   40.3  47.4  40.3  51.9   4.3
 11.|-- linode-llc.10gigabitethernet2-3.core1.fmt1.he.net       60.0%    10   40.8  38.9  36.7  40.8   1.7
 12.|-- li163-159.members.linode.com                            20.0%    10   39.4  39.4  38.2  41.2   1.1
Wed 28 Dec 2011 18:18:22 PST
  2.|-- 76-220-32-3.lightspeed.sntcca.sbcglobal.net 10.0%    10    2.2   3.4   2.2  12.3   3.3
  3.|-- ???                                         100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- 71.145.0.80                                 90.0%    10    4.6   4.6   4.6   4.6   0.0
  5.|-- 12.83.39.137                                 0.0%    10    1.7   3.4   1.7  17.6   5.0
  6.|-- 12.122.137.125                               0.0%    10    2.8  19.8   2.8  47.5  16.0
  7.|-- att-gw.sanjoseequinix.savvis.net             0.0%    10    4.6  10.5   4.5  61.3  17.9
  8.|-- er1-te-3-1.SanJoseEquinix.savvis.net         0.0%    10    4.6  24.4   4.5 169.3  52.0
  9.|-- cr1-tenge-0-3-5-0.sanfrancisco.savvis.net    0.0%    10    7.0   6.6   6.3   7.0   0.2
 10.|-- cr1-tengig-0-0-2-0.losangeles.savvis.net     0.0%    10   16.4  16.4  15.7  17.1   0.4
 11.|-- ber1-tenge-2-1.losangeles.savvis.net         0.0%    10   15.7  16.6  15.6  23.4   2.4
 12.|-- 209.144.203.214                              0.0%    10   16.3  16.2  15.1  17.1   0.6
 13.|-- arpnetworks-lax2-gw.cust.trit.net            0.0%    10   33.8  18.9  16.7  33.8   5.3
 14.|-- arpnetworks.com                              0.0%    10   16.8  16.8  16.4  17.1   0.2
Wed 28 Dec 2011 18:20:24 PST
  2.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- 12.83.39.137                                             0.0%    10    1.7   1.9   1.6   2.3   0.2
  6.|-- 12.122.200.9                                             0.0%    10    2.9   8.1   2.9  41.8  12.5
  7.|-- 208.51.134.1                                             0.0%    10    5.0  62.9   5.0 199.6  69.7
    |  `|-- 192.205.32.46
    |   |-- 192.205.32.50
    |   |-- 208.178.58.185
  8.|-- po1-20G.ar3.SJC2.gblx.net                                0.0%    10    5.5  57.7   4.9 228.9  87.6
  9.|-- Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 30.0%    10   38.0  39.1  33.9  47.7   4.5
 10.|-- 10gigabitethernet1-1.core1.fmt1.he.net                  30.0%    10   43.5  45.6  33.1  92.9  21.2
 11.|-- linode-llc.10gigabitethernet2-3.core1.fmt1.he.net       60.0%    10   34.2  37.7  34.2  40.3   2.6
 12.|-- li163-159.members.linode.com                            20.0%    10   38.1  36.7  32.7  40.1   2.6
Wed 28 Dec 2011 18:21:00 PST

Somehow no noticeable ssh packet loss whatsoever, though.


disappointed in Hurricane Electric latency

Meh. For a few hours now, seems like HE has a latency problem; about 17ms to Fremont from San Jose, instead of the usual ~5.5ms. Even some packet loss, meh.

% traceroute fremont1.linode.com ; date
traceroute to fremont1.linode.com (64.71.152.17), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.387 ms  1.964 ms  1.784 ms
 6  12.122.200.9 (12.122.200.9)  2.974 ms  3.031 ms  3.020 ms
 7  dcr2-so-3-0-0.washington.savvis.net (192.205.32.46)  5.060 ms 208.178.58.185 (208.178.58.185)  5.006 ms 192.205.32.50 (192.205.32.50)  4.965 ms
 8  po1-20G.ar3.SJC2.gblx.net (67.16.134.26)  5.210 ms  5.201 ms  5.012 ms
 9  Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net (64.214.174.246)  26.734 ms  25.329 ms  25.379 ms
10  10gigabitethernet1-1.core1.fmt1.he.net (72.52.92.109)  18.158 ms  17.942 ms  21.013 ms
11  linode-llc.10gigabitethernet2-3.core1.fmt1.he.net (64.62.250.6)  17.133 ms  17.523 ms  17.248 ms
12  fremont1.linode.com (64.71.152.17)  18.028 ms  17.511 ms  18.120 ms
Fri 23 Dec 2011 19:29:24 PST
% ping -c8 fremont1.linode.com; date
PING fremont1.linode.com (64.71.152.17): 56 data bytes
64 bytes from 64.71.152.17: icmp_seq=0 ttl=51 time=17.439 ms
64 bytes from 64.71.152.17: icmp_seq=1 ttl=51 time=17.319 ms
64 bytes from 64.71.152.17: icmp_seq=2 ttl=51 time=17.650 ms
64 bytes from 64.71.152.17: icmp_seq=3 ttl=51 time=19.009 ms
64 bytes from 64.71.152.17: icmp_seq=4 ttl=51 time=18.353 ms
64 bytes from 64.71.152.17: icmp_seq=5 ttl=51 time=19.261 ms
64 bytes from 64.71.152.17: icmp_seq=7 ttl=51 time=17.059 ms

--- fremont1.linode.com ping statistics ---
8 packets transmitted, 7 packets received, 12% packet loss
round-trip min/avg/max/stddev = 17.059/18.013/19.261/0.803 ms
Fri 23 Dec 2011 19:34:53 PST

To be fair, though, seems like HE is not the only one that has poor latency this time of the year — NAC in Newark, with nlayer.net, is also lagging noticeably:

% date ; echo {fremont,dallas,atlanta,newark,tokyo,london}1.linode.com | xargs -tn1 /usr/sbin/traceroute -M5 -w2 -m32 ; date
Fri 23 Dec 2011 19:44:03 PST
/usr/sbin/traceroute -M5 -w2 -m32 fremont1.linode.com
traceroute to fremont1.linode.com (64.71.152.17), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.455 ms  1.768 ms  1.801 ms
 6  12.122.200.9 (12.122.200.9)  159.361 ms  200.101 ms  14.856 ms
 7  192.205.32.50 (192.205.32.50)  47.598 ms 208.51.134.1 (208.51.134.1)  211.444 ms dcr2-so-3-0-0.washington.savvis.net (192.205.32.46)  5.488 ms
 8  po1-20G.ar3.SJC2.gblx.net (67.16.134.26)  4.962 ms  5.050 ms  5.161 ms
 9  Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net (64.214.174.246)  17.085 ms  19.466 ms  24.127 ms
10  10gigabitethernet1-1.core1.fmt1.he.net (72.52.92.109)  17.120 ms  18.184 ms  25.457 ms
11  linode-llc.10gigabitethernet2-3.core1.fmt1.he.net (64.62.250.6)  17.801 ms  17.672 ms  17.424 ms
12  fremont1.linode.com (64.71.152.17)  17.154 ms  17.945 ms  19.139 ms
/usr/sbin/traceroute -M5 -w2 -m32 dallas1.linode.com
traceroute to dallas1.linode.com (69.164.200.100), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.135 ms  1.828 ms  1.619 ms
 6  151.164.101.206 (151.164.101.206)  3.825 ms  3.844 ms  3.837 ms
 7  xe-0-2-0-5.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.121)  5.160 ms ppp-151-164-46-66.eulstx.swbell.net (151.164.46.66)  5.176 ms xe-0-2-0-7.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.141)  5.465 ms
 8  ae1.bbr02.eq01.sjc01.networklayer.com (128.241.219.234)  7.555 ms  5.222 ms  5.302 ms
 9  ae0.bbr02.cs01.lax01.networklayer.com (173.192.18.151)  15.055 ms  14.084 ms  13.686 ms
10  ae7.bbr01.cs01.lax01.networklayer.com (173.192.18.166)  13.854 ms  13.846 ms  13.502 ms
11  ae19.bbr01.eq01.dal03.networklayer.com (173.192.18.140)  46.931 ms  84.632 ms  46.854 ms
12  po31.dsr02.dllstx3.networklayer.com (173.192.18.227)  49.257 ms po31.dsr01.dllstx3.networklayer.com (173.192.18.225)  44.207 ms po31.dsr02.dllstx3.networklayer.com (173.192.18.227)  46.194 ms
13  * * *
14  po1.car01.dllstx2.networklayer.com (70.87.254.74)  46.798 ms po2.car01.dllstx2.networklayer.com (70.87.254.78)  46.925 ms po1.car01.dllstx2.networklayer.com (70.87.254.74)  46.261 ms
15  5a.7.1243.static.theplanet.com (67.18.7.90)  46.774 ms  43.758 ms  49.258 ms
16  dallas1.linode.com (69.164.200.100)  44.063 ms  44.278 ms  47.010 ms
/usr/sbin/traceroute -M5 -w2 -m32 atlanta1.linode.com
traceroute to atlanta1.linode.com (63.247.71.196), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.364 ms  1.770 ms  1.739 ms
 6  ppp-151-164-52-233.rcsntx.swbell.net (151.164.52.233)  3.896 ms  3.804 ms  3.910 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.46.70)  6.494 ms  4.967 ms  5.768 ms
 8  ae0-80g.cr1.pao1.us.nlayer.net (69.22.153.18)  3.798 ms  3.730 ms  3.840 ms
 9  ae1-50g.cr1.sjc1.us.nlayer.net (69.22.143.165)  5.440 ms  5.269 ms  4.353 ms
10  xe-3-3-1.cr1.lax1.us.nlayer.net (69.22.142.8)  13.218 ms xe-5-1-0.cr1.lax1.us.nlayer.net (69.22.142.126)  13.739 ms xe-3-3-1.cr1.lax1.us.nlayer.net (69.22.142.8)  13.217 ms
11  xe-1-0-0.cr1.iah1.us.nlayer.net (69.22.142.121)  48.768 ms  48.500 ms  48.476 ms
12  xe-4-2-1.cr1.atl1.us.nlayer.net (69.22.142.118)  71.607 ms  71.570 ms  71.975 ms
13  ae1-40g.ar1.atl1.us.nlayer.net (69.31.135.130)  65.750 ms  67.621 ms  67.644 ms
14  as3595.xe-2-0-5-101.ar1.atl1.us.nlayer.net (69.31.135.42)  133.773 ms  98.646 ms  80.123 ms
15  64.22.106.10 (64.22.106.10)  63.216 ms  63.468 ms  66.714 ms
16  atlanta1.linode.com (63.247.71.196)  63.546 ms  63.098 ms  63.590 ms
/usr/sbin/traceroute -M5 -w2 -m32 newark1.linode.com
traceroute to newark1.linode.com (207.192.68.6), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.245 ms  1.696 ms  1.635 ms
 6  ex2-p10-0.pxpaca.sbcglobal.net (151.164.190.83)  3.612 ms  3.633 ms  3.529 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.251.34)  4.849 ms  7.351 ms  4.755 ms
 8  ae0-80g.cr1.pao1.us.nlayer.net (69.22.153.18)  3.737 ms  10.857 ms  3.664 ms
 9  ae1-60g.cr1.sfo1.us.nlayer.net (69.22.143.169)  4.329 ms  4.346 ms  4.346 ms
10  xe-0-0-3.cr1.slc1.us.nlayer.net (69.22.142.96)  19.397 ms  19.331 ms  19.394 ms
11  xe-5-2-0.cr1.ord1.us.nlayer.net (69.22.142.101)  53.983 ms  53.913 ms  53.920 ms
12  xe-9-2-0.cr1.ewr1.us.nlayer.net (69.22.142.75)  118.761 ms  118.864 ms  120.644 ms
13  ae1-70g.ar2.ewr1.us.nlayer.net (69.31.94.118)  122.490 ms  105.794 ms  120.565 ms
14  as8001.xe-3-0-6.ar2.ewr1.us.nlayer.net (69.31.95.130)  104.817 ms  104.694 ms  104.436 ms
15  0.e1-2.tbr1.mmu.nac.net (209.123.10.118)  118.743 ms  119.052 ms  120.066 ms
16  vlan803.esd2.mmu.nac.net (209.123.10.30)  73.661 ms  73.959 ms  75.223 ms
17  207.99.53.46 (207.99.53.46)  106.657 ms  109.364 ms  104.719 ms
18  newark1.linode.com (207.192.68.6)  116.415 ms  116.762 ms  118.573 ms
/usr/sbin/traceroute -M5 -w2 -m32 tokyo1.linode.com
traceroute to tokyo1.linode.com (106.187.33.12), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.321 ms  1.849 ms  1.613 ms
 6  ggr7.sffca.ip.att.net (12.122.114.17)  2.985 ms  2.807 ms  2.836 ms
 7  192.205.36.2 (192.205.36.2)  4.349 ms  4.458 ms  4.498 ms
 8  pax-brdr-01.inet.qwest.net (205.171.234.2)  5.937 ms  5.873 ms  5.927 ms
 9  124.215.192.77 (124.215.192.77)  23.994 ms  17.741 ms  17.895 ms
10  pajbb002.kddnet.ad.jp (111.87.3.29)  120.640 ms pajbb002.kddnet.ad.jp (124.211.34.133)  16.082 ms pajbb001.kddnet.ad.jp (111.87.3.5)  84.107 ms
11  otejbb203.kddnet.ad.jp (203.181.100.5)  131.326 ms otejbb203.kddnet.ad.jp (203.181.100.209)  119.472 ms otejbb204.kddnet.ad.jp (203.181.100.201)  119.155 ms
12  cm-fcu203.kddnet.ad.jp (124.215.194.164)  121.772 ms  125.743 ms cm-fcu203.kddnet.ad.jp (124.215.194.180)  133.351 ms
13  124.215.199.122 (124.215.199.122)  113.014 ms  115.081 ms  123.389 ms
14  tokyo1.linode.com (106.187.33.12)  120.650 ms  112.850 ms  114.916 ms
/usr/sbin/traceroute -M5 -w2 -m32 london1.linode.com
traceroute to london1.linode.com (109.74.207.9), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.025 ms  13.973 ms  1.969 ms
 6  12.122.200.9 (12.122.200.9)  2.886 ms  2.792 ms  2.951 ms
 7  192.205.36.234 (192.205.36.234)  5.252 ms  4.845 ms  5.133 ms
 8  telecity.ge9-9.br02.ldn01.pccwbtn.net (63.218.13.222)  146.812 ms  146.863 ms  146.933 ms
 9  te4-1-dist65-01.lon10.telecity.net (217.20.44.218)  148.074 ms  148.169 ms  148.923 ms
10  212.111.33.234 (212.111.33.234)  148.846 ms  149.418 ms  148.772 ms
11  london1.linode.com (109.74.207.9)  149.089 ms  148.989 ms  149.142 ms
Fri 23 Dec 2011 19:44:19 PST

Just for the full picture:

% traceroute sonic.net ; date
traceroute to sonic.net (209.204.190.64), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.126 ms  1.751 ms  1.617 ms
 6  ppp-151-164-52-133.rcsntx.swbell.net (151.164.52.133)  3.852 ms  3.678 ms  3.867 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.251.34)  10.775 ms  4.978 ms  7.045 ms
 8  as7065.xe-1-0-6.ar1.pao1.us.nlayer.net (69.22.130.86)  3.723 ms  3.815 ms  3.542 ms
 9  0.ge-0-1-0.gw.sr.sonic.net (64.142.0.197)  6.273 ms  6.259 ms  6.262 ms
10  gig0-1.dist2-1.sr.sonic.net (209.204.191.160)  6.222 ms gig0-2.dist2-1.sr.sonic.net (208.201.224.160)  6.430 ms  6.510 ms
11  fast1.lb2-1.slb.sr.sonic.net (64.142.0.58)  6.376 ms  6.432 ms  6.453 ms
12  fast1.lb2-1.slb.sr.sonic.net (64.142.0.58)  6.567 ms  6.467 ms  6.449 ms
Fri 23 Dec 2011 19:47:05 PST
% traceroute netflix.com ; date
traceroute to netflix.com (69.53.236.17), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.139 ms  1.724 ms  1.667 ms
 6  ggr7.sffca.ip.att.net (12.122.114.13)  3.611 ms  36.855 ms  47.582 ms
 7  xe-1-2-0.mpr4.sjc7.us.above.net (64.125.12.117)  4.671 ms  4.588 ms  4.594 ms
 8  xe-1-0-0.mpr3.sjc7.us.above.net (64.125.26.125)  4.890 ms  4.844 ms  4.910 ms
 9  xe-4-0-0.er1.sjc2.us.above.net (64.125.30.173)  4.864 ms  4.975 ms  4.877 ms
10  64.124.65.90.allocated.above.net (64.124.65.90)  5.121 ms  5.052 ms  4.925 ms
11  xe-2-3-0-945.jnrt-edge02.sv1.netflix.com (208.75.77.197)  6.108 ms  6.032 ms  5.972 ms
12  xe-2-2-0-955.jnrt-edge02.prod1.netflix.com (69.53.225.30)  7.189 ms  17.194 ms  7.141 ms
13  te1-8.csrt-agg02.prod1.netflix.com (69.53.225.10)  7.532 ms  7.335 ms  7.367 ms
14  netflixfreetrial.com (69.53.236.17)  7.237 ms  7.172 ms  7.199 ms
Fri 23 Dec 2011 19:47:31 PST
% traceroute facebook.com ; date
traceroute: Warning: facebook.com has multiple addresses; using 66.220.149.11
traceroute to facebook.com (66.220.149.11), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.108 ms  2.252 ms  1.844 ms
 6  151.164.101.206 (151.164.101.206)  3.981 ms  4.046 ms  3.954 ms
 7  xe-0-2-0-6.r07.snjsca04.us.bb.gin.ntt.net (129.250.8.93)  6.051 ms xe-0-2-0-7.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.141)  5.274 ms ppp-151-164-46-66.eulstx.swbell.net (151.164.46.66)  5.495 ms
 8  ae-8.r21.snjsca04.us.bb.gin.ntt.net (129.250.5.56)  5.471 ms ae-7.r20.snjsca04.us.bb.gin.ntt.net (129.250.5.52)  5.731 ms ae-8.r21.snjsca04.us.bb.gin.ntt.net (129.250.5.56)  5.523 ms
 9  ae-4.r05.plalca01.us.bb.gin.ntt.net (129.250.5.32)  6.387 ms ae-4.r06.plalca01.us.bb.gin.ntt.net (129.250.4.118)  6.551 ms ae-4.r05.plalca01.us.bb.gin.ntt.net (129.250.5.32)  6.656 ms
10  ae-2.r05.plalca01.us.bb.gin.ntt.net (129.250.5.237)  6.367 ms 140.174.21.142 (140.174.21.142)  6.151 ms ae-2.r05.plalca01.us.bb.gin.ntt.net (129.250.5.237)  6.629 ms
11  ae1.bb01.pao1.tfbnw.net (74.119.76.134)  7.840 ms 140.174.21.142 (140.174.21.142)  6.885 ms ae1.bb01.pao1.tfbnw.net (74.119.76.134)  6.481 ms
12  ae1.bb01.pao1.tfbnw.net (74.119.76.134)  6.651 ms ae2.dr02.snc5.tfbnw.net (74.119.77.192)  6.813 ms ae1.bb01.pao1.tfbnw.net (74.119.76.134)  6.498 ms
13  po509.csw02b.snc5.tfbnw.net (74.119.78.28)  7.216 ms ae0.dr01.snc5.tfbnw.net (74.119.77.180)  7.469 ms po509.csw02a.snc5.tfbnw.net (74.119.78.24)  7.063 ms
14  po510.csw02b.snc5.tfbnw.net (74.119.78.26)  7.358 ms * po509.csw02a.snc5.tfbnw.net (74.119.78.24)  7.916 ms
15  * * *
16  *^C
% traceroute google.com ; date
traceroute: Warning: google.com has multiple addresses; using 74.125.224.147
traceroute to google.com (74.125.224.147), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.503 ms  1.851 ms  1.622 ms
 6  ppp-151-164-52-163.rcsntx.swbell.net (151.164.52.163)  4.657 ms  4.581 ms  4.528 ms
 7  gar7.sffca.ip.att.net (12.122.79.97)  4.569 ms  4.463 ms  4.628 ms
 8  cr83.sffca.ip.att.net (12.122.110.118)  7.015 ms  6.648 ms  6.170 ms
 9  cr1.sffca.ip.att.net (12.123.15.109)  7.134 ms  8.057 ms  7.767 ms
10  cr81.sj2ca.ip.att.net (12.122.1.118)  7.327 ms  7.347 ms  6.838 ms
11  12.122.114.21 (12.122.114.21)  5.999 ms  5.853 ms  5.662 ms
12  12.249.231.14 (12.249.231.14)  6.488 ms  6.477 ms  6.438 ms
13  209.85.249.3 (209.85.249.3)  7.001 ms  7.207 ms  6.940 ms
14  64.233.174.119 (64.233.174.119)  7.518 ms  7.382 ms  7.297 ms
15  nuq04s09-in-f19.1e100.net (74.125.224.147)  7.108 ms  6.924 ms  8.550 ms
Fri 23 Dec 2011 19:49:05 PST
% traceroute paxio.net ; date
traceroute to paxio.net (63.216.71.33), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.140 ms  1.777 ms  1.658 ms
 6  ggr6.wswdc.ip.att.net (12.122.86.89)  3.224 ms  15.199 ms  3.122 ms
 7  192.205.32.206 (192.205.32.206)  5.203 ms  5.082 ms  5.015 ms
 8  vlan80.csw3.SanJose1.Level3.net (4.69.152.190)  5.785 ms vlan90.csw4.SanJose1.Level3.net (4.69.152.254)  5.753 ms  5.649 ms
 9  ae-72-72.ebr2.SanJose1.Level3.net (4.69.153.21)  7.140 ms  7.352 ms ae-62-62.ebr2.SanJose1.Level3.net (4.69.153.17)  5.706 ms
10  ae-5-5.car1.Oakland1.Level3.net (4.69.134.37)  6.833 ms  6.644 ms  6.809 ms
11  ae-11-11.car2.Oakland1.Level3.net (4.69.134.42)  6.831 ms  6.705 ms  6.762 ms
12  PAXIO-INC.car2.Oakland1.Level3.net (4.71.202.38)  7.325 ms  7.359 ms  7.248 ms
13  173.241.16.50 (173.241.16.50)  7.468 ms  7.865 ms  7.492 ms
14  63-216-64-246.paxio.net (63.216.64.246)  7.281 ms  7.022 ms  7.169 ms
15  64-201-240-174.static.paxio.net (64.201.240.174)  7.190 ms  6.926 ms  7.955 ms
16  64-201-240-182.static.paxio.net (64.201.240.182)  8.424 ms  9.611 ms  8.657 ms
17  64-201-240-170.static.paxio.net (64.201.240.170)  8.724 ms  8.196 ms  8.426 ms
18  * * *
19  *^C

Somehow, Google and Facebook don’t seem to have any latency issues whatsoever, yet HE.net, whose sole purpose is IP transit, can’t get their latency within the Bay Area alone! WTF? NAC, a famous data centre in Newark, with its nlayer.net transit, also disappoints.

P.S. Hm, svcolo.com seems good, through nlayer.net, the latency may perhaps go as low as 3.5ms during non-peak hours, now about 4.0ms:

% traceroute svcolo.com ; date
traceroute to svcolo.com (64.13.135.35), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.181 ms  1.797 ms  1.746 ms
 6  151.164.54.183 (151.164.54.183)  3.800 ms  3.446 ms  3.567 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.46.70)  8.373 ms  7.769 ms  5.264 ms
 8  ae0-80g.cr1.pao1.us.nlayer.net (69.22.153.18)  64.425 ms  3.518 ms  3.634 ms
 9  as8121.ae0-3001.cr1.pao1.us.nlayer.net (69.22.153.114)  3.907 ms  3.849 ms  3.850 ms
10  xe1-4.core1.svk.layer42.net (69.36.239.121)  4.316 ms  4.163 ms  4.125 ms
11  www.svcolo.com (64.13.135.35)  3.969 ms  4.054 ms  4.050 ms
Fri 23 Dec 2011 21:16:51 PST

P.P.S. Apparently, according to http://lg.he.net/, the issue is that the traffic back to my place, goes through telia in Los Angeles. WTF?

core1.fmt1.he.net> traceroute 76.220.??.?? source-ip 216.218.252.161 numeric
Target	76.220.32.89
Hop Start	1
Hop End	30
Hop	Packet 1	Packet 2	Packet 3	Hostname
1	14 ms	1 ms	<1 ms	10gigabitethernet1-1.core1.pao1.he.net (184.105.213.66)
2	1 ms	1 ms	1 ms	sjo-bb1-link.telia.net (213.248.86.53)
3	14 ms	24 ms	49 ms	las-bb1-link.telia.net (213.248.80.17)
4	25 ms	24 ms	24 ms	att-ic-149001-las-bb1.c.telia.net (213.248.88.30)
5	25 ms	25 ms	24 ms	cr2.la2ca.ip.att.net (12.123.30.150)
6	26 ms	24 ms	24 ms	cr2.sffca.ip.att.net (12.122.31.134)
7	24 ms	25 ms	26 ms	12.122.149.141
8	*	*	*	?
9	*	*	*	?
10	*	*	*	?
11	74 ms	18 ms	24 ms	76-220-??-??.lightspeed.sntcca.sbcglobal.net (76.220.??.??)

HE.net pings between Fremont1 and Moscow, Russia

I’m on the lookout to getting some hosting from HE.net in Fremont. So far, on multiple occasions, they seem to have very weird ping times to webdc.ru in Moscow, about 60ms slower than the competition. This is even after the HE upgrade news from 2011-11-18.

# traceroute fremont1.linode.com; date
traceroute to fremont1.linode.com (64.71.152.17), 64 hops max, 40 byte packets
 1  gw.webdc.ru (188.120.247.254)  1.137 ms  5.414 ms  0.997 ms
 2  92.63.108.89 (92.63.108.89)  0.511 ms  1.493 ms  0.705 ms
 3  xe012-438.RT.MR.MSK.RU.retn.net (87.245.254.61)  1.671 ms  1.685 ms  1.427 ms
 4  ae2-6.RT.TC1.STO.SE.retn.net (87.245.233.134)  25.019 ms  24.852 ms  28.651 ms
 5  10gigabitethernet1-2.core1.sto1.he.net (194.68.123.187)  26.656 ms  25.006 ms  25.341 ms
 6  10gigabitethernet3-3.core1.fra1.he.net (72.52.92.233)  53.423 ms  51.915 ms  52.358 ms
 7  10gigabitethernet1-2.core1.par1.he.net (72.52.92.89)  68.890 ms  62.675 ms  58.917 ms
 8  10gigabitethernet7-1.core1.ash1.he.net (184.105.213.93)  142.921 ms  142.972 ms  143.390 ms
 9  10gigabitethernet7-4.core1.pao1.he.net (184.105.213.177)  267.813 ms
    10gigabitethernet7-1.core1.ash1.he.net (184.105.213.93)  143.011 ms  142.574 ms
10  10gigabitethernet7-4.core1.pao1.he.net (184.105.213.177)  260.838 ms  260.449 ms  257.843 ms
11  * * *
12  fremont1.linode.com (64.71.152.17)  257.255 ms  257.337 ms  257.404 ms
0.000u 0.015s 0:20.90 0.0%	20+176k 1+0io 0pf+0w
Sun Nov 20 22:22:46 EST 2011
Per looking glass at http://lg.he.net/, it seems like the reason for the slowness: the traffic from Fremont back to Russia goes through Tokyo:
core1.fmt1.he.net> traceroute 82.146.60.12 source-ip 216.218.252.161 numeric
Target	82.146.60.12
Hop Start	1
Hop End	30
Hop	Packet 1	Packet 2	Packet 3	Hostname
1	6 ms	<1 ms	<1 ms	10.123.123.2
2	126 ms	126 ms	126 ms	10gigabitethernet1-2.core1.tyo1.he.net (72.52.92.238)
3	242 ms	234 ms	242 ms	9002.tyo.equinix.com (203.190.230.36)
4	274 ms	265 ms	263 ms	ae1-9.RT.MR.MSK.RU.retn.net (87.245.233.13)
5	260 ms	283 ms	264 ms	xe000-468.webdc.ru (87.245.254.62)
6	275 ms	274 ms	280 ms	92.63.108.91
7	262 ms	267 ms	277 ms	mojo.su (82.146.60.12)
Vs. residential AT&T U-verse FTTH in San Jose:
% traceroute mojo.su ; date
traceroute to mojo.su (82.146.60.12), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.112 ms  1.610 ms  1.550 ms
 6  ggr6.wswdc.ip.att.net (12.122.86.89)  2.832 ms  2.893 ms  2.787 ms
 7  192.205.32.206 (192.205.32.206)  5.821 ms  4.915 ms  4.831 ms
 8  vlan90.csw4.SanJose1.Level3.net (4.69.152.254)  5.553 ms vlan60.csw1.SanJose1.Level3.net (4.69.152.62)  5.632 ms vlan90.csw4.SanJose1.Level3.net (4.69.152.254)  5.541 ms
 9  ae-81-81.ebr1.SanJose1.Level3.net (4.69.153.9)  5.645 ms ae-71-71.ebr1.SanJose1.Level3.net (4.69.153.5)  6.551 ms ae-81-81.ebr1.SanJose1.Level3.net (4.69.153.9)  6.185 ms
10  ae-2-2.ebr2.NewYork1.Level3.net (4.69.135.186)  74.987 ms  75.426 ms  75.202 ms
11  ae-62-62.csw1.NewYork1.Level3.net (4.69.148.34)  86.278 ms ae-82-82.csw3.NewYork1.Level3.net (4.69.148.42)  76.841 ms ae-62-62.csw1.NewYork1.Level3.net (4.69.148.34)  79.774 ms
12  ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65)  75.636 ms ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69)  74.954 ms ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65)  74.319 ms
13  ae-43-43.ebr2.London1.Level3.net (4.69.137.73)  143.784 ms ae-44-44.ebr2.London1.Level3.net (4.69.137.77)  142.820 ms ae-41-41.ebr2.London1.Level3.net (4.69.137.65)  143.232 ms
14  ae-24-24.ebr2.Frankfurt1.Level3.net (4.69.148.198)  152.786 ms ae-22-22.ebr2.Frankfurt1.Level3.net (4.69.148.190)  152.841 ms  154.634 ms
15  ae-72-72.csw2.Frankfurt1.Level3.net (4.69.140.22)  151.892 ms ae-92-92.csw4.Frankfurt1.Level3.net (4.69.140.30)  152.652 ms ae-62-62.csw1.Frankfurt1.Level3.net (4.69.140.18)  153.116 ms
16  ae-4-90.edge3.Frankfurt1.Level3.net (4.69.154.199)  152.917 ms ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135)  162.515 ms ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7)  152.150 ms
17  dialup-212.162.19.30.frankfurt1.mik.net (212.162.19.30)  152.836 ms  153.561 ms  153.265 ms
18  ae1-9.RT.MR.MSK.RU.retn.net (87.245.233.13)  197.192 ms  197.273 ms ae2-4.RT.MR.MSK.RU.retn.net (87.245.233.25)  198.449 ms
19  xe000-468.webdc.ru (87.245.254.62)  196.588 ms  197.368 ms  196.961 ms
20  92.63.108.91 (92.63.108.91)  196.286 ms  197.651 ms  196.957 ms
21  mojo.su (82.146.60.12)  197.293 ms  200.423 ms  199.022 ms
Sun 20 Nov 2011 19:25:17 PST