After many years in the BSD land, late December 2011 I’ve started using the 2011 release of Debian Linux on Linode512 upgraded to Linode768 (plan to “upgrade” back to 512 after a while on 768).

My list of bugs/annoyances is as follows (I might update this entry later on, too):

* installing ntpd package corrects your date without any trace whatsoever what the correction was; I guesstimate that it’s a package bug, where they run ntpdate or something, but maybe ntpd itself is to blame

* logs are not rotated at midnight. stupid linux runs logrotate as part of the /etc/cron.daily/ at 6:25 on my box (seriously, how entirely dumb is that?) The best fix would probably be `echo “0 0 * * * root /etc/cron.daily/logrotate” »/etc/cron.d/logrotate`, but I’m entirely amazed that anyone finds it acceptable to do daily/weekly/monthly rotation of logs at random times of the day, other than 00:00:00 (renaming a few files doesn’t take that many resources; if you really care about overconsumption at 00:00:00, why not ensure that logs are never compressed during the 00:00:00 run?)

* there is no `jot`. seriously, no jot? `apt-get install athena-jot`

* iptables has no way of storing any rules permanently. Have to install `iptables-persistent` with apt. However, iptables-persistent only works with IPv4, for IPv6 you have to do some hacking. The whole thing where IPv6 is controlled by `ip6tables`, and never by `iptables`, arguably adds to show just how little Linux cares about IPv6 adoption.

# mtr --report{,-wide} 76-220-32-XX.lightspeed.sntcca.sbcglobal.net ; date
HOST: liXXX-XXX                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 184.105.143.85                                0.0%    10    0.6   0.6   0.4   1.2   0.2
  2. 10gigabitethernet2-3.core1.fmt1.he.net        0.0%    10    0.6   5.2   0.5  11.9   4.8
  3. 10gigabitethernet1-1.core1.pao1.he.net        0.0%    10   11.4   4.3   0.8  11.4   3.4
  4. sjo-bb1-link.telia.net                        0.0%    10    1.3   3.1   1.1  19.8   5.9
  5. las-bb1-link.telia.net                        0.0%    10   14.1  15.0  14.1  22.7   2.7
  6. att-ic-149001-las-bb1.c.telia.net            30.0%    10   23.6  32.0  23.6  38.5   5.0
  7. cr2.la2ca.ip.att.net                         50.0%    10   35.0  34.8  30.0  39.7   4.1
  8. cr2.sffca.ip.att.net                         30.0%    10   32.5  36.0  30.2  41.3   4.0
  9. 12.122.149.141                               50.0%    10   27.3  35.9  27.3  40.4   5.3
 10. ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
 11. ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
 12. ???                                          100.0    10    0.0   0.0   0.0   0.0   0.0
 13. 76-220-32-XX.lightspeed.sntcca.sbcglobal.net 80.0%    10   31.1  34.4  31.1  37.8   4.8
Wed Dec 28 21:17:53 EST 2011

  2.|-- 76-220-32-3.lightspeed.sntcca.sbcglobal.net             10.0%    10    2.2   2.6   2.2   3.6   0.4
  3.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- 12.83.39.137                                             0.0%    10    1.8   1.9   1.7   2.3   0.2
  6.|-- 12.122.200.9                                             0.0%    10   45.4  17.5   2.9 105.5  33.7
  7.|-- 208.51.134.1                                             0.0%    10    4.9  10.5   4.9  58.4  16.8
    |  `|-- 192.205.32.46
    |   |-- 192.205.32.50
    |   |-- 208.178.58.185
  8.|-- po1-20G.ar3.SJC2.gblx.net                                0.0%    10    5.3   5.1   5.0   5.3   0.1
  9.|-- Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 10.0%    10   40.7  42.3  36.2  49.4   4.4
 10.|-- 10gigabitethernet1-1.core1.fmt1.he.net                  50.0%    10   40.3  47.4  40.3  51.9   4.3
 11.|-- linode-llc.10gigabitethernet2-3.core1.fmt1.he.net       60.0%    10   40.8  38.9  36.7  40.8   1.7
 12.|-- li163-159.members.linode.com                            20.0%    10   39.4  39.4  38.2  41.2   1.1
Wed 28 Dec 2011 18:18:22 PST

  2.|-- 76-220-32-3.lightspeed.sntcca.sbcglobal.net 10.0%    10    2.2   3.4   2.2  12.3   3.3
  3.|-- ???                                         100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- 71.145.0.80                                 90.0%    10    4.6   4.6   4.6   4.6   0.0
  5.|-- 12.83.39.137                                 0.0%    10    1.7   3.4   1.7  17.6   5.0
  6.|-- 12.122.137.125                               0.0%    10    2.8  19.8   2.8  47.5  16.0
  7.|-- att-gw.sanjoseequinix.savvis.net             0.0%    10    4.6  10.5   4.5  61.3  17.9
  8.|-- er1-te-3-1.SanJoseEquinix.savvis.net         0.0%    10    4.6  24.4   4.5 169.3  52.0
  9.|-- cr1-tenge-0-3-5-0.sanfrancisco.savvis.net    0.0%    10    7.0   6.6   6.3   7.0   0.2
 10.|-- cr1-tengig-0-0-2-0.losangeles.savvis.net     0.0%    10   16.4  16.4  15.7  17.1   0.4
 11.|-- ber1-tenge-2-1.losangeles.savvis.net         0.0%    10   15.7  16.6  15.6  23.4   2.4
 12.|-- 209.144.203.214                              0.0%    10   16.3  16.2  15.1  17.1   0.6
 13.|-- arpnetworks-lax2-gw.cust.trit.net            0.0%    10   33.8  18.9  16.7  33.8   5.3
 14.|-- arpnetworks.com                              0.0%    10   16.8  16.8  16.4  17.1   0.2
Wed 28 Dec 2011 18:20:24 PST

  2.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                                                     100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- 12.83.39.137                                             0.0%    10    1.7   1.9   1.6   2.3   0.2
  6.|-- 12.122.200.9                                             0.0%    10    2.9   8.1   2.9  41.8  12.5
  7.|-- 208.51.134.1                                             0.0%    10    5.0  62.9   5.0 199.6  69.7
    |  `|-- 192.205.32.46
    |   |-- 192.205.32.50
    |   |-- 208.178.58.185
  8.|-- po1-20G.ar3.SJC2.gblx.net                                0.0%    10    5.5  57.7   4.9 228.9  87.6
  9.|-- Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 30.0%    10   38.0  39.1  33.9  47.7   4.5
 10.|-- 10gigabitethernet1-1.core1.fmt1.he.net                  30.0%    10   43.5  45.6  33.1  92.9  21.2
 11.|-- linode-llc.10gigabitethernet2-3.core1.fmt1.he.net       60.0%    10   34.2  37.7  34.2  40.3   2.6
 12.|-- li163-159.members.linode.com                            20.0%    10   38.1  36.7  32.7  40.1   2.6
Wed 28 Dec 2011 18:21:00 PST

Somehow no noticeable ssh packet loss whatsoever, though.

Meh. For a few hours now, seems like HE has a latency problem; about 17ms to Fremont from San Jose, instead of the usual ~5.5ms. Even some packet loss, meh.

% traceroute fremont1.linode.com ; date
traceroute to fremont1.linode.com (64.71.152.17), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.387 ms  1.964 ms  1.784 ms
 6  12.122.200.9 (12.122.200.9)  2.974 ms  3.031 ms  3.020 ms
 7  dcr2-so-3-0-0.washington.savvis.net (192.205.32.46)  5.060 ms 208.178.58.185 (208.178.58.185)  5.006 ms 192.205.32.50 (192.205.32.50)  4.965 ms
 8  po1-20G.ar3.SJC2.gblx.net (67.16.134.26)  5.210 ms  5.201 ms  5.012 ms
 9  Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net (64.214.174.246)  26.734 ms  25.329 ms  25.379 ms
10  10gigabitethernet1-1.core1.fmt1.he.net (72.52.92.109)  18.158 ms  17.942 ms  21.013 ms
11  linode-llc.10gigabitethernet2-3.core1.fmt1.he.net (64.62.250.6)  17.133 ms  17.523 ms  17.248 ms
12  fremont1.linode.com (64.71.152.17)  18.028 ms  17.511 ms  18.120 ms
Fri 23 Dec 2011 19:29:24 PST

% ping -c8 fremont1.linode.com; date
PING fremont1.linode.com (64.71.152.17): 56 data bytes
64 bytes from 64.71.152.17: icmp_seq=0 ttl=51 time=17.439 ms
64 bytes from 64.71.152.17: icmp_seq=1 ttl=51 time=17.319 ms
64 bytes from 64.71.152.17: icmp_seq=2 ttl=51 time=17.650 ms
64 bytes from 64.71.152.17: icmp_seq=3 ttl=51 time=19.009 ms
64 bytes from 64.71.152.17: icmp_seq=4 ttl=51 time=18.353 ms
64 bytes from 64.71.152.17: icmp_seq=5 ttl=51 time=19.261 ms
64 bytes from 64.71.152.17: icmp_seq=7 ttl=51 time=17.059 ms

--- fremont1.linode.com ping statistics ---
8 packets transmitted, 7 packets received, 12% packet loss
round-trip min/avg/max/stddev = 17.059/18.013/19.261/0.803 ms
Fri 23 Dec 2011 19:34:53 PST

To be fair, though, seems like HE is not the only one that has poor latency this time of the year — NAC in Newark, with nlayer.net, is also lagging noticeably:

% date ; echo {fremont,dallas,atlanta,newark,tokyo,london}1.linode.com | xargs -tn1 /usr/sbin/traceroute -M5 -w2 -m32 ; date
Fri 23 Dec 2011 19:44:03 PST
/usr/sbin/traceroute -M5 -w2 -m32 fremont1.linode.com
traceroute to fremont1.linode.com (64.71.152.17), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.455 ms  1.768 ms  1.801 ms
 6  12.122.200.9 (12.122.200.9)  159.361 ms  200.101 ms  14.856 ms
 7  192.205.32.50 (192.205.32.50)  47.598 ms 208.51.134.1 (208.51.134.1)  211.444 ms dcr2-so-3-0-0.washington.savvis.net (192.205.32.46)  5.488 ms
 8  po1-20G.ar3.SJC2.gblx.net (67.16.134.26)  4.962 ms  5.050 ms  5.161 ms
 9  Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net (64.214.174.246)  17.085 ms  19.466 ms  24.127 ms
10  10gigabitethernet1-1.core1.fmt1.he.net (72.52.92.109)  17.120 ms  18.184 ms  25.457 ms
11  linode-llc.10gigabitethernet2-3.core1.fmt1.he.net (64.62.250.6)  17.801 ms  17.672 ms  17.424 ms
12  fremont1.linode.com (64.71.152.17)  17.154 ms  17.945 ms  19.139 ms
/usr/sbin/traceroute -M5 -w2 -m32 dallas1.linode.com
traceroute to dallas1.linode.com (69.164.200.100), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.135 ms  1.828 ms  1.619 ms
 6  151.164.101.206 (151.164.101.206)  3.825 ms  3.844 ms  3.837 ms
 7  xe-0-2-0-5.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.121)  5.160 ms ppp-151-164-46-66.eulstx.swbell.net (151.164.46.66)  5.176 ms xe-0-2-0-7.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.141)  5.465 ms
 8  ae1.bbr02.eq01.sjc01.networklayer.com (128.241.219.234)  7.555 ms  5.222 ms  5.302 ms
 9  ae0.bbr02.cs01.lax01.networklayer.com (173.192.18.151)  15.055 ms  14.084 ms  13.686 ms
10  ae7.bbr01.cs01.lax01.networklayer.com (173.192.18.166)  13.854 ms  13.846 ms  13.502 ms
11  ae19.bbr01.eq01.dal03.networklayer.com (173.192.18.140)  46.931 ms  84.632 ms  46.854 ms
12  po31.dsr02.dllstx3.networklayer.com (173.192.18.227)  49.257 ms po31.dsr01.dllstx3.networklayer.com (173.192.18.225)  44.207 ms po31.dsr02.dllstx3.networklayer.com (173.192.18.227)  46.194 ms
13  * * *
14  po1.car01.dllstx2.networklayer.com (70.87.254.74)  46.798 ms po2.car01.dllstx2.networklayer.com (70.87.254.78)  46.925 ms po1.car01.dllstx2.networklayer.com (70.87.254.74)  46.261 ms
15  5a.7.1243.static.theplanet.com (67.18.7.90)  46.774 ms  43.758 ms  49.258 ms
16  dallas1.linode.com (69.164.200.100)  44.063 ms  44.278 ms  47.010 ms
/usr/sbin/traceroute -M5 -w2 -m32 atlanta1.linode.com
traceroute to atlanta1.linode.com (63.247.71.196), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.364 ms  1.770 ms  1.739 ms
 6  ppp-151-164-52-233.rcsntx.swbell.net (151.164.52.233)  3.896 ms  3.804 ms  3.910 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.46.70)  6.494 ms  4.967 ms  5.768 ms
 8  ae0-80g.cr1.pao1.us.nlayer.net (69.22.153.18)  3.798 ms  3.730 ms  3.840 ms
 9  ae1-50g.cr1.sjc1.us.nlayer.net (69.22.143.165)  5.440 ms  5.269 ms  4.353 ms
10  xe-3-3-1.cr1.lax1.us.nlayer.net (69.22.142.8)  13.218 ms xe-5-1-0.cr1.lax1.us.nlayer.net (69.22.142.126)  13.739 ms xe-3-3-1.cr1.lax1.us.nlayer.net (69.22.142.8)  13.217 ms
11  xe-1-0-0.cr1.iah1.us.nlayer.net (69.22.142.121)  48.768 ms  48.500 ms  48.476 ms
12  xe-4-2-1.cr1.atl1.us.nlayer.net (69.22.142.118)  71.607 ms  71.570 ms  71.975 ms
13  ae1-40g.ar1.atl1.us.nlayer.net (69.31.135.130)  65.750 ms  67.621 ms  67.644 ms
14  as3595.xe-2-0-5-101.ar1.atl1.us.nlayer.net (69.31.135.42)  133.773 ms  98.646 ms  80.123 ms
15  64.22.106.10 (64.22.106.10)  63.216 ms  63.468 ms  66.714 ms
16  atlanta1.linode.com (63.247.71.196)  63.546 ms  63.098 ms  63.590 ms
/usr/sbin/traceroute -M5 -w2 -m32 newark1.linode.com
traceroute to newark1.linode.com (207.192.68.6), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.245 ms  1.696 ms  1.635 ms
 6  ex2-p10-0.pxpaca.sbcglobal.net (151.164.190.83)  3.612 ms  3.633 ms  3.529 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.251.34)  4.849 ms  7.351 ms  4.755 ms
 8  ae0-80g.cr1.pao1.us.nlayer.net (69.22.153.18)  3.737 ms  10.857 ms  3.664 ms
 9  ae1-60g.cr1.sfo1.us.nlayer.net (69.22.143.169)  4.329 ms  4.346 ms  4.346 ms
10  xe-0-0-3.cr1.slc1.us.nlayer.net (69.22.142.96)  19.397 ms  19.331 ms  19.394 ms
11  xe-5-2-0.cr1.ord1.us.nlayer.net (69.22.142.101)  53.983 ms  53.913 ms  53.920 ms
12  xe-9-2-0.cr1.ewr1.us.nlayer.net (69.22.142.75)  118.761 ms  118.864 ms  120.644 ms
13  ae1-70g.ar2.ewr1.us.nlayer.net (69.31.94.118)  122.490 ms  105.794 ms  120.565 ms
14  as8001.xe-3-0-6.ar2.ewr1.us.nlayer.net (69.31.95.130)  104.817 ms  104.694 ms  104.436 ms
15  0.e1-2.tbr1.mmu.nac.net (209.123.10.118)  118.743 ms  119.052 ms  120.066 ms
16  vlan803.esd2.mmu.nac.net (209.123.10.30)  73.661 ms  73.959 ms  75.223 ms
17  207.99.53.46 (207.99.53.46)  106.657 ms  109.364 ms  104.719 ms
18  newark1.linode.com (207.192.68.6)  116.415 ms  116.762 ms  118.573 ms
/usr/sbin/traceroute -M5 -w2 -m32 tokyo1.linode.com
traceroute to tokyo1.linode.com (106.187.33.12), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.321 ms  1.849 ms  1.613 ms
 6  ggr7.sffca.ip.att.net (12.122.114.17)  2.985 ms  2.807 ms  2.836 ms
 7  192.205.36.2 (192.205.36.2)  4.349 ms  4.458 ms  4.498 ms
 8  pax-brdr-01.inet.qwest.net (205.171.234.2)  5.937 ms  5.873 ms  5.927 ms
 9  124.215.192.77 (124.215.192.77)  23.994 ms  17.741 ms  17.895 ms
10  pajbb002.kddnet.ad.jp (111.87.3.29)  120.640 ms pajbb002.kddnet.ad.jp (124.211.34.133)  16.082 ms pajbb001.kddnet.ad.jp (111.87.3.5)  84.107 ms
11  otejbb203.kddnet.ad.jp (203.181.100.5)  131.326 ms otejbb203.kddnet.ad.jp (203.181.100.209)  119.472 ms otejbb204.kddnet.ad.jp (203.181.100.201)  119.155 ms
12  cm-fcu203.kddnet.ad.jp (124.215.194.164)  121.772 ms  125.743 ms cm-fcu203.kddnet.ad.jp (124.215.194.180)  133.351 ms
13  124.215.199.122 (124.215.199.122)  113.014 ms  115.081 ms  123.389 ms
14  tokyo1.linode.com (106.187.33.12)  120.650 ms  112.850 ms  114.916 ms
/usr/sbin/traceroute -M5 -w2 -m32 london1.linode.com
traceroute to london1.linode.com (109.74.207.9), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.025 ms  13.973 ms  1.969 ms
 6  12.122.200.9 (12.122.200.9)  2.886 ms  2.792 ms  2.951 ms
 7  192.205.36.234 (192.205.36.234)  5.252 ms  4.845 ms  5.133 ms
 8  telecity.ge9-9.br02.ldn01.pccwbtn.net (63.218.13.222)  146.812 ms  146.863 ms  146.933 ms
 9  te4-1-dist65-01.lon10.telecity.net (217.20.44.218)  148.074 ms  148.169 ms  148.923 ms
10  212.111.33.234 (212.111.33.234)  148.846 ms  149.418 ms  148.772 ms
11  london1.linode.com (109.74.207.9)  149.089 ms  148.989 ms  149.142 ms
Fri 23 Dec 2011 19:44:19 PST

Just for the full picture:

% traceroute sonic.net ; date
traceroute to sonic.net (209.204.190.64), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.126 ms  1.751 ms  1.617 ms
 6  ppp-151-164-52-133.rcsntx.swbell.net (151.164.52.133)  3.852 ms  3.678 ms  3.867 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.251.34)  10.775 ms  4.978 ms  7.045 ms
 8  as7065.xe-1-0-6.ar1.pao1.us.nlayer.net (69.22.130.86)  3.723 ms  3.815 ms  3.542 ms
 9  0.ge-0-1-0.gw.sr.sonic.net (64.142.0.197)  6.273 ms  6.259 ms  6.262 ms
10  gig0-1.dist2-1.sr.sonic.net (209.204.191.160)  6.222 ms gig0-2.dist2-1.sr.sonic.net (208.201.224.160)  6.430 ms  6.510 ms
11  fast1.lb2-1.slb.sr.sonic.net (64.142.0.58)  6.376 ms  6.432 ms  6.453 ms
12  fast1.lb2-1.slb.sr.sonic.net (64.142.0.58)  6.567 ms  6.467 ms  6.449 ms
Fri 23 Dec 2011 19:47:05 PST

% traceroute netflix.com ; date
traceroute to netflix.com (69.53.236.17), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.139 ms  1.724 ms  1.667 ms
 6  ggr7.sffca.ip.att.net (12.122.114.13)  3.611 ms  36.855 ms  47.582 ms
 7  xe-1-2-0.mpr4.sjc7.us.above.net (64.125.12.117)  4.671 ms  4.588 ms  4.594 ms
 8  xe-1-0-0.mpr3.sjc7.us.above.net (64.125.26.125)  4.890 ms  4.844 ms  4.910 ms
 9  xe-4-0-0.er1.sjc2.us.above.net (64.125.30.173)  4.864 ms  4.975 ms  4.877 ms
10  64.124.65.90.allocated.above.net (64.124.65.90)  5.121 ms  5.052 ms  4.925 ms
11  xe-2-3-0-945.jnrt-edge02.sv1.netflix.com (208.75.77.197)  6.108 ms  6.032 ms  5.972 ms
12  xe-2-2-0-955.jnrt-edge02.prod1.netflix.com (69.53.225.30)  7.189 ms  17.194 ms  7.141 ms
13  te1-8.csrt-agg02.prod1.netflix.com (69.53.225.10)  7.532 ms  7.335 ms  7.367 ms
14  netflixfreetrial.com (69.53.236.17)  7.237 ms  7.172 ms  7.199 ms
Fri 23 Dec 2011 19:47:31 PST

% traceroute facebook.com ; date
traceroute: Warning: facebook.com has multiple addresses; using 66.220.149.11
traceroute to facebook.com (66.220.149.11), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.108 ms  2.252 ms  1.844 ms
 6  151.164.101.206 (151.164.101.206)  3.981 ms  4.046 ms  3.954 ms
 7  xe-0-2-0-6.r07.snjsca04.us.bb.gin.ntt.net (129.250.8.93)  6.051 ms xe-0-2-0-7.r07.snjsca04.us.bb.gin.ntt.net (129.250.9.141)  5.274 ms ppp-151-164-46-66.eulstx.swbell.net (151.164.46.66)  5.495 ms
 8  ae-8.r21.snjsca04.us.bb.gin.ntt.net (129.250.5.56)  5.471 ms ae-7.r20.snjsca04.us.bb.gin.ntt.net (129.250.5.52)  5.731 ms ae-8.r21.snjsca04.us.bb.gin.ntt.net (129.250.5.56)  5.523 ms
 9  ae-4.r05.plalca01.us.bb.gin.ntt.net (129.250.5.32)  6.387 ms ae-4.r06.plalca01.us.bb.gin.ntt.net (129.250.4.118)  6.551 ms ae-4.r05.plalca01.us.bb.gin.ntt.net (129.250.5.32)  6.656 ms
10  ae-2.r05.plalca01.us.bb.gin.ntt.net (129.250.5.237)  6.367 ms 140.174.21.142 (140.174.21.142)  6.151 ms ae-2.r05.plalca01.us.bb.gin.ntt.net (129.250.5.237)  6.629 ms
11  ae1.bb01.pao1.tfbnw.net (74.119.76.134)  7.840 ms 140.174.21.142 (140.174.21.142)  6.885 ms ae1.bb01.pao1.tfbnw.net (74.119.76.134)  6.481 ms
12  ae1.bb01.pao1.tfbnw.net (74.119.76.134)  6.651 ms ae2.dr02.snc5.tfbnw.net (74.119.77.192)  6.813 ms ae1.bb01.pao1.tfbnw.net (74.119.76.134)  6.498 ms
13  po509.csw02b.snc5.tfbnw.net (74.119.78.28)  7.216 ms ae0.dr01.snc5.tfbnw.net (74.119.77.180)  7.469 ms po509.csw02a.snc5.tfbnw.net (74.119.78.24)  7.063 ms
14  po510.csw02b.snc5.tfbnw.net (74.119.78.26)  7.358 ms * po509.csw02a.snc5.tfbnw.net (74.119.78.24)  7.916 ms
15  * * *
16  *^C
% traceroute google.com ; date
traceroute: Warning: google.com has multiple addresses; using 74.125.224.147
traceroute to google.com (74.125.224.147), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.503 ms  1.851 ms  1.622 ms
 6  ppp-151-164-52-163.rcsntx.swbell.net (151.164.52.163)  4.657 ms  4.581 ms  4.528 ms
 7  gar7.sffca.ip.att.net (12.122.79.97)  4.569 ms  4.463 ms  4.628 ms
 8  cr83.sffca.ip.att.net (12.122.110.118)  7.015 ms  6.648 ms  6.170 ms
 9  cr1.sffca.ip.att.net (12.123.15.109)  7.134 ms  8.057 ms  7.767 ms
10  cr81.sj2ca.ip.att.net (12.122.1.118)  7.327 ms  7.347 ms  6.838 ms
11  12.122.114.21 (12.122.114.21)  5.999 ms  5.853 ms  5.662 ms
12  12.249.231.14 (12.249.231.14)  6.488 ms  6.477 ms  6.438 ms
13  209.85.249.3 (209.85.249.3)  7.001 ms  7.207 ms  6.940 ms
14  64.233.174.119 (64.233.174.119)  7.518 ms  7.382 ms  7.297 ms
15  nuq04s09-in-f19.1e100.net (74.125.224.147)  7.108 ms  6.924 ms  8.550 ms
Fri 23 Dec 2011 19:49:05 PST

% traceroute paxio.net ; date
traceroute to paxio.net (63.216.71.33), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.140 ms  1.777 ms  1.658 ms
 6  ggr6.wswdc.ip.att.net (12.122.86.89)  3.224 ms  15.199 ms  3.122 ms
 7  192.205.32.206 (192.205.32.206)  5.203 ms  5.082 ms  5.015 ms
 8  vlan80.csw3.SanJose1.Level3.net (4.69.152.190)  5.785 ms vlan90.csw4.SanJose1.Level3.net (4.69.152.254)  5.753 ms  5.649 ms
 9  ae-72-72.ebr2.SanJose1.Level3.net (4.69.153.21)  7.140 ms  7.352 ms ae-62-62.ebr2.SanJose1.Level3.net (4.69.153.17)  5.706 ms
10  ae-5-5.car1.Oakland1.Level3.net (4.69.134.37)  6.833 ms  6.644 ms  6.809 ms
11  ae-11-11.car2.Oakland1.Level3.net (4.69.134.42)  6.831 ms  6.705 ms  6.762 ms
12  PAXIO-INC.car2.Oakland1.Level3.net (4.71.202.38)  7.325 ms  7.359 ms  7.248 ms
13  173.241.16.50 (173.241.16.50)  7.468 ms  7.865 ms  7.492 ms
14  63-216-64-246.paxio.net (63.216.64.246)  7.281 ms  7.022 ms  7.169 ms
15  64-201-240-174.static.paxio.net (64.201.240.174)  7.190 ms  6.926 ms  7.955 ms
16  64-201-240-182.static.paxio.net (64.201.240.182)  8.424 ms  9.611 ms  8.657 ms
17  64-201-240-170.static.paxio.net (64.201.240.170)  8.724 ms  8.196 ms  8.426 ms
18  * * *
19  *^C

Somehow, Google and Facebook don’t seem to have any latency issues whatsoever, yet HE.net, whose sole purpose is IP transit, can’t get their latency within the Bay Area alone! WTF? NAC, a famous data centre in Newark, with its nlayer.net transit, also disappoints.

P.S. Hm, svcolo.com seems good, through nlayer.net, the latency may perhaps go as low as 3.5ms during non-peak hours, now about 4.0ms:

% traceroute svcolo.com ; date
traceroute to svcolo.com (64.13.135.35), 32 hops max, 40 byte packets
 5  12.83.39.137 (12.83.39.137)  2.181 ms  1.797 ms  1.746 ms
 6  151.164.54.183 (151.164.54.183)  3.800 ms  3.446 ms  3.567 ms
 7  asn4436-nlayer.pxpaca.sbcglobal.net (151.164.46.70)  8.373 ms  7.769 ms  5.264 ms
 8  ae0-80g.cr1.pao1.us.nlayer.net (69.22.153.18)  64.425 ms  3.518 ms  3.634 ms
 9  as8121.ae0-3001.cr1.pao1.us.nlayer.net (69.22.153.114)  3.907 ms  3.849 ms  3.850 ms
10  xe1-4.core1.svk.layer42.net (69.36.239.121)  4.316 ms  4.163 ms  4.125 ms
11  www.svcolo.com (64.13.135.35)  3.969 ms  4.054 ms  4.050 ms
Fri 23 Dec 2011 21:16:51 PST

P.P.S. Apparently, according to http://lg.he.net/, the issue is that the traffic back to my place, goes through telia in Los Angeles. WTF?

core1.fmt1.he.net> traceroute 76.220.??.?? source-ip 216.218.252.161 numeric
Target	76.220.32.89
Hop Start	1
Hop End	30
Hop	Packet 1	Packet 2	Packet 3	Hostname
1	14 ms	1 ms	<1 ms	10gigabitethernet1-1.core1.pao1.he.net (184.105.213.66)
2	1 ms	1 ms	1 ms	sjo-bb1-link.telia.net (213.248.86.53)
3	14 ms	24 ms	49 ms	las-bb1-link.telia.net (213.248.80.17)
4	25 ms	24 ms	24 ms	att-ic-149001-las-bb1.c.telia.net (213.248.88.30)
5	25 ms	25 ms	24 ms	cr2.la2ca.ip.att.net (12.123.30.150)
6	26 ms	24 ms	24 ms	cr2.sffca.ip.att.net (12.122.31.134)
7	24 ms	25 ms	26 ms	12.122.149.141
8	*	*	*	?
9	*	*	*	?
10	*	*	*	?
11	74 ms	18 ms	24 ms	76-220-??-??.lightspeed.sntcca.sbcglobal.net (76.220.??.??)