Fri 7 Apr 2006
Poul-Henning Kamp, Slagelse, Denmark, writes:
When I contacted D-Link back in November 2005 about the way D-Link products abused my NTP-server, I expected to get in touch with somebody who understood what they were talking about, I expected them to admit that D-Link had made a bad decision and I expected that D-Link would make good on the damage they were responsible for.
For the last five months I have wasted a lot of time trying to reach some kind of agreement with the Californian lawyer which D-Link put on the case. I can’t quite make up my mind if D-Link’s lawyer negotiates in bad faith or is merely uninformed, I tend to suspect the latter, but either way, as of this morning I decided to cut my losses.
Since no one else at D-Link has reacted to my numerous emails, I have no other means of getting in touch with D-Link other than an open letter. I realize that it will be inconvenient and embarrasing for D-Link to have this matter exposed in public this way, but I seem to have no other choice.
I will now lay out the case below in such detail that any moderately knowledgeable person should be able to understand it, and hopefully somebody, somewhere in D-Link will contact me so we can get this matter resolved.
What is NTP?
NTP is Network Time Protocol, a protocol that allows computers to transfer timestamps across the internet so that they can set their clocks to the correct time.
A number of NTP servers on the internet are connected to radio timecode receivers, GPS receivers or in some cases directly to national time laboratories primary atomic frequency standards.
How not to implement NTP in a product
A number of D-Link products, so far I have at least identified DI-604, DI-614+, DI-624, DI-754, DI-764, DI-774, DI-784, VDI604 and VDI624, contain a list of NTP servers in their firmware and using some sort of algorithm, they pick one and send packets to it.
This is about as wrong a way to do things as one can imagine. There is no way D-Link can change the list once the product is shipped, unless D-Link can persuade the customer to upgrade the firmware.
How to implement NTP in a product
The correct way, as I have pointed out to D-Link repeatedly, is to query a D-Link controlled DNS entry like “ntp.dlink.com” and populate this DNS entry with the list of NTP servers to be queried. That would allow D-Link to add or remove servers from the list by changing the DNS server files and all deployed devices would automatically see the update next time.
If D-Link had implemented the NTP feature this way, my complaint could have been handled to my full satisfaction with an emailed apology and a few minutes of D-Link’s DNS administrators time.
The problem
As you can see in the table on the right side, D-Link included the NTP server “GPS.dix.dk” in the list of NTP servers to query, and they did so without asking for permission.
I have no idea how many devices D-Link has sold, but between 75% and 90% of the packets which arrive at my server come from D-Link products via this mechanism.
Why D-Link needs to ask for permission
The public service of the GPS.dix.dk NTP server has been advertised in the NTP projects list of Stratum 1 NTP servers with the following text:
DK Denmark GPS.dix.dk (192.38.7.240)
Location: Lyngby, Denmark
Geographic Coordinates: 55:47:03.36N, 12:03:21.48E
Synchronization: NTP V4 GPS with OCXO timebase
Service Area: Networks BGP-announced on the DIX
Access Policy: open access to servers, please, no client use
Contacts: Poul-Henning Kamp ()
Note: timestamps better than +/-5 usec.
You will notice two restrictions here, one is the “Service Area” and the other is the “Access Policy”. D-Link makes no effort to comply with either of these two restrictions.
I should also note that this very same list is the only place where the services of GPS.dix.dk is advertised and where DLink got the name from in the first place.
Since D-Link does not comply with these restrictions, D-Link has no legitimate access to the server, and it follows trivially that D-Link should have asked for my permission before including it in the list embedded in their products firmware.
Why GPS.dix.dk has these restrictions
The GPS.dix.dk server is hosted on the DIX, the neutral Danish Internet eXchange where ISPs exchange their traffic.
Access to the DIX is only for BGP routers, you can not get your web-server hosted there.
A special permission has been given for GPS.dix.dk because it offers a vital public service to the Danish part of the Internet, and because I offer this service free of charge and NTP is a low bandwidth protocol, the organization behind the DIX has graciously waived the normal DKR 27.000,00 (approx USD 4,400) connection fee.
You might wonder why it is not a national time-laboratory which offers time service in Denmark, and the short answer is that we have no such thing. In the absense of this vital piece of public infrastructure, I have pro bono publico run this service because I am a time-geek.
Obviously, this special arrangement is contingent on several restrictions, the main one being the “Service Area” restriction set out above: The service is intended for Danish networks only.
Why I can’t mitigate D-Link’s mistake
D-Link embedded the DNS name of GPS.dix.dk directly in the firmware of their products, changing the IP number of GPS.dix.dk would not help, the D-Link products would find the new IP number and the traffic would resume.
Unfortunately, there is no way I can recognize these particular DNS queries and therefore it is not possible to deflect or avoid the subsequent NTP queries to the server from arriving from all over the world.
There are approximately 2000 legitimate users of the GPS.dix.dk server, and most of these have correctly configured their NTP software using the DNS name, so changing the name would be a very timeconsuming effort for both me and for the hundreds of system administrators this would affect.
Filtering the D-Link packets requires inspection of fields which are not simple to implement in Cisco routers, and in particular such filtering seems to send all packets on the interface through the CPU instead of fast switching, so ingress filtering the packets at the ingress of AS1835 is totally out of the question.
So the short and the long of it is that there is nothing I can do to avoid the packets arriving at my server, until all D-Link customers have updated to a fixed firmware or thrown out the D-Link device in 3-5 years time.
The impact of of D-Link’s abuse
Negotiations with the DIX management are ongoing, but the current theory is that I will have to close the GPS.DIX.dk server or pay a connection-fee of DKR 54.000,00 (approx USD 8,800) a year as long as the traffic is a significant fraction of total traffic to the server.
I owe $5000 to an external consultant who helped me track down where these packets came from.
I have already spent close to 120 non-billable hours (I’m an independent contractor) negotiating with D-Link’s laywers and mitigating the effect of the packets on the services provided to the legitimate users of GPS.dix.dk.
Finally I have spent approx DKR 15.000,00 (USD 2,500) on lawyers fees trying to get D-Link to negotiate in good faith.
If I closed the GPS.dix.dk server right now, wrote off all the time I have spent myself, then my expenses would amount to between DKR 45.000,00 and DKR 99.000,00 (USD 7,300 to 16,000) and several hundered administrators throughout Denmark would have to spend time reconfiguring their servers.
If on the other hand we assume I leave the service running and that the unauthorized packets from D-Link products continue for the next five years, the total cost for me will be around DKR 115.000,00 + 54.000,00 per year (approx USD 18,500 + USD 8,800 per year) or DKR 385.000,00 over the next five years (USD 62,000).
All of this is entirely due to D-Link’s incompetent product design and I have no way to mitigate it.
What I asked of D-Link
I have asked D-Link to remove all affected firmware versions from their download servers and to stop shipping products which query GPS.dix.dk.
I have asked D-Link to issue a prominent product notice to induce the affected customers to upgrade the firmware of their products as soon as possible.
I have asked D-Link to cover my cost to the amount of DKR 220.000,00 under the assumption that the above remedies would limit the period where bandwidth charges would have to be paid to two years.
What D-Link has done
Following my contact to D-Link in November, D-Link have released new firmware for some of the affected products where the list of NTP servers have been revised. One such revision can be seen in the table on the right.
Last time I looked, March 16, there were still at least 25 products for which firmware files had “GPS.dix.dk” in them, so obviously D-Link either has a very poor software development or hasn’t allocated enough resources to the task.
I can not publically disclose the specific offers D-Link’s lawyer has made, but these documents are obviously available to D-Link management through internal channels.
I can however summarize them: I have been accused of extortion. I have been told that I have no claim, been told that I exaggerate the claim. I have been told to submit myself to California law but would have to sign away all my rights under it.
I have also been offered a specfic amount of “hush-money” if I would just shut up and go away, but the amount offered would not even cover my most direct expenses.
In return D-Link would admit to nothing, promise nothing and do nothing to induce their customers to upgrade their firmware.
And nowhere in five months of correspondence have I seen the word “sorry” or “apology” forwarded to me.
The Future
I don’t know.
I’m publishing this open letter in a last ditch attempt to get a responsible person at D-link to shoulder their responsibility.
From the webserver logs I can see that approximately 50 different IP numbers from DLink’s Irwine facility (192.152.81.0/24) has read it. Hopefully one or more of them have sufficient integrity to escalate the case.
I can’t afford to sue D-Link. It seems that they have managed to arrange their corporate affairs so that there is no way I can sue them here in Denmark, but will have to do it either in Taiwan or USA. Needless to say, I can’t afford that.
So unless D-Link comes to their senses, I guess the outcome is that I lose a lot of money and the gps.dix.dk server will cease to offer a public service to the Danish part of the internet.
That is why the title of this open letter is titled “NTP vandalism”.
So if you, dear reader, know somebody who works at D-Link, please point them at this open letter. If you don’t, feel free to spread news of it through other channels, my only hope is that eventually somebody in D-Link management becomes embarrased enough to do the honourable thing.
Didn’t something like this happen before?
Yes, D-Link is not the first vendor to make a hash of the NTP protocol. Some years back NetGear products blasted University of Wisconsin off the net. I have repeatedly pointed D-Link’s lawyer at this case. Fortunately, in my case it is not that bad.
The NetGear incident caused the NTP protocol designers to add a “kiss of death” option to the Latest (S)NTP standard but D-Links devices does not respect that option. I have tried.
email:
phone: