Is Dedicated Hosting for Critical DTDs Necessary?

pcause asks: "Recently there was a glitch, when someone at Netscape took down a page that had an important DTD (for RSS), used by many applications and services. This got me thinking that many or all of the important DTDs that software and commerce depend on are hosted at various commercial entities. Is this a sane way to build an XML based Internet infrastructure? Companies come and go all of the time; this means that the storage and availability of those DTDs is in constant jeopardy. It strikes me that we need an infrastructure akin to the root server structure to hold the key DTDs that are used throughout the industry. What organization would be the likely custodian of such data, and what would be the best way to insure such an infrastructure stays funded?"

I know!

ICANN!

Mhahahahaha. Yeah. I know, I crack myself up.

 

Re:Catalog files?

Or better yet why can't you just copy the blasted thing to your own site if your going to use it?

Is there some technical reason I'm not aware of that means it has to stay somewhere central?

There shouldn't be, yet I would be greatly surprised if some application didn't match on the entire DTD string, hostname and all.

I am equally baffled at what applications need the DTD for anyway. Except for generic XML applications, what use is a DTD? Most applications only handles a fixed few XML document types anyway.

Finally, if they really need that DTD... any distro have most major DTDs available. No reason why they couldn't carry a few extra. Should be easy to just search for them locally.

Re:I know!

ICANN!

Mhahahahaha. Yeah. I know, I crack myself up.

No you cann't!

Centralization

Nothing too insightful to write, but worth saying in today's volatile political climate. Centralization makes me nervous.

Regards.

Re:Centralization

There needs to be a way to refer to decentralized internet resources in a unique fashion. We need the equivalent of the URL for a file that is hosted simultaneously in many places.

This is known as a URN [wikipedia.org]. URLs and URNs are together known as URIs.

Re:Centralization

The defects of the URN/URI/URL mechanism were well known at the time this was discussed in the working groups and SIGs while XML was gestating.

The correct solution would have been to fix the outstanding problems with FPIs and use a combination of local catalog and DNS-style resolution, but this was turned down. Perhaps it's time to wake it up.

In the 1990s I did try to devise a resolution server for FPIs, in the hope that someone like the (then) GCA (now IdeAlliance) -- who were the ISO 9070 Registration Authority and theoretically still are -- would pick up the idea.

I still have the large collection of SGML DTDs used at the time, now largely redundant, but replacing it with current XML is not the problem. This is something that should probably be discussed at the Markup conference in Montreal this summer.

Don't use them

If the absence of these files will break your app or service, then you need to make your app or service more robust.

Sure, DTD files are necessary for development. If your app requires that they be used to validate something in real time each time it is comes in from a client or whatever, then use an internal copy of the version of the DTD file that you support. If the host makes a change to it (or drops it, or lets it get hacked), your app won't break, and you can decide when you will implement and support that change.

I really don't see what is gained by making the real time operation of your application dependent on the availability and pristinity of remotely and independently hosted files. It just makes you fragile, and you can get all the benefits you need from just checking the files during your maintenance and development cycles.

Re:Don't use them

Exactly. The only point of having a URL associated with a DTD is to assure a unique identifier for each one. It wasn't worth starting a group specifically to regulate DTD identifiers, so they hooked it to a system that's already regulated. Yeah, it's nice to have the DTD live at that location, so if you get a file with a reference to an unfamiliar DTD you can pull it down on the spot, but it shouldn't be required.

w3c

w3c.org [w3c.org] . There's no better place to keep the standards related to the web.

Re:w3c

There's no better place to keep the standards related to the web.

Some say that wistfully, others begrudgingly.

DTD?

and DTD stands for? Distributed Technical Dependency?

Re:DTD?

Document Type Definition

Re:DTD?

It's the sound Carlos Mencia makes...

In case of death...

...keep a copy, host it on your own site and reference that instead. There was no problem except that some were using that file to download the definitions. Or just expand the definition to include a checksum and a list of mirrors. Is this even a problem worth solving? I mean except for the slashdot post it seemed to me like this went by without anyone noticing.

Not only DTDs, but also ontology definitions

Such a system should also allow stable storage and management of ontology definitions, used within the semantic web.

I would suggest someone like OSTG or the Mozilla foundation...

Sane?

Well, I wouldn't call it sane if anybody who is actively using XML and needs a DTD isn't hosting it right along with whatever web site they're using the XML for. Relying on somebody else to maintain a critical DTD that you use isn't sane. It's pretty dumb.

No

You shouldn't be using DTDs any more. Validation is better achieved with RelaxNG, and you shouldn't use them for entity references because then non-validating parsers won't be able to handle your code.

For those document types that already use DTDs, either you ship the DTDs with your application, or you cache them the first time you parse a document of that type.

The Netscape DTD issue was caused not by the DTD being unavailable, but by some client applications not being sufficiently robust. You shouldn't be looking at the hosting to solve the problem.

DTDs, XML entities and the non-breaking space

Unfortunately, DTDs aren't just for validation... they're also the only good way to define "entities" (e.g. "&foo;") in XML. This comes up a lot when trying to put HTML in XML feeds, because HTML has a lot of entities that aren't in the XML spec. Specifically, you may notice that you can't type " " in ordinary XML.

It's trivial to define "&nbsp;" yourself in a DTD, (<!ENTITY nbsp "&#a0;">) and many of the standard DTDs out there do define it, but by the XML 1.0 standard it's got to be defined somewhere or else the XML won't parse.

Don't know what a DTD is?

This can help:
http://en.wikipedia.org/wiki/Document_Type_Definit ion [wikipedia.org]

Doctypes are completely broken design.

The only other language I know of that even allows file sourcing over HTTP is PHP, and there it's a gaping security hole that defaults to off. In everything else, the dependencies *get installed to the local file system*.

Perhaps something like "pool.ntp.org"?

NTP.org" [ntp.org] maintains a pool of public NTP servers that are accessible via the hostname "pool.ntp.org", so perhaps something similar would work for a global DTD repository. An industry organization with a vested interest, the W3C seems like the most logical, could maintain the DNS zone and organizations could volunteer some server space and bandwidth to host a mirror of the collected pool of DTDs. Volunteering organizations might come and go, but when that happens it's just a matter of updating the DNS zone to reflect the change and everyone using DTDs just needs to know a single generic hostname will always provide a copy of the required DTD.

Just a thought...

using non-local cached copy considered harmful

Most tools provide a way to refer to a DTD on a public URL, yet use the local copy instead. (ie: taglib-location directive in java)

Doing anything else strikes me as fundamentally dangerous and insecure: it makes a remote dns vulnerability into an easy application DoS (or worse).

Call me crazy...

but just have your DTD as a W3C standard, distribute copies with your software, and don't bother a remote server until a new version of the DTD is released. Then distribute it with a new version of your software.

Seriously, what the fuck were they thinking relying on a server to be always available?

Re:Call me crazy...

It's not like someone would update it, and then give it the old version number.

Your trust in the world is cute. :-)

URI vs URL

A key mistake in your assumptions was brought up when the Netscape fiasco was news, and I will bring it up again...

"http://my.netscape.com/publish/formats/rss-0.91.d td" is a URI. It uniquely identifies a file. It *HAPPENS* to also be the URL for that same file, for now, but that is just a fortunate intentional coincidence. Your software should not rely on or require the file to be located at that URL. /var/dtd/rss-0.91.dtd is a perfectly valid location for the file identified by the URI "[whatever]/rss-0.91.dtd". What we need is for XML-using-software authors to support and embrace local DTD caches, AND package DTDs along with their applications (with the possibility of updating them from the web if neccessary).

It is silly that millions of RSS readers fetch a non-changing file from the same web site every day. It is only very slightly less silly that they fetch it from the web at all.

EXACTLY

Exactly right, but it is even worse than that:

A DTD spec SHOULD have both a PUBLIC identifier and a SYSTEM identifier. The system identifier is strongly recommended to be a URL so that a validating parser can fetch the DTD if the DTD is not found in the system catalog.

The system catalog is supposed to map from the PUBLIC identifier to a local file, so that the parser needn't go to the network.

If you are running a recent vintage Linux, look in /etc/xml/ - there are all the catalog maps for all the various DTDs in use.

So:

1. The application writers SHOULD have added the DTDs to the local system's catalog.
2. Failing that, the application SHOULD have cached the DTD locally the first time it was fetched, and never fetched it again.

XML Catalogs

I think there is an OASIS standard called XML Catalogs for redirecting offsite schema requests to a local copy...

Uhhm.... I thought we were using XML Schema now???

People are still using DTD's? I thought everybody switched to XML Schema a while back. God, I can't keep up with this constant flux!

I need some chinese food. Hmm...

Schezuan!

Not again

This has been covered before here and elsewhere... anyone who is using a DTD as a URL rather than a URI needs to be taken out and shot. I say bring them all down and let all the apps that rely on them die or be fixed.

Supply local DTDs with your app

I recently (within the last year) deployed an application that end users use for downloading and viewing custom content, and are intended to install the app onto laptops, tablets, and other portable devices allowing them view said content both on and off-line.

When prototyping our "offline mode", we ran into this exact same problem because the Xml APIs we used wanted to validate xml against online dtds. We ammended the validator's resolver to use locally embedded or cached dtds for all our doctypes, problem solved.

In in my app it was an obvious problem to solve because offline usage was a big scenario, but I could imagine that being "out of scope" for a less-than-robust website.

The DNS root servers are run by...

Wikipedia's Root nameserver [wikipedia.org] entry says that 4 of the 13 root nameservers are run by private companies.

I have a server in my basement we could use.

Linux box with an uptime of 153 days. It does have to go down now and again so I can clean the dust and cat fur out of it, but that doesn't take too long.

HTML 5

I don't know why important DTDs aren't just turned into serializations. HTML 5 (and, in practice, HTML in general) has a text/html serialization because the major browsers don't care about DTDs. It seems like well-published specifications like RSS should just be serialized and DTDs ignored, even though they are presented, instead of breaking when the DTD can't be found. I guess that wouldn't work if a generic XML parser was used for RSS, but for RSS readers, the DTD shouldn't matter.

DTDs are Useless

Quick, someone register http://all.your.dtds.are.belong.to.us/ [belong.to.us] :-)

Seriously though, we don't need dedicated hosting for DTDs. We need XML language spec writers, authors and user agent vendors to realise that DTDs are useless. Web browser vendors realised this a long time ago. No browser ever read HTML's SGML DTDs, and they do not use validating parsers for XHTML either (although, they use a hack to parse a subset of the DTD to handle XHTML and MathML entity references).

DTDs are bad for several reasons [hsivonen.iki.fi]:

1. DTDs pollute the document with schema-specific syntax. Since the document itself declares the rules, the question on answered by DTD validation is not the question that should be asked. DTD validation aswers the question "Does this document conform to the rules it declares itself?" The interesting question is "Does this document conform to these rules?" when the person who asks the question chooses the rules the question is about.

2. DTDs mix a validation mechanism, an inclusion mechanism and an infoset augmentation mechanism. The inclusion mechanism is mainly used for cheracter entities, which solve (but only it if the DTD is processed and processing it is not required!) an input problem by burdening the recipient instead of keeping input matters between the editing software and the document author.

3. DTDs aren't particularly expressive.

4. DTDs don't support Namespaces in XML.

Plus, if a UA needs to request the DTD every time it parses the file, that adds significant overhead by the time it fetches the DTD, parses it and checks the document for validity. It's just not worth it. The Netscape RSS DTD issue was a mistake, and it's time to learn from that. There are much better alternatives available for validating XML than DTDs, such as RelaxNG or Schematron.

Isn't this addressed already?

Isn't this what doctypes like this are for:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transition al.dtd">

That whole PUBLIC thing means that the browser can have its own copy so that it doesn't have to fetch it off the website. Is there a reason that this is not the standard way of doing this?

short answer: no

Validation is overrated. Especially, when it comes to RSS. There's so many competing "compatable" standards, that really aren't. feedparser.org [feedparser.org] has a great write up about the state of RSS. It's pathetic.

If you're reading a doc, don't bother validating it. You're probably going to have handle "invalid" XML anyway. When you're constructing XML, you should write it according to the DTD, but if you're relying on a remote site, then you're asking for trouble. Just cache the version locally, but seriously, you're tool shouldn't really need it. You're engineers do, but not the tool.

Finally, it's trivial to reconstruct a dtd from sample documents.

Builtin DTDs everywhere!

Now I may have not quite grasped the importance of DTDs, but I can think of only one scenario where retrieving a DTD from a to-be-determined location would be useful: Validating XML against any DTD. (Solution: Whomever wants to validate will also provide the DTD.)
To my knowledge any other application could just depend on builtin DTDs for validating the formats it knows and don't care about whatever it doesn't know as it wouldn't be able to intelligently use them, anyways.

Did I forget to take in account one of those nice tiny little huge details somewhere?

URLs were never sane

Think about it.

A URL has:

• A hostname
• A PORT number
• A path on that machine

The only one of those that the machine itself has any control over hiding from the user is the path, which can be virtualized. However, many aren't. DTDs certainly don't seem to be.

A distributed system for this kind of mission-critical information is what we need. Think DNS for documents, rather than just hosts.

XML catalog files let your app use local copies...

You can map public and system identifiers to local resources. Use them for dtds, schemas, stylesheets, etc. Here's the spec [oasis-open.org]. Google for more information.

DNS?

Do it like the DNS system, have a bunch of companies (*cough* google yahoo verizon mozilla microsoft (or not) *cough*) host this stuff, I doubt they'll all go bankrupt at the same time ;P

DTD Critical Hosting

Why doesnt the content provider just provide the dtd. Why have to worry about caching it or random errors poping up in it, when the DTD can be stored on the very same server as the website, or stored with the application. Then it doesnt matter if another company screws up or if some miliscious hacker decideds to attack the DTD it doesnt effect your product...
  Some might think well what if it changes?
well its obvious download the new one update your xhtml/xml or application to the specific changes.

the best host is localhost

Having anything in a live project linking externally is insane! I never understood how developers can risk this.

We use maven, use dtd's schemas wsdl etc. Much of the wsdl and other files refer to online areas. We download these and alter the references to be local. Otherwise we would have a build fail because of an internet issue, which is just nuts.

Same with maven, we have our own local repository where we keep a subset of what we use. Again same situation. In these cases this is just for building, I can't imagine doing this on a live site. This can especially go for externally referenced javascript... local copies are your friend.

Well...

Just flush XML and then it wouldn't be an issue...

guess who will want control over it

I am all for it but such a think needs to be international NOT in control of a company based in any country *cough* ICANN *cough*

Missing 4ml.org

Just the other day, I was looking for a DTD for 4ML related to music and lyric notations. But the website is not working. Most probably the guy got bored with it and forgot to pay the hosting company.

We definitely need some sustainable way to host the DTDs.

~Sivaraj

The security implications are extremely ugly

1) There are some sensitive environments (military, etc) where you simply do *NOT* connect your internal network to "teh interweb". No ifs, ands, ors, buts. The result is a broken browser where the DTD's are required.

2) Remember the incident where popular "safe" Superbowl sites were compromised and laced with malware-installing code? What happens to millions of Firefox-on-Windows users when a bunch of Russian mobsters or Chinese government agents hijack a DTD host and load it with a zero-day Windows exploit?

3) Remember "pharming", where DNS servers are hijacked to redirect *CORRECTLY TYPED URLS* to malware-infested sites. Even if the bad-guys can't hijack the DTD host, they can still hijack Windows-based DNS servers (ptui!) and anybody who relies on them gets redirected to a malware-install site.

That's the problem; here's my solution. It's composed of two parts.

A) DTDs will be *LOCAL FILES ON YOUR WORKSTATION* (excepting "thin clients").

B) Browsers (or possibly Operating Systems) will include new DTDs with updates. In posix OS's (*NIX, BSD) DTDs will be stored in /etc/dtd/ and users will be able to add their own DTDs in ~/.dtd
Windows will have its own locations. When you get your regular update for your browser (or alternatively, your OS), part of the update will be any new DTDs. There will be a separate file for each DTD and version, so that your browser can properly handle multiple tabs opening to sites using different versions of the same base DTD.

What organization?

Why, the same organization that should probably be responsible for *all* critical Internet infrastructure standards, just as it is responsible for the standards relating to telecommunications and radio communications.

The ITU [wikipedia.org] (also here [itu.int].

Go ahead, laugh, but I think it's long past time for control of such functions as DNS, NTP, assigned numbers, et cetera, to be transferred out of the hands of primarily US-based corporations and loosely coupled organizations such as the IETF and IANA and into the hands of some sort of international treaty organization.

Since the ITU not only fits this description, but in fact was founded to deal with precisely these sorts of issues, why not let it do what it does for the Internet as well?

Stupid question :)

What organization would be the likely custodian of such data?

Is it not obvious that it may as well be the W3C? XML is their standard, operating a registry for public-use DTDs would be a rather reasonable service to provide..

Bad idea any way you slice it

As many people have alluded to already, it's an incredibly bad idea to make your application have an unnecessary dependency on an external service. Keep a local copy, just copy it down once and you're done, simple as can be.

But maybe there are urls out there pointing to "the latest and greatest" version, rather than a specific version, and you like the idea of using "the latest and greatest". So, think for a moment what happens when the DTD/schema changes. Is your app magically going to change how it deals with the xml at the same time? Of course not!

So, until you can get out a patch, you'd be refusing xml docs your code/xsl has been built to handle, and possibly letting in xml docs that your code/xsl has not been built to handle. Whereas, if you just kept a local DTD/schema, you would have no trouble keeping it, and the code/xsl behind in in sync.

No...

...they are unique identifiers, not URLs.

They don't need to be hosted for the same reason that there isn't a machine out there called com.sun.java.util.dates.FunnyDate

Re:hmm...

Yeah? Well whatever you can do, ICANN do better.

Re:XML? What?

Well it is, isn't it? And if it doesn't, there's always XML 2.0