Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?

With XML, is the Time Right for Hierarchical DBs? 276

DullTrev asks: "The hierarchical database model existed before the far more familiar relational model. Hierarchical databases were blown away by relational versions because it was difficult to model a many-to-many relationship - the very basis of the hierarchical model is that each child element has only one parent element. However, we now live in a web world that demands quick access to a variety of data on a variety of platforms. XML is being used to facilitate this, and XML has, of course, a hierarchical structure." Do you think a hierarchical database would really be a better answer for storing XML data over the existing relational counterparts?

"There have been some pushes to create pure XML databases (info on XML in connection to databases is here and info on XML database products is here) with claims that as they support XML natively, they can offer many advantages over relation databases.

Some of these claims include speed, better handling of audio, graphic and other digital files, easier administration, and handling of unexpected elements. Software AG, a German firm, produce and sell a suite of XML products, including Tamino, a native XML database. They have lots of information on why they think there database is great, not surprisingly, but no benchmarks. So, do the Slashdot community think that with XML the time has come for hierarchical databases? Or is it better simply to use a relational database that can output in XML, or script your way to achieve the same goal?"

This discussion has been archived. No new comments can be posted.

With XML, is the Time Right for Hierarchical DBs?

Comments Filter:
  • I found this SQL queried XML database in PHP [f2s.com]. Seems very kewl.
  • Hierarchical databases won't take over because they're relational counterparts are already so well developed. A relational database can do everything a hierarchical one can, with few exceptions. Even if there is a slight gain to using a hierarchical system, there are much fewer solutions, and consequently the one's that do exist aren't as well developed, so implenting one is more difficult.
    • by Netmonger ( 3253 ) on Sunday November 18, 2001 @03:53PM (#2581605) Homepage
      I don't agree - look at LDAP. The benefits for LDAP'fying services is clear. With a hierarchial database, specific queries can target a subset of the entire database, without the over head of having seperate tables and/or database for varying information. For keeping track of 'real world' objects: People, Printers, IPS, etc.. the advantage is that the system used to organize them is similar to the actual grouping going on. Managers have employees 'underneath' them. Its basically taking the organizational concepts used for filesystems and applying them to database design. I havent done any performance testing LDAP vs. SQL for similar schema setup, but from what I understand one of the other benefits is fast lookups. Sounds like a good project! To implement databases in both LDAP and SQL and measure the performance of similar queries!! :)
    • by drodver ( 410899 )
      Why do you assume relational databases are more developed than hierarchical?? The company I work for has been using our own hierarchical database for 25 years. They had the potential to become what Oracle is today but decided to stay focused on the medical industry. The serious problem with relational databases is they have traditionally not handled sparse data well at all. In the case of a patient every time they come for a visit there are tens of thousands of possible data points that can be entered, but most usually are empty. For tasks such as these relational databases have been completely impractical. With the use of indexing a heirarchical database can do everything a relational database can do.
    • Horses for courses.

      LDAP is an example of a really good and well developed implimentation of the hierarchial database idea. However, try keeping track of whay your customers bought from who with LDAP. So while LDAP (and other hierarchical dbs)do certain things better, don't try to run a CRM suite off one.

      The basic problem is that the entire database is rarely hierarchical in nature even though some queries may be.
      • by tzanger ( 1575 )

        LDAP is an example of a really good and well developed implimentation of the hierarchial database idea. However, try keeping track of whay your customers bought from who with LDAP. So while LDAP (and other hierarchical dbs)do certain things better, don't try to run a CRM suite off one.

        LDAP isn't designed to do that. It's funny that you picked a CRM application, because that's the type of thing I've been playing with.

        Everyone that comes in contact with our company goes into an LDAP directory (benefits: works with almost every email client, replicates great along the boundaries we have, provides logical/protocol barrier between the contact data in the directory and the business data in the RDBMS) and then Postgres takes care of the actual relational (business) data. The ties between the RDBMS and the directory are done by DN; the directory format was carefully designed to avoid DN changes while still making the DN "make sense" when browsing the tree.

        Our products, once manufactured, are assigned a serial number and entered into the directory as well, under a different node. We get the benefit of being able to track our product like we can our customers and the RDBMS takes care of all the stuff that changes on a frequent basis (trouble tickets for Customer Service, quotations, acknowledgements, shipping schedules, etc. The directory is only used to store the data that shouldn't change (or will only change very infrequently) during the lifetime of the entry. Looks good on paper; We'll see how well it works in reality. :-)

  • Reversed Question (Score:5, Insightful)

    by devnullkac ( 223246 ) on Sunday November 18, 2001 @03:47PM (#2581585) Homepage
    From a purist perspective, I suspect the question is actually reversed: we shouldn't be talking about "XML data" is if it was somehow the core representation. It's usual intent is as a transmission format and, as such, needn't correspond directly to the organization of the source data.

    Rather than discard the advantages of relational and object databases, should we instead ask how XML can be used to represent those kinds of relationships?

    • Having worked with representing just the table layouts in XML, its really not that hard to represent say, a NxM set of data. That's always been easy, its a 2d table. It winds up being a very shallow xml document, no more than say 3 levels: root node, node representing the data record, node representing an entry in the record.

      You are 100% right, in that we should discard relational db's. Objects are a little more natural for a representation in XML. If an object contains objects, even if they are of the same type, ala trees, its a more natural representation than a 2d table.

      • There's already something that works along these lines called ODBCSocketServer. Basically it's an ODBCServer that sits on your WinBloze box and ships out the results in XML. Something like:

        <row><col name="foo">6</col>
        <col name="bar">john</col></row>

    • by Florian Weimer ( 88405 ) <fw@deneb.enyo.de> on Sunday November 18, 2001 @04:34PM (#2581723) Homepage
      You have a point. In addition, we should ask ourselves: "Do we really need XML if it doesn't fit in our established technology framework?"

      Often, the answer is a plain "No", from a technical standpoint. However, you have to market your product somehow, and this means that you need Java, Linux, LDAP, XML, and SOAP. (As time passes, some entries will drop off the beginning of this list, and others will show up at the end.)
    • XML is a poor choice for data storage. Arguably it is also a poor choice for data modelling as it does not have a strict constraint model.

      Relational databases are here to stay and will be with us for at least the next fifty years. It is better to think of ways of translating relational data than supplanting it.

      • Relational databases are here to stay and will be with us for at least the next fifty years. It is better to think of ways of translating relational data than supplanting it.

        Yeppers - taking the LDAP example, the best of both worlds would be to keep the actual data in a relational DB, and use a tool to "publish" it as an LDAP directory, or just use an LDAP interface to that data, along with an indexing scheme that optimizes for LDAP-like queries.

      • Yeah, it'll be around for the next fifty years (or more). Just like COBOL. Better to find ways of interfacing with COBOL than to look at other languages in your development environment. Yes, that was sarcasm.

        Strict constraint model? Like datatypes (XML Schema)? Like document structure (XML Schema)?

        XML has this model. It is simply not as used yet (not as optimized yet). There is nothing inherent to XML that precludes its use for data storage. The fact that it is plain text in its *serialized* form is immaterial to its internal storage format in a hierarchical database. Nor does the fact that it's XML preclude the possibility of indexing the information just as you would a table column.

        Something that relational databases are not as good at handling: web accessible data where the data does not allow for rigid guidelines. For example, in a web magazine, many articles are somewhat structured with author, date, title, etc., but otherwise tend to be very free-form.

        Not a problem in and of itself, but what happens when you try to search it? How do you differentiate between a search for info in the title of a component use case and a search within a biliography? So you create a relational model that handles an arbitrary number of use cases and biliography entries -- all indexed by article. But some use cases have more information than others. Some have associated graphics. Suddenly we are shown not a many-to-many relationship, but a many-to-many-to-many-to-"Aw screw it. It'll take two days to query" relationship. Do you put markup data in the database? A regular expression on all of the content? Yeah, THAT's efficient.

        We tend to think that anything that we put into a relational database can be adequately represented in XML. And we'd be right. Unfortunately many people believe that the reverse is also true. It is not.

        Others have made the point of LDAP and naively assumed (as I once did) that a full blown relational database on the backend would be a better solution than the pansy in-process, flexible data model, file-based BerkeleyDB that's commonly used. What was found? User queries (VERY common query) from a listing of 400 users took about five seconds with BerkeleyDB behind OpenLDAP, and over a minute (!) using PostgreSQL behind LDAP. Why? The overhead involved in trying to represent a hierarchical tree in a relational model proved to have more overhead than it was worth.

        An object database may have performed better than the relational model, but if you are mainly handling text or simple datatypes such as dates and integers (as most databases do), why not use XML and optimize for that case?

        People scream that their relational database is enough and can be used for anything that an XML database can be used. These people sound very much like people screaming that a singly linked list is inherently better than a red-black binary tree. After all, they both hold data just as well. In fact, a linked list does it more efficiently (look! fewer pointers!) and there's nothing stopping you from sorting the singly linked list (plenty of efficient algorithms already out there for this). Yes, that was also sarcasm. And objects are useless, just use C. :-/

        Use the right tool for the job. In many (most?) cases,a relational database fits the bill. Sometimes an object database is called for. Sometimes a hybrid of the two. Is it so hard to accept that maybe, just maybe, when the only thing that you do is XML processing and XML data sharing (more and more common these days) that a dedicated XML datastore might be what the doctor ordered?
    • Rather, non-relational databases already exist -- there's lots of object databases. And when you consider that many of these object databases support objects that can also be serialized to XML (e.g., for XMLRPC), then you see that they are pretty much orthogonal to an XML database.

      Object databases are being used more and more, I think -- though they aren't taking off or even biting into RDBMS's much...

    • we shouldn't be talking about "XML data" is if it was somehow the core representation.

      Exactly. The question of whether to use hierarchical databases is orthogonal to the question of whether to use XML. NetInfo, the DNS, LDAP, and smalltalk source code trees are all good examples of where you should use a hierarchical approach, and this was true long before anyone thought of XML.

  • by disarray ( 108 ) on Sunday November 18, 2001 @03:52PM (#2581603)
    Wouldn't object-oriented databases qualify as hierarchical (or some of them, at least)? A rather lengthy story [slashdot.org] ran a while back covering various reasons why object-oriented databases are useful, followed by various comments on cases where they aren't and why they aren't as common as relational ones today. The bottom line seems to be that they are in use today. One notable example comes to mind: LDAP. The aforementioned story has more. Despite the rather preachy tone, it's an interesting read.

    1337ness for sale. [ebay.com]

    • Wouldn't object-oriented databases qualify as hierarchical (or some of them, at least)?

      Object-oriented databases are what used to be called network databases, and can represent arbitrary graphs. Any network database can be hierarchical, just by imposing some limitations on the kinds of likages that are allowed. In fact, network databases allow the most flexible data structures; anything you can build with pointers.

      In fact, the correct model for storing XML data *is* a network model. The relational model obviously doesn't fit, but although it's less obvious, the addition of the XLink specification to XML means that the hierarchical model doesn't either. XML documents can have arbitrarily complex structure because of all the pointers -- they map perfectly onto an OODB.

  • by hodeleri ( 89647 ) <drbrain@segment7.net> on Sunday November 18, 2001 @03:52PM (#2581604) Homepage Journal
    XML has, of course, a hierarchical structure

    Just because XML is a hierarchical markup language does not mean that it can only be used for hierarchical things. Perhaps you should look at RDF [w3.org] which can use many to many mappings through resources and groupings (sequences, bags, and alternates). (A resource in one grouping can refer to another grouping i.e. many to many.)

    • Just because XML is a hierarchical markup language does not mean that it can only be used for hierarchical things.

      Yes, but XML is hugely inefficient for table structures, because of all the redundant metadata.
  • Discussions (Score:3, Informative)

    by Lozzer ( 141543 ) on Sunday November 18, 2001 @03:53PM (#2581608) Journal

    There is lots said on this over at Database Debunkings [firstsql.com]

  • XML vs. ERwin (Score:3, Insightful)

    by imrdkl ( 302224 ) on Sunday November 18, 2001 @03:53PM (#2581609) Homepage Journal
    IANADG, but the folks that do our models still use good, old, ERwin. Something about the relationship-specification capabilities, I guess. I was not aware that XML limited number of parents specifically. You sure that ain't just a limitation of your programming language? :)

    An afterthought, databases are about storage and speed of insertion/extraction. I honestly don't believe that fitting the database to the data structure is worth the cost or the trouble, just yet.

  • No Chance... (Score:3, Insightful)

    by augustz ( 18082 ) on Sunday November 18, 2001 @03:54PM (#2581610) Homepage
    I think these discussions come up part of the time because people want something new and sexy. In this case OO DB's, which 'XML DB's' are a variant of, may have benefits in specific and limited cases. But I have not been impressed.

    Take your classic orders table. Part NO, Custoemr NO, etc. etc. The number of apps with only one parent is tiny, the flexibilty limited, and the whole metadata scanning business awkaward.

    For anyone doing and serious larger scale database work some of this stuff is a joke. The idea these vendors have is that we'll be storing XML data in these DB's, ignoring that even for a simple phone directory, the XML data probably takes up a significantly greater amount of space than a simple relational DB would require

    And this ignores the significant amount of time and energy invested in toolsets and models for the existing setup. Sure, someone might come out with a chip that runs 2x as fast as an intel at the same price, but unless it is intel compatible how many people would buy it or care?
  • Indexing? (Score:4, Insightful)

    by aralin ( 107264 ) on Sunday November 18, 2001 @04:00PM (#2581625)
    Anyone can explain to me what is suddenly so wrong about relational database with hierarchical indexing?

    Maybe its just me, but the goal today is integration and having a special database for XML and special database for this and that just because its faster for this particular problem creates such a level of complexity, which prevents accomplishing even of the most trivial tasks.

    Still, XML is only a way how to describe data, that might be often in their structure relational. Why do not store data in their native form and create XML documents out of database on fly by filters?

    This question of hierarchical databases is just plain trolling in my eyes.

    • Still, XML is only a way how to describe data, that might be often in their structure relational. Why do not store data in their native form and create XML documents out of database on fly by filters?

      Quite. Not only would the XML markup probably take more space than the data itself, but storing it as XML seems to be not only pointless, but also a little shortsighted. What if your XML spec changes? What if you want the data in another form?

      Just storing the data and then dynamically creating the XML doc on the fly is sooo much easier.
      • Re:Indexing? (Score:2, Insightful)

        by captredballs ( 71364 )

        The problems that you mention, both concerning storage space and flexibility of the data model are what XML databases are attempting to solve.

        Listing the problems in opposition to the solutions does not make for a good arguement.
  • In my experience with XML and RDBMS systems, mapping one onto another is always a dicey task. The primary reason (IMHO) is that XML's ability to represent order as well as structure as data doesn't fit into an RDBMS database without some work. I've seen people try to map both XML and regular DB's onto each other, and my opinion is that the results don't "feel right" on one side or the other unless great pains are made to preserve the structure of the XML doc in the DB schema.

    That said, I'm not sure a hierarchial DB will necessarialy be any better than something like an OODBMS with well-modeled objects.
  • I don't think that heirarchical db's have any real chance of taking over or replacing relational dbs in the future. There may start to be more of a place for them, but many application service providers that use XML still have a fair amount of relational data that needs to be maintained. XML is mainly being used for communication protocals and not so much for internal data structure storage. I think the more likely db trend in the future will be for many users to maintain both relational and heirarchical databases..
  • A data format will NEVER dictate a system's design. XML is nothing other than a data format.

    The relational model has no major shortcomings. The only thing XML offers that is not already very well done is easier data interchange. As a database administrator, I can tell you there is NO chance XML will dictate a change of how we store data. There are much higher priorities in database management than easier data interchange.

  • by coyote-san ( 38515 ) on Sunday November 18, 2001 @04:14PM (#2581660)
    Relational databases didn't come to dominate the database market because they pushed aside equally valid alternatives, they dominate the market because relational databases implement relational calculus. Indeed, that's the very touchstone that distinguishes relational databases from something like DBM and its many descendants.

    And *that* is important because it assures the desiger and user that every possible operation is well-defined and (hopefully) correctly implemented. The exact syntax for a "join" may differ, and a specific implementation may be flawed, but everyone agrees to a common baseline.

    For hierarchial databases to really take off, they need to have an equally strong mathematical underpinning. For now, AFAIK, there is none other than that you get when you map a hierarchial database into relational tables and use exactly those relational properties. That's a good start, but if you're only using the properties in relational databases, why not stick with them?

    As for XML, that's completely irrelevant. It's a good format for transferring data, but that's about it. You can store hierarchial data in an XML file, but you can also use it to store purely relational data or completely unstructured data (in some CDATA block).
    • everything looks like a nail. The relational model is pretty good for its original purpose: allowing non-specialists quick access to large amounts of statistical and business data (sales records, etc.) via an easy-to-learn query language. But for many other applications, it has proven to be completely insufficient.

      Indeed, that's the very touchstone that distinguishes relational databases from something like DBM and its many descendants.

      The alternative to relational databases is not "DBM", it is object oriented, tree structured, logical, and other kinds of database models. Those are just as well defined as relational databases.

      And *that* is important because it assures the desiger and user that every possible operation is well-defined and (hopefully) correctly implemented. The exact syntax for a "join" may differ, and a specific implementation may be flawed, but everyone agrees to a common baseline.

      Relational databases provide a common baseline for a primitive set of relational operations. Real-world implementations of those models have been augmented by zillions of operations that weren't part of the original relational model and that often don't even fit into the relational model. And without those extra operations, relational databases would not be useful in practice.

      For now, AFAIK, there is none other than that you get when you map a hierarchial database into relational tables and use exactly those relational properties.

      Are you kidding? It is a major pain trying to express hierarchical data in a relational database model: the relations that describe hierarchical data and the operations that you might want to execute often require complex, multiple, inefficient queries and updates, and the relational model provides few tools to ensure that the corresponding relations remain consistent.

      The semantics of tree structures are trivial to define. People do it in programming language classes all the time. And it is trivial to formulate a database model corresponding to it. In fact, if you have an object-oriented database that respects language semantics, you get hierarchical databases automatically when you define an abstract tree datatype.

      Still, so-called "relational" databases will continue to dominate the market for a long time to come. That's not because the relational model is particularly well-suited to a lot of applications. In part, that's because "relational databases" are not purely relational anymore: they generally include numerous facilities for object-oriented and hierarchical databases, under a "relational veneer". They even include the old "navigational" database systems, combined with the widespread use of stored procedures that do whatever they want whenever they want it on the database server.

      In different words, traditionally relational databases will provide increasingly better support for hierarchical and object-oriented data, but they will continue to also support the relational model, as well as relational access to these other data types. And newly developed databases with other kinds of data models will provide an SQL or other relational frontend to their content. And marketing will continue to include "something-relational" in all the advertising because otherwise the old database hands won't buy it.

    • by Zeinfeld ( 263942 ) on Sunday November 18, 2001 @09:39PM (#2582670) Homepage
      Relational databases didn't come to dominate the database market because they pushed aside equally valid alternatives, they dominate the market because relational databases implement relational calculus.

      That's rubbish. Back in in the 1960s when the first relational databases emerged nobody had a formal specification for a relational calculus. Today we can create a formal calculus for any data model, the Entity relational model is no different in that regard.

      SQL is a very 1960s / COBOL way of looking at a data structure. Most of the people using it simply do not have the breadth of experience of other data models to know its strengths or weaknesses. Most of the posts in the thread are as empty as those in an editor choice flamewar.

      The entity relationship model has been discarded by the programming language community in favor of typed set theory. Java and C# both have representations of sets, lists, etc., the only reason to use an entity relational model is to get persistence for the data structure.

      So you get this impedance mismatch and a pile of code whose sole purpose is to rewrite the data structures used in the program so that they match the data structures used in the persistence store.

      What we need is a persistence store with a data model that matches our programming language data model. Unfortunately most of the attempts to do this are half baked. All it should take is to add transaction statements into the language so that you declare a procedure to be transactional, it will be all or nothing.

      Unfortunately Sun made a pact with Oracle over Java and so they have remained stuck in the obsolete SQL world. C# looks to me to be a much better opportunity, Microsoft has little to lose from unifying the data model of the language with that of the persistence store and everything to gain.

      • by tim_maroney ( 239442 ) on Sunday November 18, 2001 @11:13PM (#2582959) Homepage
        So you get this impedance mismatch and a pile of code whose sole purpose is to rewrite the data structures used in the program so that they match the data structures used in the persistence store.

        Exactly. What's more, this pile of code takes months to write even for a few dozen object types; it doesn't understand the idea of dependencies between objects so you have to add a whole layer to make sure that objects get persisted in the right order; it's incredibly hard to change, so the system design can't iterate; and simple objects like collections proliferate tables to the point of significant performance losses. It's a terrible way to build a software system unless the user model just happens to be adequately modeled by a fill-in-the-blanks table.

        This is why serious applications traditionally roll their own file formats. It's actually less work to manage most data models from scratch than it is to map them into the straitjacket of a relational database. Custom file formats serve in essence as hand-rolled object databases. Unfortunately, the rise of the three-tier client-server architecture has made the RDBMS layer an unquestioned assumption, with the result that modeling two dozen object types winds up generating over 50,000 lines of convoluted, slow and buggy source code. Modeling the same objects from scratch on a custom B-tree would take less than one fifth the code size. Doing it in a good ODBMS would be almost as trivial as specifying the data structures in XML.

        On my latest project, we ran into a strange issue when specifying the user interface of a discussion system. The designers wanted to mark read and unread messages per user -- in other words, functionality critical to providing a friendly user experience, which rn had fifteen years ago. The engineers hit the roof and said it was impossible. It turned out the reason was that this is an intrinsically hard problem on an RDBMS, although it's a trivial problem to solve in a hand-rolled .newsrc text file. Over the course of the project we ran into tons of these issues, and the interface design took a severe beating because of compromises to the limitations of an RDBMS back-end.

  • by Carnage4Life ( 106069 ) on Sunday November 18, 2001 @04:16PM (#2581671) Homepage Journal
    I wrote a paper on native XML databases and SQL databases that support XML [25hoursaday.com] that appeared on Slashdot [slashdot.org] a little while ago. While doing research for that paper I asked myself the same question, whether instead of coming up with hybrid methods to store relational and hierarchical data we should store XML in already existing hierarchical databases. Unfortunately things are not so clear cut.

    First of all, a lot of data out there is relational and people aren't ready or willing to transition all that data to XML based storage so mixing of relational and XML data will probably be with us for a while. The biggest problem with object oriented databases is that they didn't understand this fundamental issue but it seems that with XMKL databases the vendors understand that hybrid data will be with us for quite a while which is why Tamino supports importing data from relational sources and even ships with a SQL engine.

    Secondly, XML documents have a lot of metadata beyond the hierarchical parent-child relationships such as processing instructions, comments and entities which are require more intelligence in the support from the database than just storing parent-child relationships.

    Finally all the major [commercial] relational database vendors have included some sort of native suppport for XML including XML types and there is a an ANSI standard in the works [sqlx.org] for combining XML and SQL. From what I've seen, none of the hierarchical databases plan to support XML as much as the relational databases have or plan to.

    Now if you were simply asking whether a native XML database can be built on top of a hierarchical database then I believe the answer is yes. Then again native XML databases can and have been built on object oriented databases and relational databses so it makes sense that they can be implemented in a database system that is more suited to handling hierarchical data.
  • XML Data Bloat (Score:2, Insightful)

    by trp0 ( 155951 )
    It certainly seems like the same thing is happening with XML that happens with any new toy: "my friend told me XML was cool for stuff, so I'm going to convert everything to XML so I can be cool too."

    I was pretty sure that XML was useful in that it was a human-readable data-encoding mechanism that "average" users could get a grip on and utilize in sharing information between heterogenous systems, but it seems like people are completely missing the point these days in how to use XML effectively.

    A lot of the benefit of using XML is quickly becoming negated by everyone coming up with their own DTDs and the lack of standard formats for encoding data that is to be shared. As an example, here at the university I attend, there is a project for sharing information about biological species' population data amongst sister organizations. The goal is make the information possessed by all these organizations available to all the others. The trouble is that they have all come up with their own format for storing the data they collect and can not agree on what standard should be used, so each organization is encoding all their information with a different XML labeling scheme. My first questions was: "Why in the heck are you using XML to encode the data anyway?" Seems easier and saner to just store it in your relational database and make the database accessible to sister organization who can then encode the information however they want for their end-users through their client applications rather than the organization holding the information imposing order on people wanting access to the information.

    To make a long story short, XML encoding doesn't help you store the information more efficiently at all and with the state of the "formatting standards" today doesn't even really provide an efficient way of sharing information between organization or an efficient way of encoding the information for transmittal to other organizations. It seems as if people are missing the forest for the trees in how XML can be useful in its relation to data encoding and we should stick with our trusty ole relational and object-oriented database models as they have shown their usefulness and efficiency.
  • by bwt ( 68845 ) on Sunday November 18, 2001 @04:24PM (#2581693) Homepage
    Or is it better simply to use a relational database that can output in XML, or script your way to achieve the same goal?"

    I believe that RDBMS's should add functionality to read/write XML, especially as the XML Schema recommendations is basically done.

    The idea that XML should be the permanent storage format is a bad one. There is a lot of power in a normalized data model -- it enforces data integrity , while eliminating data fragmentation automatically and it minimizes transaction resources.

    Consider XML representations for different entities that all share some kind of child entity. For example: people, businesses, and schools all share addresses. In XML, you want the addresses to appear in the description of the individual object. Does that mean you want to store the addresses separately that way? Absolutely not, because then when you enforce constraints or ask questions about addresses, your data is fragmented in three places. For that matter, how do you know all the entities that might use addresses? In an RDBMS, you can inspect all the foreign keys to the address entitity. What's the XML analog?
  • Pros and Cons (Score:2, Informative)

    by Multispin ( 49784 )
    I work for a company that has been doing hierarchical DBMSs for years. The company is Applied Technical Systems [apptechsys.com]. We make a database engine called CCM.

    XML is a great way for exchanging data, but the term XML databases is very misleading. If the database engine actually stores data in native XML, it's going to be *very* slow. I think the point behind XML is that nobody should really have to care what your backend is as long as you can export reasonable XML. Note that I say reasonable XML. And XML export that simple encodes the rows and fields in a table to XML with <row> and <col> tags is NOT reasonable. It conveys no actual knowledge of the real structure of the data.

    Storing XML data in a relation DB can either be a very hard problem or a very easy one. Let me explain.You could look at some XML and define a DB schema for it, not too hard to do. Problem? It's not generic; a human has to re do it each time the XML structure changes. The alternative is to store it all in one big table and index the hell out of it. Problem? It's slow. At that point you aren't using any structure of the XML or the power of relational DBs.

    I'm a firm believer that efficient XML storage, querying and retrieval will require a hierarchical database. The problem is that there's several features (bugs IMHO) in XML (and XPath) that, in a way, are throwbacks to relational DBs. IDREFs and the notion of document order particularly bug me. I ran into these this summer when I was on a team trying to build a XPath and XQuery front end for CCM.

    We're gradually seeing the XML world change. Early XML documents were similar to the type mentioned above. They were flat. When you start adding depth the information inherent in the structure of the data becomes apparent. Another thing I'm glad to see the industry move away from is the notion that XML resides in files. Many (if not all) of the early XML parsers made this assumption. It was a pain in the ass to parse from some other source, like a buffer in memory.
  • Repeat after me ... (Score:5, Informative)

    by Serpent Mage ( 95312 ) on Sunday November 18, 2001 @04:32PM (#2581717)
    XML is not a magic bullet. Relational database won out over the Hierarchical model for a lot of reasons. For instance, there exists a number of integrity constraints with the Hierarchical model such as

    1) No record occurrences except root records can exist without being related to a parent record occurrence. This means that
    a) a child record cannot be inserted unless it is linked to a parent record.
    b) a child record may be deleted independently of its parent however, deletion of the parent record automatically results in the deletion of all its child and descendent records.
    c) the above rules do not apply to virtual child records and virtual parent records.

    2) If a child record has 2 or more parent records from the SAME record type, the child record must be duplicated once under each parent record.

    3) A child record having 2 or more parent records of DIFFERENT record types can do so only by having at most 1 real parent, with all the others represented as virtual parents. IMS limites the number of virtual parents to 1.

    In addition to these flaws, relational databases have had over a decade to become mature, optimized, and enterprise scalable. Harddrive partitioning for such databases as oracle work out perfectly with the cylinder, sector, and tracks of a hard drive to allow for the fastest read/write times as can be possible.

    Too often people see that XML "can" do so many things and decides that it should be the way things are done but XML is NOT a magic bullet and just because it has the potential to do something does not make it the best methodology for doing so.
  • I was surprised to see so many questions of the form "what's wrong with relational databases"? Relational databases have a well-known problem called "impedance mismatch" when mapping multi-linked object structures. Many links on the impedance mismatch issue can be found at this Google search [google.com].

    Anyone who has tried to take a natural set of application-side objects and map them onto a relational database is already quite familiar with the problems created by the proliferation of tables needed to map simple application data structures, as well as the large amount of development effort needed to deal with simple relationships that would be trivial to specify in an object model such as Java's or XML's.

    There is clearly a need to move on to object databases, but installed base and skill set inertia have blocked this transition, with the result that database-oriented applications have remained hamstrung in their friendliness and feature set.

    • There is clearly a need to move on to object databases, but installed base and skill set inertia have blocked this transition, with the result that database-oriented applications have remained hamstrung in their friendliness and feature set.

      The "impedance mismatch" is little more than the fact that object oriented approaches generally do not obey the rules of 3rd normal form data modelling, especially in the way they represent many-to-many relationships. If anything, it's a problem caused with object orientation and it's assumption that efficient software development overhead is "the goal". That's true if the data is throwaway or only persistent in small quantities. When the amount of data is large, and you are paying for "big iron" to support many simultaneous users and transactions, as are both typical for enterprise grade applications, the software reuse benefits of object oriented methods lose significance relative to structural data integrity enforcement and transaction efficiencty.

      OO works great in the GUI and business rule layers, but consider the way OO represents many-to-many relationships. For example, suppose I have students and courses. Generally, I might have students with a collection of course objects or vice versa or both. If you use both, then you've got redundant data and ACIDity and data integrity will add resource overhead and complexity. If you put the collection in only one of the objects (say in the student object), then when you ask a question like "who are all the students in class X" then your application will crawl as you have to ask every student who exists if they are in taking that class. If there are a couple thousand students, then it's not a big deal. If there are 400 million, then it is a very big deal.
  • Some thoughts... (Score:5, Interesting)

    by Coventry ( 3779 ) on Sunday November 18, 2001 @05:10PM (#2581822) Journal
    I have been struggling with these issues for awhile now, for various reasons. Why? Because I like Zope [zope.org], but am, like most developers, more comfortable with relational data structures.

    Zope uses an object database known as the ZODB. Some forms of many-to-many relationsships and such can be handled via the use of selection and multi-selection properties, which are designed to distinguish between a selected element and the list of available elements. The list of elements can be derived from a property on the current object, a property on a parent object, or be created via a method call - allowing for non-traditional (for OODBMS) cross-linking of objects. Of course, since this sort of thing is a workaround, no true relational links are created... 'Soft Relations' may be ok for MySQL [mysql.org], but in big application development, relationships must be enforced! Thus, the big-boys in RDBMS all enforce foreign keys (mysql does not)...

    Of course, I've found that by careful creation of object heirarcies, very complex applications can be created on top of a OODBMS that are in fact more robust, in some ways, then the relational couterparts. The Bigest hurdle (Short-term) I see to OODBMS (including ones based upon XML [the ZODB can export objects as XML but they are stored differently internally]) is the lack of a true query and data manipulation language - like SQL. Sure, OQL exists, and is even technically a standard, but it A) sucks and B) is geared towards large java applications with huge amounts of active objects, not general purpose OODB queries. Thus, without such language, OODBMS are all disimilar in how one queries and creates/updates data, and in many cases, the only interface is a truely procedural one! Thus OODBMS are forced to use proprietary tools, and are locked into one system - not to mention speed of development (something normally associated with OO development and OODBMS in general) is hindered by the excessive amount of procedural calls one needs to simply query thier data...

    Recently, an add-on to Zope addressed some of these issues. Called 'ZOQL' - it uses a SQL like syntax and allows for very discrete querying of the ZODB (something one had to do programatically using the 'ZCatalog' before) with all of the familar aggregate and comparison operators SQL users love... Of course, this _still_ doesn't address the issue of soft-relationships:

    I think the bigest hurdle to OODBMS in the long term (tools like ZOQL are interfaces to existing systems, thus can be mplemented easily) is the lack of handling relationships. It seems that most RDBMS force a developer to think in Relational terms about the data, and most OODBMS force you to think in terms of objects... Most problems can be mapped to either of these domains, but you are forcing the data-model-type onto the problem. What is needed is a hybrid system, an 'Object-Relational' DBMS. This is to say that OODBMS system makers desist with the traditional OO idea that relations are of the following types:
    • Object A is a Object B
    • Object A Has a/many Object B(s)
    What RDBMS systems excelled in (and thus fell into pupular use for) was ease of management and allowing common data to be moved and grouped. A 'Look-up Table' - for instance, which simply holds a list of common data (an enumerated list) and can be centrally maintained is a Boon in the RDBMS world. For example, you have a lookup table of car manufactureres, and one of them changes its name... Instead of updating all N Cars that are made by the manufacturer, you simply update the single record in lookup table. Since each car would have somehting akin to a 'Manuafactuer_ID' column linking it to the lookup table, the Cars belonging to the manufacturer are all taken care of.

    How does one do this in a hierarchal system? Well, the easy answer would be that each manufacturer object contains all the cars that manufacturer makes. Simple, right? WRONG. Why?

    Because each car also has a body-type (compact, sedan, SUV, truck, van, etc...) - which in a relational database would simple by another lookup table, but in an OODBMS poses data management issues. Do we put body-type higher then manufacturer? If so, then we have to maintain the list of manufacturers for each body type, causing headaches. Or do we put body-type below manufacturer, causing us to need to maintain a seperate list of body types for each manufacturer - these lists of course need to match exactly if we ever plan on being able to search or do reports based upon all cars of a specific body type.
    Sadly enough, this sort of seperate-enumeration-relationship isn't implemented (well) in any OODBMS I've found.
    Take the ZOBD for example, its selection and multiselection lists Try to handle this situation, but fail because relational integrety is not maintained! That is to say, behind the scenes it's not a true reference to a value in the enumerated list, but just a text entry representing a value in the list. If the value in the list changes, the selection-property does not update, leaving you with the equivilent of MySQL's bastard-children, the orphaned records.
    This sort of soft-relationship handling is Ugly and BAD for maintainaility, but OODBMS users are faced with two ugly choices each time they map such a relationship: Do I store this as a plain-text property and just update N records each time this changes, or do I map it into the hierarchy and deal with the headaches incurred by doing so...?

    I don't think I've answered the question, but hopefully I've at least shed some light on the subject for members of both the OODBMS camps and RDBMS camps... Now if only a useful ORDBMS were to come along...

    (Note that PostgreSQL and some other RDBMS actualy can be used in a semi-OO manner, but this is usually reserved for inheritable structures of data to be used for specific extensions to the data model - thus the SUV table inherits from the Cars table and adds some columns - but all other relationships SUV has will still be relational)
    • Re:Some thoughts... (Score:2, Interesting)

      by maxm ( 20632 )
      It could be solved easily enough I think, and I am currently writing a module that I belive would solve most of the problems I have when using Zope.

      All that is needed is a relation "product".

      relations.add([obj1, obj2], [obj3, obj4, obj42])
      >>>[obj3, obj4, obj42]
      >>>[obj1, obj2]

      Every object in zope is defined by its id, and it's path, so it could be done relatively easily.

      Then you would get the advantages of a relational model in the ZODB.

      You could even use a different instance of the class for different object types. Like you make many relation tables in a traditinal rdbm.
    • Because each car also has a body-type (compact, sedan, SUV, truck, van, etc...) - which in a relational database would simple by another lookup table, but in an OODBMS poses data management issues.

      The OO way to answer this is that body-type is a class and compact, sedan, SUV, etc... are instances of it. Each car would have some instance of body-type as a member. I've implemented this sort of thing in a roll-your-own OODB (in Ruby) and in a OODB-on-SQL (in Delphi); in both cases it was painless. The only thing that is remotely tricky is to avoid infinite loops in your low-level serialization code, by doing lazy streaming or by having a serialization flag, or stack, etc. just in case some later person creates a body-type (e.g. batmobile) that somehow refers back to one or more instance of car.

      -- MarkusQ

      • You're missing the idea that multiple levels of classification here... take a look at my example for the detail on deciding about a manufacturer or body-type as the base class.
        • You're missing the idea that multiple levels of classification here... take a look at my example for the detail on deciding about a manufacturer or body-type as the base class.

          You can have as many levels as you wish, and even store things that don't fall into nice "levels." Just use the OODB to store objects and then have members in the objects that refer to other objects (not enumerations) for your classifications. Thus you don't need an instance of SUV for each manufacturer that makes one...because SUV is a (single) object. In the same way, Datsun and Delorean etc. are all objects.

          The class of a particular instance of make (say, Model-T or Bug) wouldn't be a manufacturer or a body-type, it would be the class make. And, like all instances of make, it would have (as members) both a manufacturer and a body-type.

          The problem isn't with using an OODB, but with using an enumeration (or a collection of strings) when what you want is an object.

          -- MarkusQ

  • A Hierarchy of Myth (Score:2, Interesting)

    by droleary ( 47999 )
    While a hierarchy is often used by humans to organize and structure things, that should in no way impact how the data/information/objects are treated as individuals. Look at the common file system hierarchy and it's easy to see that burying files under a hierarchy of directories actually makes access to that information harder. It wasn't so noticeable when we were all just managing a few MB of files, but now people are beginning to store large picture, movie, and sound libraries. File managers have mistakenly stuck with the hierarchy instead of using information associated with the file itself (ID3 tags, etc.) to organize it all. What is really needed is a better approach to representing metadata so that information can be accessed directly based on those metadata attributes and not have it hidden in the hierarchy. I have a short essay on this from the work I've been doing on a Meta Object Manager (MOM), but it needs to be cleaned up before it could be published.

    The desire to impose a hierarchy on the data itself instead of considering a hierarchy as simply one view on the data is a step backwards. Nobody who manages large amounts of data is looking to jam it into a static hierarchy, and so XML is not an answer, nor is any hierarchical representation.
  • persistence layer (Score:2, Informative)

    by budGibson ( 18631 )
    In design, the logical construction of the program and its data structures should be relatively independent of the physical implementation of said.

    Basically, as I read your question, you are using a logical design that is hierarchical (an object structure experessed in XML) and wondering if it would not make more sense to store it in a hierarchical database. Maybe.

    However, relational databases form the current state of the art and have been highly optimized such that any theoretical performance gains from better matching of logical structure to physical lay-out in the database are likely outweighed. More generally, by insisting on a match between logical and physical lay-out, you would potentially be limiting yourself to a specific physical implementation, one that may not provide good performance relative to others.

    A better solution to your problem might be something referred to as a persistence layer. This adds another layer of abstraction to your application, in the form of a mapping, between your logical design and your actual physical mode of storage. There now exist publically available free (as in beer, and in some cases open-source) tools that will automate this mapping. Generally, any performance hit from the abstraction should be made up in the speed of the superior physical implementation, and the freedom to switch later is also important.

    Two that exist for java are castor available from exolab [exolab.org] and a pilot implementation for Sun's emerging Java Data Objects standard (see http://java.sun.com [sun.com] for that tool).
  • First, hierarchical databases weaknesses are not limited to many-to-many relationship modeling. The simple fact is that some data is better represented in a hierarchical fashion (a directory service IS a hierarchical database) and others in a relational fashion. XML is a tool for exposing that data to external sources regardless of its internal representation(s).
  • The enXyme [sourceforge.net] project attempts to map XML data onto a relational database schema. The goal is to allow complex, specific queries of XML data. It's not easy to capture ALL of XML, with all its possibilities, but you can do parts of it. The project already has a basic XML schema parser, a script that takes an XML schema and generates a series of sql CREATE statements that reflect the hierarchy described by the schema.

    I guess a pure XML database like the ones mentioned in the article would be better at this, but the advantage is that relational dbs are already in wide use.
  • by nsample ( 261457 ) <nsample AT stanford DOT edu> on Sunday November 18, 2001 @06:00PM (#2581965) Homepage
    Anyone can explain to me what is suddenly so wrong about relational database with hierarchical indexing?

    Maybe its just me, but the goal today is integration and having a special database for XML and special database for this and that just because its faster for this particular problem creates such a level of complexity, which prevents accomplishing even of the most trivial tasks.

    Forgive me for tooting my own horn on this one, but I believe that (for once on /.) there is a correct answer.

    I summarize the answer in a paper written for VLDB 2001 (www.vldb.org [vldb.org]). The paper presents joint work between Stanford, Berkeley, and RightOrder, Inc. It can be found online here [vldb.org] (in PDF).

    What we found is that relational systems, with appropriate indexes for XML data, give the advantages of both worlds. XML is a hierarchical representation in only the loosest sense. It's written linearly in a flat text document, just as a child learns to write things down on a piece of paper. However, you wouldn't convince anyone but that same child that something written on paper can only represent two-dimensional objects just because the paper itself is flat. XML in many variants is plainly richer in concept than its simple hierarchical representation and thus quite suited to ER. I believe a previous poster mention RDF... a perfect example.

    Punchline: XML is neat, XML is tasty, but XML is not inherently more or less expressive than ER; it just requires a little critical thinking (and index tweaking) to tune ER engines to deal with it. (Once tuned, the ER engines dominate all others in performance.)
  • SGML/XML is an interchange format and this is the only domain it has proven useful over the years. For years people have tried to construct useable SGML and/or XML databases and the results have been consistently disappointing.

    For more complex hierarchical relationships, an object database is more apt, or an XML translation kit for your relational DBs.

  • by Dasein ( 6110 )
    Okay, I've worked for two different network model database companies -- the network database model is just an extension of the network model to allow graph schemas instead of a strict hierarchy. I've also worked with two companies that we mapping hierarchical structures onto relational databases.

    You can think of data structures as (leaving ternary relationships and such aside) some sort of network of relationships. When you think of it this way, relational and network model databases have more similarities than they have differences, especially when you consider that using surrogate keys is the moral equivalent of a network model "pointer".

    Okay so you have this network of relationships, mapping a hierarchical structure onto that is simply picking a starting point and traversing the structure from that "viewpoint" without visiting a node via the same relationship twice (simplified algorithm but...) One of these groups used to think about this like you had a multi-legged turkey. You grab one leg and hold it up. All the other legs hang down -- you grab another leg and a different set of legs hang down.

    So, if you buy that, does it really make sense to represent any sort of network of information in a hierarchical form? Well, yes and no. It makes sense from a presentation and maybe interchange perspective but not from a native storage perspective. It's simply to constrictive and you and up representing relationships that don't fit into a neat hierarchy programmatically in the application code instead of explicitly in the database schema. 25 years from now, someone is trying to reverse engineer your code and figure out how all this data is related -- blech. Ever wonder why IMS application are generally left alone and newer applications are not usually written to IMS. This is part of the reason why. (yes there are some but they are the exception).

    Throw in to this my experience working with a bank that had hierarchical data and the extent to which they went to circumvent that restriction, and I'd say that native hierarchical storage for XML is a bad idea. Granted it's tempting but it seems ill advised since it's very likely that your data will survive long beyond the lifecycle of the system used to originally store it.

    The original question didn't provoke this but I've seen a couple of responses about using XML as a native data storage format. Let me say that, unless the data is very static, it's a monumentally stupid idea to do that. XML is not a replacement for a database.

    I find that most of the people who really want to do this are ignorant of all the work that goes into real database systems. They don't understand lock management, transactions, rollback and recovery, free space management nor the scalability issue that real databases take care of under the covers. If you feel tempted read this [amazon.com]

    You throw this plus the representation of non-hierarchical relationship with IDs and sooner or later you will find yourself in a text editor tracking down ID/IDREF pairs to find out where your data is corrupted. Or writing scripts to validate your "entire data set" -- above a few megabytes it can be really painful.

    For God's sake, expect to use XML to store data that you are going to update with any regularity.
  • by dgroskind ( 198819 ) on Sunday November 18, 2001 @07:00PM (#2582169)

    XML may be hierarchical but the data it is used to markup is not necessarily hierarchical. For instance, XML can be used to markup conventional fielded (flat file) data to serve as an interchange format.

    More importantly, XML is used to impose some structure on inherently unstructured text. The structure it provides is based on some assumptions of how the data will be used or how it will be presented. If the data is used in some otherway, the markup can be useless.

    An example is a book. For XML purposes, it can be described as structured by chapter, section, subsection, and paragraph. For information purposes, tags are assigned to represent the ideas, terminology, names and other index-like content. There is virtually no structure in these index type of tags but they convey the most important information in the book.

    Or not. These tags are assigned based on assumptions about what readers are interested in. A different set of assumptions would produce a different set of tags even thought the structure of the document would stay the same. If the sentences and paragraphs are shuffled and exerpted for some other publication, even the structure becomes irrelevant.

    How this inherently unstructured information is stored is relevant to how it is managed, that is, how it is backed up, how access is controled, how changes are tracked. However, when it comes to putting the information to some useful purpose, it is the retrieval mechanisms that are important. The issues here are how easily the user can specify the type of information he wants and how accurately the mechanism can find it. This process is usually independent of the underlying structure and uses some higher level concepts of relevance and context.

    The question of whether to use a hierarchical, relational or object-oriented data structures misses the point for textual data, for which XML is commonly used, because none of these structures capture meaning.

    Topic maps [topicmaps.org] make a heroic stab at capturing meaning in XML markup but still only within a set of assumption. I suspect a true meaning markup language is theoretically impossible, or at least theoretically very far in the future.

  • LDAP is a *protocol* (Score:2, Informative)

    by bigbird ( 40392 )
    Contrary to some of the comments I've read here, LDAP isn't an implementation of a database, it is a *protocol* for accessing directories. LDAP data could be stored in anything - a hierarchical database, a relational database, an object database or a flat file. Let's not confuse the issue under discussion.
  • by wytcld ( 179112 ) on Sunday November 18, 2001 @08:29PM (#2582484) Homepage
    99+% of all corporate data that isn't in a flat-file or (possibly three-dimensional) spreadsheat is in relational tables. The typical task that XML has been designed for is to standardize data exchanges between differently-structured relational systems, by providing sets of tags specific to the standards of specific industries. The whole point of XML is to enable companies to continue to use their current investment in relational databases, without the drag of having to do custom data conversions when dealing with suppliers or distant divisions in the company.

    If you're going to throw out the installed investment in relational databases, you might as well just design a common database standard per industry (rather than an XML data exchange standard) and let them all exchange native data rather than translating in and out of any exchange format. Obviously that won't happen.

    Now, if you're a new firm, you might decide it's easier to go OO or heirarchical or keep your data in slips of paper in a shoe box. But most of the available tools and solutions will continue to respect that relational works real, real well for inventory, manufacturing, accounts ... just about everything industry consists of. So if there's an impedence mismatch between relational and XML that's enough to make trouble, it's XML that should be replaced by another model.

    What design changes would be required to produce XML's relational equivalent?
  • See RDF (Score:2, Interesting)

    by shirro ( 17185 )

    I don't think XML by itself carries enough metadata to understand much beyond whether a document is valid or not. I think RDF and RDFS have a big role to play in getting XML database ready.

    Perhaps hopping on the XML database bandwagon before RDF technologies mature could be a mistake. Forget the semantic web, I want to see the sematic database.

    W3 RDF [w3.org]

    A Good RDF resource [bris.ac.uk]

  • Native XML Databases (Score:4, Informative)

    by idomeneo ( 196902 ) on Sunday November 18, 2001 @09:54PM (#2582709)
    I recently wrote an introduction to native XML databases [xml.com] article for xml.com. My main point there and it applies to this discussion too, is that native XML databases are a tool like any other. For some jobs they're right and for some they're not. I've been working on the technology in the form of dbXML [dbxml.org] for about a year and a half and in some cases it's great and in others it really stinks. It's all about the right tool for the job.

    It's easy to dismiss a new database technology as irrelevant because of the dominance of the RDBMS, but you should really learn more about it and when it is appropriate and when it's not. It's not going to replace relational, and isn't intended to. Here's a few links where you can learn more beyond what's available on Ronald Bourret's site mentioned in the original post.

    The XML:DB Initiative [xmldb.org]
    The dbXML Project (open source native XML database) [dbxml.org] Soon to become an Apache XML project named Xindice
    eXist (another open source native XML database) [sourceforge.net]

    My blog on the subject. [xmldatabases.org]
    Kimbro Staken

  • by pegacat ( 89763 ) on Sunday November 18, 2001 @10:04PM (#2582735) Homepage

    A bit surprised to hear that 'Hierarchical databases were blown away by relational versions' - since I'm pretty sure they've been paying my pay check for the last three years... :-)

    There are a large number of heirarchical databases out there. The big fellas are the X500 directories (X509 certs came out of this work). More common are X500's demented kid sisters, the LDAP directories ( rfc2251 [faqs.org]). The DNS system also fits the description 'heirarchical database'.

    As far as XML goes, there are people storing XML in directories - although they're still fussing about exactly how to do it. There are a bunch of people trying to come up with standards - check the directory services markup language people www.dsml.org [dsml.org].

    There are people trying to sell XML enable directories - Novell sells an XML directory, but most directories can be used to store XML (including our 'eTrust Directory').

    As a final quicky - when do you use a directory over an RDBMS? Directories are good for naturally heirarchical data with few cross connections. They are usually optimised for slow writes/fast reads. They are *very* good for distributed data (e.g. DNS, international organisations etc.). The X500 spec defines a very fine grained security model, which can also be useful. However, if your data is closely cross-linked with lots of relationships... well, use an RDBMS!

  • The question of whether hierarchical or relational databases are better, more flexible, dependable and so on has already been answered decades ago. I fail to see how a popular hierarchical data transfer protocol re-opens the question. If XML is going to be pushed far beyond what it was designed for then it needs to become far less hierarchical rather than having the rest of the world, and especially mainstream DBMS, re-arranging itself according to XML's limitations.
  • Hierarchies are often used to represent a view of data that is appropriate for a given purpose, but different scenarios will often need different views of the same data, so it's not a good idea to lock the data into one hierarchy or another.

    Today you might want to store your history of employee-department assignments by nesting employees under departments, but at some point you may also want to nest work histories under employees.

  • I have been toying for a while now with how to represent data efficiently in a manner in which it can be efficiently walked with an XML document-object-model, if not in XML itself.

    My core concern is actually building the indexing tools to handle the matching of insane numbers of actual Xlinks (or a mildly simplified form thereof), to make a dataset that is distributed and provided by multiple subsystems hold together.

    My most recent iteration used a simplified form of Xlink to join data supplied by code that implemented a specialized DOM model (code that understood how to browse a given object model, for reflection purposes), with data taken from a flat text XML document (browsed with a different implementation of that model).

    The real concern I have with this sort of system is the time it takes to register a document against all of the current X-links and deriving an efficient system for parent-child connection when multiple parents can exist (transparent links from other nodes). Registering a new node or worse, changing data in a node is an expensive operation in this model.

    My need for extensive X-link support is to be able to provide a sort of on-the-fly XSL translation of one branch of data into another. The links would then connect to the translated data rather than the original.

    A secured view of the data could be provided by handing a link to a DOM node that was walking the XSL view, rather than the original data. User security becomes a XSL-like document.

    My filesystem becomes a branch of the tree. It is admittedly an awkward security mechanism to do XSL-style matching against the tree, but it does not have to be done in XSL itself, another utility which walked a base dataset and returned a filtered view in the same DOM model provides the same security mechanics.

    There are some issues with the current mechanisms for conveying XML information (DTDs do not localize well, etc).

    Method invocations are shaky at best and rely on sharing handles of some sort (SOAP, CORBA, etc). I transmit data either by copying a branch of the tree, or handing off a handle to a CORBA or SOAP object that can walk the DOM on the local system.
    Either way looks the same to the client.

    A hierarchal model has a certain level of appeal because of the simplification of the new-branch registration process, but severely limits the effectiveness of the tree processing tools you can build. OTOH, the relational model could be reconstructed with some stylesheet tricks.

    My present use is in an operating system project as the mechanism for accessing system resources, the 'file system', user data, etc.

    There is appeal when it comes to generating a backwards compatible view of the data, because you can provide a translator which takes the current data and translates it back to a view compatible with what a given application expects, etc. Method invocations through a function call-translator can allow for constrained arguments to methods, etc.

    The transparent linking model also has appeal for simplified remote method invocation, a filename is just an Xlink, etc.

    Active Directory, eat your heart out. ;)

I was playing poker the other night... with Tarot cards. I got a full house and 4 people died. -- Steven Wright