MI W 03Practical Guide to Migration of TEI Documents from SGML to XML

Instance conversion workflow

There are four distinct steps for migrating document instances from SGML to XML. First, of course, the documents need to be converted. You will then probably need to normalize the case in your tags (i.e., be sure that the tags use proper capitalization), since XML is case-sensitive and your SGML may not be. You may also wish to format the files to make them easy to read. Finally, we strongly suggest that you develop procedures for checking your results for any unexpected bugs that may have been introduced during the migration process. These steps are discussed below.

Processing environment and DTDs

The processing environment refers to the software that you use to manipulate your documents. It includes editing tools, parsers, transformation engines, stylesheets, and catalogs. You and your staff should carefully consider what kind of tools will best suit your immediate migration needs as well as your project's long-term development. Tools are discussed below.

osx

Post-processing

The osx tool will not preserve non-significant whitespace (e.g., space between elements) in its output. If you want your converted files to have neatly wrapped lines and indented elements for human readability, you should use another tool to reformat osx's output. Many XML editing tools will do this.

Sebastian Rahtz's tei2tei.xsl stylesheet will provide some other post-processing (case normalization and removal of attributes which have default values) ³ . It will indent the output nicely, but it will also resolve entity references, which may not be a desired behavior. You could use xmllint to escape entity references first and then tei2tei.xsl to create output with entity references in their original state. Some projects may wish to develop their own stylesheets for post-processing. Note that not all systems will be able to apply an XSLT stylesheet to very large input documents, due to memory limitations.

Other tools

General remarks

SDATA entities are "special entity references" which were available in SGML but do not exist in XML. This section will give some simple recommendations for handling them in the migration from SGML to XML.

In the SGML world, SDATA entities have been used mainly to provide a handle to characters that were not available in the coded character set used by a document instance. Accordingly, this use will be discussed in some detail and other cases will be briefly mentioned.

Public entity reference sets

In XML the document character set is specified as Unicode, which currently includes almost 100,000 characters. John Cowan prepared a list of SGML public entity mappings to Unicode, which is available via ftp at the Unicode website and from OASIS. For entity references in this list the conversion process is straightforward, but other cases may require more tweaking.

Handling SDATA entities that do not represent characters

Besides being used to represent characters, SDATA entities have seen a variety of other, less frequent usages. Due to the variety of uses, which range from recording specific information during data capture or data conversion, to highly technical application-specific information, no general solution can be outlined here. In many cases, however, the conversion to general XML internal entities (which osx will do for you if --xml-output-option=no-expand-internal is specified) should be good enough.

General remarks

This section is for projects that have modified the TEI DTD and want to migrate these modifications from SGML to XML (i.e., want to use the XML-based P4 DTD with equivalent modifications). We begin with some general remarks, then describe a sample DTD modification that covers the most important issues, then outline a recommended migration procedure and demonstrate the key steps using the example.

If the elements or content models that the TEI provide don't quite meet the requirements of your project, there is an official escape route: you can modify the DTD in a number of well-defined ways and your documents will remain ‘TEI conformant.’ This involves creating two extension files, setting some parameter entities, possibly defining new elements or redefining existing ones, and making these modifications known to the parser in the DTD subset at the beginning of the document.

Although the process is a lot simpler than it looks at first glance, many people have taken unofficial escape routes, especially the users of the TEI Lite DTD, who would have been required to first switch to a full TEI DTD before applying local extensions. It is admittedly simpler to just open your local copy of teilite.dtd and change a few lines. Only later will you find out why the TEI Guidelines strongly discourage this, and one of those moments could be the migration of your customized DTD to XML.

The following is a short list of some critical issues involved. In the following subsections, we will work through a fictitious example that covers most of these issues;

1. Element and attribute name case is significant in XML.
2. It is likely that some of the modifications in your existing P3 extension files involved copying (and then probably modifying) pieces of the TEI DTD files. You should check whether those DTD pieces have changed from P3 to P4.
3. Some people have made modifications to work around problems in the TEI P3 DTD; if they are fixed in P4, the workaround could cause errors (a notorious example is <persName> ).
4. The SGML DTD syntax for element declarations requires two characters of ‘-’ or ‘O’ that indicate whether start and end tags are required or can be omitted. These indicators don't exist anymore in XML DTDs and your private DTD snippets need to be modified.
5. The content model for XML elements is more restricted than for SGML elements. We won't go into fine detail, but the following two points deserve attention:
   • The only type of character data is PCDATA. You cannot define CDATA content to bypass the parser.
   • The inclusion exception and exclusion exception syntax does not exist in XML DTDs. In SGML, you could declare an element to be legal everywhere within element <X> and its children in a single line by using the inclusion exception syntax. This is not possible in XML. Instead, you have to add <X> to all content models individually.

A tutorial example

In this section we will do some simple TEI DTD modifications in SGML. This will then serve as a tutorial example for the migration to XML. While working on this example, the main problems in converting DTD extensions should be covered. Not everyone will need to take all of the steps treated here, and some needs might not be covered, but this should be an easy, hands-on starting point for most projects. ⁴

This example shall be migrated to TEI P4 XML in the next two subsections.

Suggested migration procedure

Although the following step-by-step list may sound over-cautious, this approach is recommended to help those new to the process while converting your DTD and documents. You are switching from P3 to P4, from SGML to XML DTDs, from SGML to XML documents, and from SGML to XML parsers at the same time, and it can be difficult to find your way through the many potential pitfalls.

1. Pick an interesting test document from your repository and make sure you can parse it as it is (in SGML form) against your current DTD setup (TEI P3 with your extensions), perhaps making up an example as we did above.
2. Set up a parallel SGML parser environment to parse against the TEI P4 DTD. Before you try to parse your sample file, make sure the parser works with very simple standard TEI files.
3. Now try to parse your sample document against P4 (in SGML mode). Theoretically, this shouldn't be a problem, but you might encounter errors. Create an intermediate version of your SGML extension files to fix any problems (the cause is most likely that your extensions work around a bug of the P3 DTD that was repaired in P4).
4. Create new XML extension files based on the SGML ones. We will do this for our example in the next section. If you want to continue supporting SGML documents, consider making the extension files ‘dual-use’ (i.e., compatible with XML and SGML, as described below) so you only need to maintain one set. If you had to make changes for the previous step (moving from P3 to P4 SGML), you have to decide whether the SGML part of your dual-use setup will be compatible with P3 or P4. Normally, you should be fine using P4 SGML, so this is the assumption in the remaining text.
5. If you have created dual-use extension files, use them to parse your SGML document against P4 in SGML mode. Don't proceed until they are correct.
6. Make sure that your XML parser is set up properly by parsing a minimal TEI XML document without extensions.
7. Convert the test document to XML. A short fictional sample document will make this step easier and will avoid any extra confusion from errors arising in a large-scale text migration.
8. Try to parse your converted test XML document. Errors at this stage may be a result of the document conversion or the migrated DTD extensions. Fix them all.
9. If you have dual-use extensions, go back and try the SGML parse again.

Migrating the example DTD

We will now focus on rewriting the DTD extensions in XML, with the sample DTD modifications described earlier as a base. We will be creating files called my_xml.ent and my_xml.dtd.

Before we start, however, a strategic decision has to be made: Shall we burn bridges and support only XML in the future? The P4 DTD provides mechanisms to parse both XML and SGML and we can do the same for our customized DTD, if we need to support SGML in parallel for the time being. It takes a little more thought and effort, but in return you get the comfort of a safe transition period. The parameter entity TEI.XML gives you an important tool for accomplishing this: it is defined as INCLUDE in XML mode and as IGNORE otherwise, so that you can segregate XML and SGML entity and element definitions. DTD extensions that use this technique are called ‘dual-use’ extensions in this document. Our example demonstrates both dual-use and pure XML extensions. ⁵

One obvious syntactic difference between SGML and XML DTDs is the ‘omitted tag minimization parameters’ that appear as ‘-’ and ‘O’ in SGML element declarations to indicate whether start and end tags need to be present or not. They are superfluous in XML, where minimization is not allowed, and therefore are not used. The TEI P4 DTD provides and uses parameter entities om.RO and om.RR as a substitute. For SGML parsing, these entities expand to - - and - O respectively, and for XML parsing they expand to nothing. Elements that require only a start tag (mostly empty elements) use om.RO, and elements that require start and end tags use om.RR (non-empty elements should be defined that way). We can make use of this mechanism for our dual-use DTD extensions. ⁶

So let's go to work:

Now let's move towards XML: if we first check for consistent case of the element and attribute names in our DTD, we discover that element <ps> was once written in uppercase and once in lowercase. We decide that lowercase shall be the correct spelling.

Some things are easy: the renaming of <div1> and <div2> to <volume> and <letter> , and the declaration of the <ps> element can remain untouched, except that the ‘- -’ in the definition of <ps> needs to be either removed or (if we aim at a dual-use DTD) replaced by ‘%om.RR;’.

The dual-use decision comes up again with the <pb> tag. In XML, we can just write an ATTLIST containing only ‘imageUrl’ and it will be merged with the existing ATTLIST in the TEI DTD files; there is no need to suppress and copy the definition of <pb> . For continuing support of SGML, we have to suppress and redefine the element as before. We find it in the TEI DTD files (teicore2.dtd), copy the P4 definition and modify it, adding our extra attribute.

The most difficult problem is the <toDo> tag. For one, the content model CDATA needs to become PCDATA. This means that existing documents will most probably break, but there is no choice. It might be a solution to turn all the <toDo> content into CDATA marked sections with an automated search and replace as part of the document conversion, or to escape the contained markup to entity references using a similar procedure.

Also, the simple way of allowing <toDo> everywhere is no longer possible. This could be a good occasion to check how that element is actually used in practice and where it is really needed. A compromise in our example could be to add it to the class ‘Incl’ that is part of every content model within <text> . Sometimes, a more complex redefinition of content models could be necessary. If that is your situation, you may want to consult the unofficial TEI document ED W 69, chapter 8 for in-depth coverage.

The first runs with the XML parser result in many warnings because of redefined parameter entities; this is normal. Some syntax correction is required where XML is more strict than SGML: we forgot a semicolon in a parameter entity reference, ‘%paraContent’ must not be in parentheses while the ‘#PCDATA’ for <toDo> has to be. When you don't know how to get rid of an error, it can be useful to browse the TEI DTD files and compare with your own usage.

The cross-check of the dual-use version with the SGML parser exposes a little additional problem: the document now uses the character entities &lt; and &gt; which are predefined in XML, but not in SGML; once discovered this is easily fixed. In the example file, you will find a solution that looks a little complicated but works flawlessly with SGML and XML.