Notes on Packing TEI files
 
C. M. Sperberg-McQueen, 5 Nov 92
a propos of the problem of packing TAG files
 
1 What We Need
 
We need a poor-folk's version of SDIF.  The TEIPack Packer should:
    - parse the document, recognizing entity declarations and entity
      references (and expanding all entity references, both general
      and parameter)
    - note the system DTD file
    - note which entities are external entities
    - note which external entities are referred to in the course of
      the document
    - pack all the system entities needed for the document into a
      single data stream
 
Packing can occur in the same pass as parsing, or in a second pass,
possibly with a separate program.
 
In Phase 2, we can add function to the same processor or a separate
filter to read WSDs for the local character set and transmission
character set, recognize characters not in the transmission character
set and pack them using the appropriate entity references.  Such a
character-set packer need not be used just for interchange; it could be
useful to perform such a transformation on local files.
 
The TEIUnPack unpacker should:
    - read the packed file and unpack each distinct system entity,
      where possible giving it the same name and directory location
      as the source
 
In Phase 2, the unpacker should read the WSDs for the local and
transmission character sets and expand entity references appropriately.
Like the character-set packer, the character-set unpacker might be
useful locally, not just in connection with interchange.
 
The central problems are:  delimiting the original system entities in a
foolproof way, and recording relevant information about the system
entities (such as their names and locations) so they can be unpacked
appropriately.
 
2 Proposal for Getting It
 
2.1 The Packer
 
For the Packer, we use a two-pass operation.  In the first pass, we
create a file called a packing-list file, which looks like this:
 
  <!--TEIPack Document Begins-->
  <!--TEIPack Entity: "*SGMLDOC*" -->
  <!--TEIPack Definition:  P2PRDRIV.ODD -->
  #include "P2PRDRIV.ODD"
  <!--TEIPack &#EE; -->
  <!--TEIPack Entity: "*DTDENT*" -->
  <!--TEIPack Definition:  system "c:\tei\odd\tiny.dtd" -->
  #include "c:\tei\odd\tiny.dtd"
  <!--TEIPack &#EE; -->
  <!--TEIPack Entity: "p2.co"-->
  <!--TEIPack Definition:  system "p2co.odd" -->
  #include "p2co.odd"
  <!--TEIPack &#EE; -->
  etc. ...
  <!--TEIPack Document Ends-->
 
etc.  Each entity is preceded by two comments giving its entity name and
its "entity text", and followed by a comment indicating an entity end.
The file being processed has no explicit entity name and neither does
the base DTD; they get the names '*SGMLDOC' and '*DTDENT' as in ArcSGML.
Everything else gets the name in the entity declaration.  In this
packing list, the entity itself is represented by an #include line
indicating the file name to be included in the packed file; the second
pass uses the existing SPP.SPT program to read the packing list and
produce the packed file.
 
As long as no input file contains the string '<!--TEIPack' at the
beginning of a line, no special measures need to be taken to distinguish
the meta-comments indicating entity boundaries from normal comments in
the files themselves.
 
This can be done manually, in a pinch.
 
2.2 Handling Dirty Tricks in the SGML Comments
 
If one wishes not to constrain the contents of one's comments, a
separate pre-processor can read all the files mechanically (without
any SGML knowledge) and change '<!--TEIPack' in column 1 to
'<!--TEIPack Literal'.  The unpacker can then reliably act upon every
line beginning '<!--TEIPack', using the keyword following 'TEIPack'.
The keywords are thus:
 
    Document   (signals start and end, for error checking)
    Entity     (signals beginning of an entity)
    Definition (gives the formal declaration of the entity)
    &#EE;      (signals end of the entity)
    Literal    (signals a content line:  the rest of the line should
               be copied to output, preceded by '<!TEIPack')
 
2.3 The Unpacker
 
For the Unpacker, I propose the following behavior, for a first version:
 
  read the packed document
  when it sees a TEIPack Entity line, it reads the following
     TEIPack Definition line, then prompts the user:
 
     prog:  Unpacking entity fufoo, defined as system 'c:\tei\bufar.doc'
            Where do you want this, o my lord and master?
            (Default:  /tei/bufar.doc)
     user:  tei/fubar/bufar.doc
     prog:  Done.
 
  when it sees a TEIPack &#Ee line, it closes the file (and says 'Done.'
     as shown above)
  when it sees a TEIPack Literal line, it changes the string
     '<!--TEIPack Literal' to '<!--TEIPack'
 
For the first cut at this, I propose whoever implements the unpacker
simply do the best job they can about putting stuff in the correct
directories, etc.  A later refinement could put information on
default directories, source operating system, etc. between the
Document Begins line and the first entity.
 
Like packing, this could also be done manually.
 
3 Summary
 
We need TEIPack and TEIUnpack programs; they would make it much
simpler to exchange chapters with their accompanying TAG files.
As separate projects, character-set packers and unpackers could be
written, to operate either upon uninterpreted data streams or more
usefully upon interpreted SGML documents.