Conference Agenda

This presentation reports on the methodology and challenges of building digital dictionary data for the Hatoma-Japanese Dictionary, a comprehensive linguistic resource for the Hatoma language, one of the endangered languages spoken in the Yaeyama islands, Okinawa Prefecture, Japan. The dictionary consists of over 16,000 entries and provides thorough documentation of the language and culture of Hatoma Island. Recently, audio databases of its headwords and examples have been published online.

The Hatoma Yaeyaman language is classified as a South Ryukyuan language; all languages in this group are endangered, and proficient speakers are mostly confined to the elderly. Amidst emerging efforts to revitalize regional languages across the Yaeyama Islands, a digital dictionary featuring bilingual lookup, faceted search, and audio resources is essential for younger learners. To this end, we are compiling XML-encoded dictionary data based on TEI Lex-0. Developed by the EU-DARIAH Working Group on Lexical Resources, TEI Lex-0 is a forked schema of TEI that aims to provide a clearer and more unambiguous rule set for lexicographers.

Building upon the previous work of Nakagawa and Miyagawa (2023), we successfully translated most of the existing structures from the original book to the TEI Lex-0 framework. We encountered challenges in three primary areas and propose solutions for them:

• Cultural Content: The dictionary focuses on songs—a cultural pillar of the language—necessitating new ways to accommodate lyrics and metadata in the absence of specific elements for verses.

• Data Canonicalization: The original data contains unstructured definition texts with numerous free markups, which complicates the process of canonicalization for accurate structure detection.

• Schema Limitations: Currently, TEI Lex-0 does not support <media> or <ruby> elements. Furthermore, minor attribute values require further codification to allow for more precise descriptions within East Asian linguistic environments.

There is an increasing need for methods to encode algorithmic or programmatic content in digital humanities projects. For example, a comprehensive digital edition of an interactive novel must include some means to express the work's state transition logic. The predominant solution to this problem is to simply embed code written in a conventional programming language, such as JavaScript or Python, in a TEI document. However, such languages are unsuited to digital preservation because programs written in them require huge, brittle webs of transitive dependencies. Will code which relies on particular versions of dozens of Python packages, a particular version of Python itself, and a particular operating system still be executable in 5 years, yet alone 100?

In computer science, it is common practice to express algorithms in pseudocode. However, pseudocode, as the name suggests, is not actually code and has no precise computational meaning. The programming language Algol (in its various incarnations) was in fact first specified as a standard notation, with a formal semantics, for describing algorithms. But as Algol fell into disuse, so did the notion of a standard, executable notation for algorithms.

In this paper I introduce AlgolML, a new, XML-based executable notation for algorithms and computer programs, designed to function as a companion technology to TEI. Unlike nearly all programming languages, the AlgolML standard includes a formal specification of its semantics. As a programming language, AlgolML is conservative in its design, but includes unique features related to digital preservation and draws inspiration from the related notion of Permacomputing. Crucially, AlgolML is easily implementable: a conforming implementation should run no more than a few thousand lines of code in any common programming language. This property of AlgolML ensures that algorithms and programs so encoded will continue to be interpretable as long as the specification itself persists.

Mapping Materiality: Formalising structural representation for the <objectContents> element

In 2019, after having created elements like <object> and <objectIdentifier>, I added a provisional note to the TEI Guidelines chapter on Names, Dates, People, and Places to signal ongoing generalisation of the manuscript-centric elements in the context of the <object> element:

“Please note: The object element is a recent addition to TEI P5 Guidelines as of version 3.5.0 and as such might be more prone to further revision in the next few releases as its use develops. This may be particularly evident where its contents have been borrowed from msDesc and have yet to be generalized from their use in the context of manuscript descriptions.”(https://tei-c.org/Vault/P5/4.11.0/doc/tei-p5-doc/en/html/ND.html#NDOBJ)

While some elements later had additional tweaks that mostly involved changing their description slightly to allow for their use in describing objects rather than manuscripts and early-printed books, by and large no further modifications were made.

One remaining target for modification inside the <object> element was its wholesale inclusion of the <msContents> element. This element gives encoders a way to describe “the intellectual content of a manuscript, manuscript part, or other object either as a series of paragraphs or as a series of structured manuscript items”.(https://tei-c.org/Vault/P5/4.11.0/doc/tei-p5-doc/en/html/ref-msContents.html) As mentioned, the ‘or other object’ part of this and other element descriptions was added in the process of creating the <object> element and integrating it with the existing TEI framework. Nevertheless, there was a desire to duplicate the functionality of the <msContents> and modify it for the more general case of describing objects.

This short paper will discuss the creation of a more generalised <objectContents> element and whether the corresponding sub-structures of <msItem> and <msItemStruct> truly need to also be replaced in an object-agnostic form of description.

One co-author has argued¹ both that our encoding strategies must balance our desire for expressivity on the one hand against our desire for conformance and understandability on the other, and that while the TEI system deliberately makes it easy for documents to be interchangeable, it has made it all but impossible for generic TEI files to be interoperable.

One particularly important type of interoperability to consider, if not have, is the ability to write software that is able to do something useful (e.g., produce passable HTML output) from any valid input TEI file, i.e. from a generic TEI file rather than one that meets the criteria established by a particular customization.

While we do not think favoring interchangeability over interoperability is a bad thing (as we believe that, in general, the more we reach for interoperability the more we squelch expressivity and thus impair individual researchers), we do believe these decisions should be made deliberately, and that there may be cases in which TEI eschews interoperability for little or no benefit, in that particular TEI encodings make it harder to write software that processes generic TEI files.

In this presentation we will provide specific examples where the design of the TEI system makes writing generic software harder, some for little or no gain. We will further attempt to provide our thoughts, where applicable, on how a new TEI project might sidestep some of these issues; and for others suggest that TEI Council may wish to consider modifications to P5, or revamp how these issues are handled in P6.

Note

¹ See https://balisage.net/Proceedings/vol7/html/Bauman01/BalisageVol7-Bauman01.html.

We are all familiar with the date format used on @when: YYYY-MM-DD. It originated in ISO 2014 (published 1976-04) which was superseded by ISO 8601 (published 1988-06).¹
P5 provides three sets of attributes for recording a formal representation of a temporal expression: 1) @when and friends (@notBefore, @notAfter, @from, & @to from att.datable.w3c), 2) @when-iso and friends (@when-iso, @notBefore-iso, @notAfter-iso, @from-iso, & @to-iso from att.datable.iso), and 3) @when-custom and friends. The first set allows users to record a limited set of temporal expressions using the formats supported by the W3C, which are a subset of ISO 8601:2004. While not the main focus of this presentation, some time will be spent discussing their advantages & disadvantages. (The main disadvantage is that these formats are quite limiting, e.g., there is no way to express a time precise only to the hour or minute — all times must be precise to the second (or decimal fraction thereof). The enormous advantage of these formats is they are already supported by various computer systems, XSLT and XQuery processors in particular.)
The second set allows users to record a somewhat less limited set of temporal expressions using the formats supported by ISO 8601:2004. Some of these formats are unavailable via att.datable.w3c, but may still be useful; e.g. imprecise times, decimal fractions of less precise units, and recurring time intervals. However, these values cannot be handled easily by XPath; in fact, this author is not aware of any out-of-the-box software that will know how to handle these values.
The third set allows users to record temporal expressions however they like, and is mentioned here only for completeness.
Since P5 was published ISO 8601 has gotten a major overhaul, published in two parts in 2019. ISO 8601:2019, apparently based heavily on the US Library of Congress “Extended Date/Time Format”,² allows formal representation of quite a few types of temporal expression that TEI encoders might find quite useful. E.g., expression of uncertainty or approximation (or both), disjunction, and conjunction. But again, there is little to no software support for these formats.
This presentation will

• present some of the capabilities of ISO 8601:2019;
• point users to a profile of ISO 8601:2019 with detailed examples intended for a DH audience;³
• argue that it is high time for TEI to update @when-iso (and friends) such that they are based on 8601:2019 rather than 8601:2004;
• suggest a method for allowing users to say “no, I will stick to ISO 8601:2004” in their customization files; and
• (if sufficient time remaining) demonstrate code for validation of attribute values that (are supposed to) conform to the profile that was presented.

Notes
¹ P1 referred only to ISO 2014; P2 refers to both of them a bit schizophrenically; P3 & P4 still have ISO 2014 in the bibliography, but do not actually refer to it; P5 does not mention ISO 2014.
² https://www.loc.gov/standards/datetime/
³ Yes, a bit of self-promotion, here: https://github.com/NEU-DSG/wwp-public-code-share/tree/main/docs/dates_and_times_in_DH