Three of us – Kyle Johnson, Clément Besnier, and Todd Cook – presented to the Digital Classicist Berlin program at the Berlin–Brandenburgischen Akademie der Wissenschaften.
Last month, the annual ACL conference published the CLTK’s de facto “white paper” (“The Classical Language Toolkit: An NLP Framework for Pre-Modern Languages”).
Some time prior to this, we also officially promoted our version 1.0 to “production”.
]]>The core maintainers are pleased to announce the first pre-release of the CLTK’s version 1.0. More information will follow, but at the highest level the guiding principles have been (a) to add a single pre-configured interface and (b) to fully rationalize software’s architecture for adding new languages with a minimum of friction.
Preferably in a new virtual environment, in either Python 3.7 or 3.8, pull the latest “alpha” with:
$ pip install –pre cltk
The docs should be enough to begin stress-testing the new code:
For now, please open issues in this new development repo. In the coming weeks, we will merge the two Git trees together.
]]>The CLTK has as a central goal to provide complete NLP coverage of all pre-modern languages. In practice, this ambitious goal needs to be tempered by availability of language resources, digital and human. With some frequency, especially around the time of Google Summer of Code application, we are approached by potential contributors who hope to pitch in by adding NLP tools for a given language. Over the past six years, those of us centrally involved in the project have learned a great deal about what characteristics distinguish a successful from unsuccessful project. I’ll describe these characteristics in some detail below, but to summarize, a successful “add-a-language” project for the CLTK depends the presence already-available, free digitized data.
Languages which lack such digitized data, I’ll call “under-resourced.” To help explain, allow me to borrow a definition of a “language resource”:1
The term Language Resource refers to a set of speech or language data and descriptions in machine readable form, used for building, improving or evaluating natural language … for language studies, electronic publishing, international transactions, subject-area specialists and end users. Examples of Language Resources are written and spoken corpora, computational lexica, terminology databases, speech collection, etc.
For the CLTK, language resources normally take the form of static data sets, such as text corpora (annotated or plaintext), treebanks, and lexica (dictionaries of various sort). Resources also take the form of algorithms, which are usually rules-based and optionally rely upon data sets. For an example of the former sort of algorithm, see Luke Hollis’s stemmer for Latin, which is itself a Python-language implementation of rules defined by previous peer-reviewed scholarship (Schinke et al., 1996). For an example of rules-plus-data, Todd Cook’s Latin syllabifier first defines a variety of character types in ScansionConstants.py and then uses these in Syllabifier.syllabify(). Examples of well-resourced languages, in both data and algorithm, include Ancient Greek and Latin, for which not coincidentally the CLTK has excellent coverage (look for Greek or Latin data sets on GitHub, to get an idea).
The CLTK is an NLP project, and it eschews ventures outside of this mission. Simply put, this means that the CLTK is neither a data annotation nor a (non-technical) user-facing project. In reference to the classic three-tier software architecture, the CLTK is exclusively restricted to the middle presentation tier, relying upon the data storage tier as upstream dependencies and adopted downstream by the presentation tier. The CLTK has a vested interest in the health of data storage and presentation tiers, however our project’s core NLP task, to write NLP algorithms by leveraging already available data, is formidable enough. To satisfy the needs of downstream makers of applications, we write what we hope are sensible and well documented APIs.
To illustrate briefly the challenges of even relatively simple data creation, I am reminded of several students whose natives tongues were, respectively, Telugu and Kannada. They approached with proposals to do NLP in their languages, and when I pointed out they’d need data, each came up with the idea of doing OCR to obtain plaintext corpora. After all, plenty of digitized book images could be found online. However, preliminary experiments demonstrated that OCR for these particular non-Latin characters was of very low accuracy. Simply making an OCR model would have constituted a summer project itself (one outside our bounds, nevertheless). The creation of annotated texts is a laborious in the extreme, requiring equal degrees of passion and technical expertise.
Having explained what a language resources are and why they are so important to the CLTK, I will next explain the states in which under-resourced (dead) languages exist and how one can decide whether a given language is under-resourced. Languages generally fail to meet the minimum bar of “resourced” due to one or more of the following attributes:
First, simply enough, if data has not been digitized, then it is not available for computational processing. As mentioned above, book page images are quite a distance from even plaintext files.
Second, non-free data poses a significant problem. My use of “free” here corresponds generally to that expounded by Richard Stallman as something more particular than no-cost and open source:
When we call software “free,” we mean that it respects the users’ essential freedoms: the freedom to run it, to study and change it, and to redistribute copies with or without changes. This is a matter of freedom, not price, so think of “free speech,” not “free beer.” (“Why Open Source misses the point of Free Software”)
Intended to support rigorous quantitative scholarship, users of the CLTK simply must have the ability to manipulate and redistribute data used to create results. Contrary to the understanding of many humanists I have met, “open access” and “open source” resources that have proprietary licenses are unstable foundations for projects like ours, the legacy of which (we hope) will be measured in decades, not years. It is undeniably practical for a scholar to quickly publish an article which uses proprietary data or software, however his results will likely not be reproducible, frozen in time, and lost to history.
Third, The line between a resourced and under-resourced language is not always clear-cut. For example, there could be annotations available, however they be of a rather small in number (the case with Tibetan POS); or some data sets be very robust for some tasks (e.g., a lexicon and word-lookup) however be completely lacking treebanks (Pali); or in the absence of any digitized data, certain within-reach algorithms may be written. For such in-between languages, if a potential contributor will first do diligent research on data sets, project mentors will be delighted to discuss algorithmic possibilities.
Is there some limited data creation that could fall within scope of the CLTK? We have had some success, for example, in generating stopword lists either according to specific algorithms minimizing manual curation (e.g., run tf-idf on a corpus and removing nouns) or with very narrow scope (e.g., writing every possible inflection of a definite article). I don’t want to preclude other ideas, so I would generalize that this issue may be evaluated on a case-by-case basis.
All the above is intentionally discouraging of those who might want to disembark on an ill-fated proposal to add new language support to the CLTK. On the flip side, we ought to highlight that there are several shining examples of languages I might consider well-resourced and not currently covered by the CLTK. Those are, in no particular order: Hebrew, Arabic, Sanskrit, Chinese, and Old English.2 (There are likely others, too.) To conclude, I encourage those who would like to work with a mentor from our team, to first consider and have preliminary answers to the following questions:
With these questions answered, even if not perfectly, the core CLTK will be able to respond with concrete advice, criticism, and recommendations to further develop your proposal.
1: From the Under-resourced Languages group of the European Language Resources Association (ELRA).
2: Old and Middle English have good footing, due to the major pre-modern Germanic contributions by Eleftheria Chatziargyriou and Clément Besnier, however relative to the huge amounts of source data, lots of valuable work remains.
]]>I’d like to thank my friends who have read it for me and provided valuable comments.
While I do not plan on keeping this up-to-date, if you have feedback for me, I would appreciate hearing your thoughts.
]]>The first, to the Arts and Humanities Research and Computing group, introduced the project, the in-development frontend (which was discussed by Luke Hollis), and some of my own personal research into genres of the Ancient Greek canon. The question guiding me was: Can AI classify texts better than the ancients did themselves? An intentionally naive question, however the research allows us to dissect the relative consistency of various genres in terms of lexical patterns, morphology, and syntax. I have always been concerned with handy-wavy definitions of genres, and the corresponding difficulty of quantitatively defining them. My hope is that this research will help to ground some such considerations of genre in early European literature.
The second was not so much a lecture as a hands-on introduction to using the CLTK. If you are looking for a comfortable introduction to NLP and the project, this might be the best place to start.
]]>Since the announcement, while we’ve been in the student application period, interest in the project has skyrocketed, already with dozens of commits and hundreds of emails. The CLTK website, led by Luke Hollis, has seen tremendous growth already after only these two weeks. As of this moment, our stars on GitHub have doubled, watchers tripled, and – most importantly IMO – six times the forks!
There’s been a bit of publicity about our funding opportunities in the Classics world. Among this has been an interview of me and Luke in the newsletter of Harvard’s Center for Hellenic Studies, “Q&A with Kyle P. Johnson and Luke Hollis of the Classical Language Toolkit”.
Much more to come!
]]>The group to whom we lectured, Ascent fellows, was interesting, being made up of science PhDs from Columbia, NYU, CUNY, and Cornell. The purpose of this group, by my understanding, is to give students of the hard sciences the tools they need – business, techincal, social – to succeed in the job market. This kind of grooming is very important, and equally rare, for those acclimated to culture of academia. As “digital humanities” (or whatever it will morph into) matures, a program like Ascent’s could prove even more valuable for the success of post–PhD humanists. Since Ascent is funded by no less than the NSF, good arguments could be made for leading humanist–funding groups to sponsor something similar.
Our GitHub notebook is available. The task we tacked was the prediction (binary classification, to be precise) of “viral” tweets. We made a collection of unpopular tweets (those with less the 10 RTs) and popular (over 500). Then, using nothing more than the text of the tweet, we explored (a) how many features we could extract and (b) how well various algorthms performed. To our pleasant surprise, using only a tweet’s text, we came to an 84% precision and 85% recall (random forest and decision tree did equally well). Results would surely be improved were we to leverage two valuable feature sets – bag of words and topic modelling – however processing time on our local computers was too long for purposes of the class. Nevertheless, BOW and topic modeling code is included in the Jupyter notebooks should anyone want to give it a try. (I’ll note that, from casual review of the data, unpopular tweets have higher occurences of profanity. For this reason, I think BOW especially would raise precision.)
]]>The first lecture was last November, when I gave a guest lecture, to an NYU graduate class of Peter Meineck, on an introduction to NLP and the CLTK. This was a lot of fun, as I had time to dialog with the class and explore some texts (some plays of Aeschylus) together. The Jupyter notebooks I prepared are on GitHub. Here’s the lecture’s slide deck, which might function as an informal introduction for newcomers.
A second lecture, which I gave last week, was a much briefer, 5–minute “lightning talk” to the San Francisco Python Meetup Group. The subject of this talk was narrow, being about problems encountered when working with linguistic corpora. The core difficulty pertained to the sharing of experiments and reproduction of results. Too often, folks working in NLP settle for hard–to–get, poorly documented, and un–versioned data sets. As I worked with others’ corpora and created my own, I came to understand poor data set management to be a significant problem, not only for digital Classics but NLP in general. In a nutshell, the CLTK’s solution to this problem is to leverage the capacities of Git and GitHub, that data sets can be, among other things, precisely versioned and easily updated by end users.
]]>The difference with these latest versions is that the inflected occurances were lemmatized, then counted. This was made possible by the CLTK’s latest release, which now offers good lemmatization for Greek and Latin. You can see the lemmatizer in action in the notebook which generated the following two files.
Greek: The 10,000 most common Greek words, grouped by lemma in the Classical Greek canon.
Latin: The 10,000 most common Latin words, grouped by lemma in the Classical Latin canon.
]]>