The following is a guest post by Richard Sproat:
Regular readers of Language Log will remember this piece discussing the various problems with a paper by Rajesh Rao and colleagues in their attempt to provide statistical evidence for the ...
The following is a guest post by Richard Sproat:
Regular readers of Language Log will remember this piece discussing the various problems with a paper by Rajesh Rao and colleagues in their attempt to provide statistical evidence for the status of the Indus “script” as a writing system. They will also recall this piece on a similar paper by Rob Lee and colleagues, which attempted to demonstrate linguistic structure in Pictish inscriptions. And they may also remember this discussion of my “Last Words” paper in Computational Linguistics critiquing those two papers, as well as the reviewing practices of major science journals like Science.
In a nutshell: Rao and colleagues’ original paper in Science used conditional entropy to argue that the Indus “script” behaves more like a writing system than it does like a non-linguistic system. Lee and colleagues’ paper in Proceedings of the Royal Society used more sophisticated methods that included entropic measures to build a classification tree that apparently correctly classified a set of linguistic and non-linguistic corpora, and furthermore classified the Pictish symbols as logographic writing.
But as discussed in the links given above, both of these papers were seriously problematic, which in turn called into question some of the reviewing standards of the journals involved.
Sometimes a seemingly dead horse has to be revived and beaten again, for those reviewing practices have yet again come into question. Or perhaps I should in this case say “non reviewing practices”: for an explanation, read on.
Not being satisfied by merely critiquing the previous work, I decided to do something constructive, and investigate more fully what one would find if one looked at a larger set of corpora of non-linguistic symbol systems, and contrasted them with a larger set of corpora of written language. Would the published methods of Rao et al. and Lee et al. hold up? Or would they fail as badly as I predicted they would? Are there any other methods that might be useful as evidence for a symbol system’s status? In order to answer those questions I needed to collect a reasonable set of corpora of non-linguistic systems, something that nobody had ever done. And for that I needed to be able to pay research assistants. So I applied for, and got, an NSF grant, and employed some undergraduate RAs to help me collect the corpora. A paper on the collection of some of the corpora was presented at the 2012 Linguistic Society of America meeting in Portland.
Then, using those corpora, and others I collected myself, I performed various statistical analyses, and wrote up the results (see below for links to a paper and a detailed description of the materials and methods). In brief summary: neither Rao’s nor Lee’s methods hold up as published, but a measure based on symbol repetition rate as well as a reestimated version of one of Lee et al.’s measures seem promising — except that if one believes those measures, then one would have to conclude that the Indus “script” and Pictish symbols are, in fact, not writing.
So for example in a paper published in IEEE Computer (Rao, R., 2010. Probabilistic analysis of an ancient undeciphered script. IEEE Computer. 43~(3), 76–80), Rao uses the entropy of ngrams — unigrams, bigrams, trigrams and so forth, which he terms “block entropy” — as a measure to show that the Indus “script” behaves more like language than it does like some non-linguistic systems. He gives the following plot:
For this particular analysis Rao describes exactly the method and software package he used to compute these results, so it is possible to replicate his method exactly for my own data.
The results of that are shown below, where linguistic corpora are shown in red, non-linguistic in blue, and for comparison a small corpus of Indus bar seals in green. As can be seen, for a representative set of corpora, the whole middle region of the block entropy growth curves is densely populated