A bag but is language nothing of words

(language is nothing but a bag of words)

THIS IS A DRAFT

Michael Murtaugh

In text indexing and other machine reading applications the term "bag of words" is frequently used to underscore how processing algorithms often represent text using a data structure (word histograms or weighted vectors) where the original order of the words in sentence form is stripped away. While "bag of words" might well serve as a cautionary reminder to programmers of the essential violence perpetrated to a text and a call to critically question the efficacy of methods based on subsequent transformations, the expression's use seems in practice more like a badge of pride or a schoolyard taunt that would go: Hey language: you're nothin' but a big BAG-OF-WORDS.

Bag of words

In information retrieval and other so-called machine-reading applications (such as text indexing for web search engines) the term "bag of words" is frequently used to underscore how in the course of processing a text the original order of the words in sentence form is stripped away. The resulting representation is then a collection of each unique word used in the text, typically weighted by the number of times the word occurs.

Bag of words, also known as word histograms or weighted term vectors, are a standard part of the data engineer's toolkit. But why such a drastic transformation? The utility of "bag of words" is in how it makes text amenable to code, first in that it's very straightforward to implement the translation from a text document to a bag of words representation. More significantly, this transformation then opens up a wide collection of tools and techniques for further transformation and analysis purposes. For instance, a number of libraries available in the booming field of "data sciences" work with "high dimension" vectors; bag of words is a way to transform a written document into a mathematical vector where each "dimension" corresponds to the (relative) quantity of each unique word. While physically unimaginable and abstract (imagine each of Shakespeare's works as points in a 14 million dimensional space), from a formal mathematical perspective, it's quite a comfortable idea, and many complementary techniques (such as principle component analysis) exist to reduce the resulting complexity.

What's striking about a bag of words representation, given is centrality in so many text retrieval application is its irreversibility. Given a bag of words representation of a text and faced with the task of producing the original text would require in essence the "brain" of a writer to recomposing sentences, combined with the patience of a devoted cryptogram puzzler (and ideally) to draw from the precise stock of available words. While "bag of words" might well serve as a cautionary reminder to programmers of the essential violence perpetrated to a text and a call to critically question the efficacy of methods based on subsequent transformations, the expressions use seems in practice more like a badge of pride or a schoolyard taunt that would go: Hey language: you're nothing but a big BAG-OF-WORDS. Following this spirit of the term, "bag of words" celebrates a perfunctory step of "breaking" a text into a purer form amenable to computation, to stripping language of its silly redundant repetitions and foolishly contrived stylistic phrasings to reveal a purer inner essence.

Book of words

Lieber's Standard Telegraphic Code, first published in 1896 and republished in various updated editions through the early 1900s, is an example of one of several competing systems of telegraph code books. The idea was for both senders and receivers of telegraph messages to use the books to translate their messages into a sequence of code words which can then be sent for less money as telegraph messages were paid by the word. In the front of the book, a list of examples gives a sampling of how messages like: "Have bought for your account 400 bales of cotton, March delivery, at 8.34" can be conveyed by a telegram with the message "Ciotola, Delaboravi". In each case the reduction of number of transmitted words is highlighted to underscore the efficacy of the method. Like a dictionary or thesaurus, the book is primarily organized around key words, such as act, advice, affairs, bags, bail, and bales, under which exhaustive lists of useful phrases involving the corresponding word are provided in the main pages of the volume.

• 
• 
• 
• 

[...] my focus in this chapter is on the inscription technology that grew parasitically alongside the monopolistic pricing strategies of telegraph companies: telegraph code books. Constructed under the bywords “economy,” “secrecy,” and “simplicity,” telegraph code books matched phrases and words with code letters or numbers. The idea was to use a single code word instead of an entire phrase, thus saving money by serving as an information compression technology. Generally economy won out over secrecy, but in specialized cases, secrecy was also important.

In Katherine Hayles' chapter devoted to telegraph code books she observes how:

The interaction between code and language shows a steady movement away from a human-centric view of code toward a machine-centric view, thus anticipating the development of full-fledged machine codes with the digital computer.

Aspects of this transitional moment are apparent in a notice included prominently inserted in the Lieber's code book:

After July, 1904, all combinations of letters that do not exceed ten will pass as one cipher word, provided that it is pronounceable, or that it is taken from the following languages: English, French, German, Dutch, Spanish, Portuguese or Latin -- International Telegraphic Conference, July 1903

Conforming to international conventions regulating telegraph communication at that time, the stipulation that code words be actual words drawn from a variety of European languages (many of Lieber's code words are indeed arbitrary Dutch, German, and Spanish words) underscores this particular moment of transition as reference to the human body in the form of "pronounceable" speech from representative languages begins to yield to the inherent potential for arbitrariness in digital representation.

What telegraph code books do is remind us of is the relation of language in general to economy. Whether they may be economies of memory, attention, costs paid to a telecommunicatons company, or in terms of computer processing time or storage space, encoding language or knowledge in any form of writing is a form of shorthand and always involves an interplay with what one expects to perform or "get out" of the resulting encoding.

Along with the invention of telegraphic codes comes a paradox that John Guillory has noted: code can be used both to clarify and occlude. Among the sedimented structures in the technological unconscious is the dream of a universal language. Uniting the world in networks of communication that flashed faster than ever before, telegraphy was particularly suited to the idea that intercultural communication could become almost effortless. In this utopian vision, the effects of continuous reciprocal causality expand to global proportions capable of radically transforming the conditions of human life. That these dreams were never realized seems, in retrospect, inevitable.

• 
• 
• 

Far from providing a universal system of encoding messages in the English language, Lieber's code is quite clearly designed for the particular needs and conditions of its use. In addition to the phrases ordered by keywords, the book includes a number of tables of terms for specialized use. One table lists a set of words used to describe all possible permutations of numeric grades of coffee (Choliam = 3,4, Choliambos = 3,4,5, Choliba = 4,5, etc.); another table lists pairs of code words to express the respective daily rise or fall of the price of coffee at the port of Le Havre in increments of a quarter of a Franc per 50 kilos ("Chirriado = prices have advanced 1 1/4 francs"). From an archaeological perspective, the Lieber's code book reveals a cross section of the needs and desires of early 20th century business communication between the United States and its trading partners.

The advertisements lining the Liebers Code book further situate its use and that of commercial telegraphy. Among the many advertisements for banking and law services, office equipment, and alcohol are several ads for gun powder and explosives, drilling equipment and metallurgic services all with specific applications to mining. Extending telegraphy's formative role for ship-to-shore and ship-to-ship communication for reasons of safety, commercial telegraphy extended this network of communication to include those parties coordinating the "raw materials" being mined, grown, or otherwise extracted from overseas sources and shipped back for sale.

"Raw data now!"

Tim Berners-Lee: [...] Make a beautiful website, but first give us the unadulterated data, we want the data. We want unadulterated data. OK, we have to ask for raw data now. And I'm going to ask you to practice that, OK? Can you say "raw"?

Audience: Raw.

Tim Berners-Lee: Can you say "data"?

Audience: Data.

TBL: Can you say "now"?

Audience: Now!

TBL: Alright, "raw data now"!

[...]

So, we're at the stage now where we have to do this -- the people who think it's a great idea. And all the people -- and I think there's a lot of people at TED who do things because -- even though there's not an immediate return on the investment because it will only really pay off when everybody else has done it -- they'll do it because they're the sort of person who just does things which would be good if everybody else did them. OK, so it's called linked data. I want you to make it. I want you to demand it.

Un/Structured

As graduate students at Stanford, Sergey Brin and Lawrence (Larry) Page had an early interest in producing "structured data" from the "unstructured" web.

The World Wide Web provides a vast source of information of almost all types, ranging from DNA databases to resumes to lists of favorite restaurants. However, this information is often scattered among many web servers and hosts, using many different formats. If these chunks of information could be extracted from the World Wide Web and integrated into a structured form, they would form an unprecedented source of information. It would include the largest international directory of people, the largest and most diverse databases of products, the greatest bibliography of academic works, and many other useful resources. [...]

2.1 The Problem
Here we define our problem more formally:
Let D be a large database of unstructured information such as the World Wide Web [...]

In a paper titled Dynamic Data Mining Brin and Page situate their research looking for rules (statistical correlations) between words used in web pages. The "baskets" they mention stem from the origins of "market basket" techniques developed to find correlations between the items recorded in the purchase receipts of supermarket customers. In their case, they deal with web pages rather than shopping baskets, and words instead of purchases. In transitioning to the much larger scale of the web, they describe the usefulness of their research in terms of its computational economy, that is the ability to tackle the scale of the web and still perform using contemporary computing power completing its task in a reasonably short amount of time.

A traditional algorithm could not compute the large itemsets in the lifetime of the universe. [...] Yet many data sets are difficult to mine because they have many frequently occurring items, complex relationships between the items, and a large number of items per basket. In this paper we experiment with word usage in documents on the World Wide Web (see Section 4.2 for details about this data set). This data set is fundamentally different from a supermarket data set. Each document has roughly 150 distinct words on average, as compared to roughly 10 items for cash register transactions. We restrict ourselves to a subset of about 24 million documents from the web. This set of documents contains over 14 million distinct words, with tens of thousands of them occurring above a reasonable support threshold. Very many sets of these words are highly correlated and occur often.

Un/Ordered

In programming, I've encountered a recurring "problem" that's quite symptomatic. It goes something like this: you (the programmer) have managed to cobble out a lovely "content management system" (either from scratch, or using one of dozens of popular framewords) where your user can enter some "items" (for instance bookmarks) into a database. After this ordered items are automatically presented in list form (say on a web page). The author: It's great, except... could this bookmark come before that one? The problem stems from the fact that the database ordering (a core functionality provided by any database) somehow applies a sorting logic that's almost but not quite right. A typical example is the sorting of names where details (where to place a name that starts with a Norwegian "Ø" for instance), are language-specific, and when a mixture of languages occurs, no single ordering is necessarily "correct". The (often) exascerbated programmer might hastily add an additional database field so that each item can also have an "order" (perhaps in the form of a date or some other kind of (alpha)numerical "sorting" value) to be used to correctly order the resulting list. Now the author has a means, awkward and indirect but workable, to control the order of the presented data on the start page. But one might well ask, why not just edit the resulting listing as a document? This problem, in this and many variants, is widespread and reveals an essential backwardness that a particular "computer scientist" mindset relating to what constitutes "data" and in particular it's relationship to order that makes what might be a straightforward question of editing a document into an over-engineered database.

Recently working with Nikolaos Vogiatzis whose research explores playful and radically subjective alternatives to the list, Vogiatzis was struck by how from the earliest specifications of HTML (still valid today) have separate elements (OL and UL) for "ordered" and "unordered" lists.

The representation of the list is not defined here, but a bulleted list for unordered lists, and a sequence of numbered paragraphs for an ordered list would be quite appropriate. Other possibilities for interactive display include embedded scrollable browse panels.

Vogiatzis' surprise lay in the idea of a list ever being considered "unordered" (or in opposition to the language used in the specification, for order to ever be considered "insignificant"). Indeed in its suggested representation, still followed by modern web browsers, the only difference between the two visually is that UL items are preceded by a bullet symbol, while OL items are numbered.

The idea of ordering runs deep in programming practice where essentially different data structures are employed depending on whether order is to be maintained. The indexes of a "hash" table, for instance (also known as an associative array), are ordered in an unpredictable way governed by the representation's particular implementation. This data structure, extremely prevalent in contemporary programming practice sacrifices order to offer other kinds of efficiency (fast text-based retrieval for instance).

Data mining

In announcing Google's impending data center in Mons, Belgian prime minister Di Rupo made the link between the history of the mining industry in the region and the present and future interest in "data mining" as practiced by Google. Apart from a metaphorical linkage, what in fact perhaps better links these different notions of "mining" is the way in which the notion of "raw material" obscures the hidden labor and power differentials employed to secure them.

( bags of words, bales of cotton, barrels of oil )

"Raw" is always relative: "purity" depends on processes of "refinement" that typically carry heavy social / ecological impact.

There's a parallel between the "disembodiment" implied in stripping language of its essential order (and thus situated it in the act of reading or writing) and a colonial logic of profit from the trade in resources produced by undervalued labor and practices with devastating ecological impact.

There's a parallel in a shifting of responsibilities away from the individual / political related to the "human scale" and the obscured responsibilities and seemingly inevitable forces of the "machine scale", be it the machine of market or the machine of algorithm.

Parallel shifts, telegraph and telephony a shift occurs from language as something performed by a human body, to becoming captured in code, and occurring at a machine scale. In document processing a similar shift occurs from language as writing to language as symbolic sets of information to be treated to statistical methods for extracting knowledge in the form of relationships.

Often when we speak of machine or computer in terms like machine reading or computer vision, we speak of a displacement of human labour ... often to apply condensed human labour typically embodied in the form of (trained) statistic models ... and a displacement of responsibility as values become encoded in the form of algorithms/software.

The interest in "machinic" (minimal human intervention) involves on first glance "machinic" in the traditional sense of automating labour, replacing the human work of categorizing with an automated process; in this way opening up the process to a larger quantity of pages and a range of "esoteric" topics which would not be possible to handle with traditional editorial processes. This "machinic" shift is a business model that learns to extract the value of web surfers behaviour; this process is then echoed in google's book digitization which similarly "extracts" / exploits the value of the collection librarian (on top of the work of the author, the typesetter, the publisher)

The computer scientists view of textual content as "unstructured", be it in a webpage or the pages of a scanned text, underscore / reflect the negligence to the processes and labor of writing, editing, design, layout, typesetting, and eventually publishing, collecting and cataloging .

Separation of "content" and "presentation".

• Tim Berners-Lee TED Talk, February 2009
• In a sense, chastising those already publishing to the web; ironically it's in large part the blindness to the value of "free-form" web publishing (that pretty website sure is nice, but what I want is the underlying DATA), that helps collapse the web and shift back to centralized services where labor is explicitly performed/harnessed web 2.0 style.

In other words, by "unstructured" it is meant: unstructured in relation to the machine -- that is, not explicitly structured in a format directly amenable to use by automated means. "Structuring" then is a process by which structure is made explicit through the use of standards of markup (such as HTML/XML). In this way, the computer scientist is viewing a text through the eyes of their reading algorithm, and in the process (voluntarily) blinding themselves to the work practices which have produced, and maintain, the given textual resources, choosing to view them as instead somehow "freely given" and available to exploit as a "raw material".