Tag Archives: japanese

Using Collections – Virtually

I heard a remark the other day that struck a cord – or, churned my butter, a bit.

The gist of it was, “we should make digital facsimiles of our library materials (especially rare materials) and put them online, so they spark library use when people visit to see them in person, after becoming aware of them thanks to the digitized versions.”

Now, at Penn, we have digitized a couple of Japanese collections: Japanese Juvenile Fiction Collection (aka Tatsukawa Bunko series), Japanese Naval Collection (in-process, focused on Renshū Kantai training fleet materials), and a miscellaneous collection of Japanese rare books in general.* These materials have been used both in person (thanks to publicizing them, pre- and post-digitization, on library news sites, blogs, and social media as well as word-of-mouth), and also digitally by researchers who cannot travel to Penn. In fact, a graduate student in Australia used our juvenile fiction collection for part of his dissertation; another student in Wisconsin plans to use facsimiles of our naval materials once they’re complete; and faculty at University of Montana have used our digital facsimile of Meiji-period journal Hōbunkai-sui (or Hōbunkai-shi).

These researchers, due to distance and budget, will likely never be able to visit Penn in person to use the collections. On top of that, some items – like the juvenile fiction and lengthy government documents related to the Imperial Navy – don’t lend themselves to using in a reading room. These aren’t artifacts to look over one page at a time, but research materials that will be read extensively (rather than “intensively,” a distinction we book history folks make). Thus, this is the only use they can make of our materials.

The digitization of Japanese collections at Penn has invited use and a kind of library visit by virtue of being available for researchers worldwide, not just those who are at Penn (who could easily view them in person and don’t “need” a digital facsimile), or who can visit the library to “smell” the books (as the person I paraphrased put it). I think it’s more important to be able to read, research, and use these documents than to smell or witness the material artifact. Of course, there are cases in which one would want to do that, but by and large, our researchers care more about the content and visual aspects of the materials – things that can be captured and conveyed in digital images – rather than touching or handling them.

Isn’t this use, just as visiting the library in person use? Shouldn’t we be tracking visits to our digital collections, downloads, and qualitative stories about their use in research, just as we do a gate count and track circulation? I think so. As we think about the present and future of libraries, and people make comments about their not being needed because libraries are on our smartphones (like libraries of fake news, right?), we must make the argument for providing content both physically and virtually. Who do people think is providing the content for their digital libraries? Physical libraries, of course! Those collections exist in the real world and come from somewhere, with significant investments of money, time, and labor involved – and moreover, it is the skilled and knowledgable labor of professionals that is required.

On top of all of this, I feel it is most important to own up to what we can and cannot “control” online: our collections, by virtue of being able to be released at all, are largely in the public domain. Let’s not put CC licenses on them except for CC-0 (which is explicitly marking materials as public domain), pretending we can control the images when we have no legal right to (but users largely don’t know that). Let’s allow for free remixing and use without citing the digital library/archive it came from, without getting upset about posts on Tumblr. When you release public domain materials on the web (or through other services online), you are giving up your exclusive right to control the circumstances under which people use it – and as a cultural heritage institution, it is your role to perform this service for the world.

But not only should we provide this service, we should take credit for it: take credit for use, visits, and for people getting to do whatever they want with our collections. That is really meaningful and impactful use.

* Many thanks to Michael Williams for his great blog posts about our collections!

Taiyō project: first steps with data

As I begin working on my project involving Taiyō magazine, I thought I’d document what I’m doing so others can see the process of cleaning the data I’ve gotten, and then experimenting with it. This is the first part in that series: first steps with data, cleaning it, and getting it ready for analysis. If I have the Taiyō data in “plain text,” what’s there to clean? Oh, you have no idea.

taiyo_data Continue reading Taiyō project: first steps with data

how to make Japanese udon/soba broth

After packing my copy of Japanese Cooking Contemporary and Traditional (an awesome vegan cookbook) a little hastily before my upcoming move, I scoured the Internet for the basic udon/soba noodle broth recipe. To my surprise, it is not on the Oracle. So here I’ll provide the standard Japanese recipe for soba/udon broth for posterity.

You need:

  • 4 cups any kind of broth (for example konbu-dashi, katsuo-dashi, or chicken, or fake-chicken as I used last night)
  • 2-5 tablespoons light (usu-kuchi) soy sauce as desired for saltiness
  • 1 tablespoon or so ryōri-shu (cooking sake, the cheap kind)
  • 1 teaspoon mirin or sugar if you don’t have mirin handy
  • 1/2 teaspoon salt

Simmer all this together for about 5 minutes and pour over the noodles. Adjust all the salty and sugary elements based on your taste. This makes enough for 2 servings.

You’re welcome, Internet!

Japanese tokenization – tools and trials

I’ve been looking (okay, not looking, wishing) for a Japanese tokenizer for a while now, and today I decided to sit down and do some research into what’s out there. It didn’t take long – things have improved recently.

I found two tools quickly: kuromoji Japanese morphological analyzer and the U-Tokenizer CJK Tokenizer API.

First off – so what is tokenization? Basically, it’s separating sentences by words, or documents by sentences, or any text by some unit, to be able to chunk that text into parts and analyze them (or do other things with them). When you tokenize a document by word, like a web page, you enable searching: this is how Google finds individual words in documents. You can also find keywords from a document this way, by writing an algorithm to choose the most meaningful nouns, for example. It’s also the first step in more involved linguistic analysis like part-of-speech tagging (thing, marking individual words as nouns, verbs, and so on) and lemmatizing (paring words down to their stems, such as removing plural markers and un-conjugating verbs).

This gives you a taste of why tokenization is so fundamental and important for text analysis. It’s what lets you break up an otherwise unintelligible (to the computer) string of characters into units that the computer can attempt to analyze. It can index them, search them, categorize them, group them, visualize them, and so on. Without this, you’re stuck with “words” that are entire sentences or documents, that the computer thinks are individual units based on the fact that they’re one long string of characters.

Usually, the way you tokenize is to break up “words” based on spaces (or sentences based on punctuation rules, etc., although that doesn’t always work). (I put “words” in quotes because you can really make any kind of unit you want, the computer doesn’t understand what words are, and in the end it doesn’t matter. I’m using “words” as an example here.) However, for languages like Japanese and Chinese (and to a lesser extent Korean) that don’t use spaces to delimit all words (for example, in Korean particles are attached to nouns with no space in between, like saying “athome” instead of “at home”), you run into problems quickly. How to break up texts into words when there’s no easy way to distinguish between them?

The question of tokenizing Japanese may be a linguistic debate. I don’t know enough about linguistics to begin to participate in it, if it is. But I’ll quickly say that you can break up Japanese based on linguistic rules and dictionary rules – understanding which character compounds are nouns, which verb conjugations go with which verb stems (as opposed to being particles in between words), then breaking up common particles into their own units. This appears to be how these tools are doing it. For my own purposes, I’m not as interested in linguistic patterns as I am in noun and verb usage (the meaning rather than the kind) so linguistic nitpicking won’t be my area anyway.

Moving on to the tools. I put them through the wringer: Higuchi Ichiyō’s Ame no yoru, the first two lines, from Aozora bunko.

One, kuromoji, is the tokenizer behind Solr and Lucene. It does a fairly good job, although with Ichiyō’s uncommon word usage and conjugation, it faltered and couldn’t figure out that 高やか is one word; rather it divided it into 高 や か.  It gives the base form, reading, and pronunciation, but nothing else. However, in the version that ships with Solr/Lucene, it lemmatizes. Would that ever make me happy. (That’s, again, reducing a word to its base form, making it easy to count all instances of both “people” and “person” for example, if you’re just after meaning.) I would kill for this feature to be integrated with the below tool.

The other, U-Tokenizer, did significantly better, but its major drawback is that it’s done in the form of an HTTP request, meaning that you can’t put in entire documents (well, maybe you could? how much can you pass in an HTTP request?). If it were downloadable code with an API, I would be very happy (kuromoji is downloadable and has a command line interface). U-Tokenizer figured out that 高やか is one word, and also provides a list of “keywords,” which as far as I can tell is a bunch of salient nouns. I used it for a very short piece of text, so I can’t comment on how many keywords it would come up with for an entire document. The documentation on this is sparse, and it’s not open source, so it’s impossible to know what it’s doing. Still, it’s a fantastic tool, and also seems to work decently for Chinese and Korean.

Each of these tools has its strengths, and both are quite usable for modern and contemporary Japanese. (I really was cruel to feed them Ichiyō.) However, there is a major trial involved in using them with freely-available corpora like Aozora bunko. Guess what? Preprocessing ruby.

Aozora texts contain ruby marked up within the documents. I have my issues with stripping out ruby from documents that heavily use them (like Meiji writers, for example) because they add so much meaning to the text, but let’s say for argument’s sake that we’re not interested in the ruby. Now, it’s time to cut it all out. If I were a regular expressions wizard (or even had basic competency with them) I could probably strip this out easily, but it’s still time consuming. Download text, strip out ruby and other metadata, save as plain text. (Aozora texts are XHTML, NOT “plain text” as they’re often touted to be.) Repeat. For topic modeling using a tool like MALLET, you’re going to want to have hundreds of documents at the end of it. For example, you might be downloading all Meiji novels from Aozora and dividing them into chunks or chapters. Even the complete works of Natsume Sōseki aren’t enough without cutting them down into chapters or even paragraphs to make enough documents to use a topic modeling tool effectively. Possibly, run all these through a part-of-speech tagger like KH Coder. This is going to take a significant amount of time.

Then again, preprocessing is an essential and extremely time-consuming part of almost any text analysis project. I went through a moderate amount of work just removing Project Gutenberg metadata and dividing into chapters a set of travel narratives that I downloaded in plain text, thankfully not in HTML or XML. It made for easy processing. With something that’s not already real plain text, with a lot of metadata, and with a lot of ruby, it’s going to take much more time and effort, which is more typical of a project like this. The digital humanities are a lot of manual labor, despite the glamorous image and the idea that computers can do a lot of manual labor for us. They are a little finicky with what they’ll accept. (Granted, I’ll be using a computer script to strip out the XHTML and ruby tags, but it’s going to take work for me to write it in the first place.)

In conclusion? Text analysis, despite exciting available tools, is still hard and time consuming. There is a lot of potential here, but I also see myself going through some trials to get to the fun part, the experimentation. Still, stay tuned, especially for some follow-up posts on these tools and KH Coder as I become more familiar with them. And, I promise to stop being difficult and giving them Ichiyō’s Meiji-style bungo.

android slashdot reader: 和英コメントで言語学び!

Now that I have an Android phone and have found some pretty great things on the Android Market for getting ahold of Japanese content, I would like to start sharing with you all what I’ve been using and whether it’s worth downloading.

First up is my favorite new find: Slashdot Reader. Yeah. It’s an RSS feed reader for Slashdot. Why so great?

Well, can you imagine my reaction when I read its description and saw images of posts from slashdot.org and slashdot.jp showing up all mixed in together? Then I read the description: “just a feed reader, nothing more” – for both Japanese and English Slashdot.

It’s like I found an app made by my doppelganger. Really.

If you don’t want one or the other of the languages, it allows you to toggle both, Japanese only, and English only.

Because it’s a feed reader, you only get the headlines and leads from Slashdot, but can easily click through to the full story, and therein lies the amazing language learning tool that somehow never really occurred to me.

All of these years, I could have been learning Japanese through Slashdot comments! That’s right. Of course it’s not textbook Japanese. I already know how trolls (荒らし) talk after just a minute or two of reading. How nerds talk. (They always use が and never けど, although they do use ね sparingly for emphasis. A certain language teacher from several years ago who forbid us from using けど in class for an entire semester would be proud.) And how random users talk.

I also know how they’re basically saying exactly the same things that commenters do on Slashdot in English, only they’re saying it in Japanese. (open source != free as in beer, anyone? I seriously just read this. 無償 is free as in beer, and note that it’s not the same as the widely-used word for “free” 無料 – so I just learned something new about software licensing.) So if you’re a Slashdot reader, this is going to help you immensely. It’s all about context.

Yes, so there are people out there who would disparage the idea of learning language from internet comments. But I counter that with: it’s real language! And this is a specific forum where you know what is coming: some nerdspeak, some posturing, some trolls, some reasonable people, talking about a rather limited set of topics. So you are going to learn voices, not just “Japanese.” You are going to learn what people say in a certain situation, and also what not to say. I can’t think of anything more helpful than that!

And here you go: Slashdot Reader for Android (this takes you to Android Market).

podcast: 学問のススメ

New podcast for me, and best podcast ever in my opinion. Especially for learning! And you have to love the title. It is…

学問のススメ – “Special Edition” (ラジオ版) (Gakumon no susume – radio)

(Why is the title great? Because Fukuzawa Yukichi, who was all about enlightenment, published a book in the 1870s called “Gakumon no susume.” In other words (and in my words), “The furthering of knowledge/study.” This podcast takes the same name, somewhat sincerely and somewhat tongue-in-cheek.)

The mission: Learn things that you either didn’t learn in school, or that you forgot. The format: Interviews with a variety of interesting people, from authors to reporters to photographers to athletes. Why? It’s interesting, you’re exposed to many different fields, and for Japanese learners, it’s fast-paced but obviously a great source of vocabulary on different topics and practice listening to “real Japanese” at native speed, intended for the average layperson. And best of all, it’s free!

You can download individual episodes, or subscribe via the Japanese iTunes store. (You don’t need an account or Japanese credit card to download episodes that are free – just change your location to “Japan” at the bottom of your iTunes Store home screen and search for the title. If you’d like to find more podcasts, just browse away by topic!)

Click here for 学問のススメ!