Category Archives: technology

Convert docs with OS X terminal

I’m teaching a workshop on Japanese text mining this week and am getting all kinds of interesting practical questions that I don’t know the answer to. Today, I was asked if it’s possible to batch convert .docx files to .txt in Windows.

I don’t know Windows, but I do know Mac OS, so I discovered that one can use textutil in the terminal to do this. Just run this line to convert .docx -> .txt:

textutil -convert txt /path/to/DOCX/files/*.docx

You can convert to a bunch of different formats, including txt, html, rtf, rtfd, doc, docx, wordml, odt, or webarchive. It puts the files in the same directory as the source files. That’s it: enjoy!

* Note: This worked fine with UTF-8 files using Japanese, so I assume it just works with UTF-8 in general. YMMV.

Using Collections – Virtually

I heard a remark the other day that struck a cord – or, churned my butter, a bit.

The gist of it was, “we should make digital facsimiles of our library materials (especially rare materials) and put them online, so they spark library use when people visit to see them in person, after becoming aware of them thanks to the digitized versions.”

Now, at Penn, we have digitized a couple of Japanese collections: Japanese Juvenile Fiction Collection (aka Tatsukawa Bunko series), Japanese Naval Collection (in-process, focused on Renshū Kantai training fleet materials), and a miscellaneous collection of Japanese rare books in general.* These materials have been used both in person (thanks to publicizing them, pre- and post-digitization, on library news sites, blogs, and social media as well as word-of-mouth), and also digitally by researchers who cannot travel to Penn. In fact, a graduate student in Australia used our juvenile fiction collection for part of his dissertation; another student in Wisconsin plans to use facsimiles of our naval materials once they’re complete; and faculty at University of Montana have used our digital facsimile of Meiji-period journal Hōbunkai-sui (or Hōbunkai-shi).

These researchers, due to distance and budget, will likely never be able to visit Penn in person to use the collections. On top of that, some items – like the juvenile fiction and lengthy government documents related to the Imperial Navy – don’t lend themselves to using in a reading room. These aren’t artifacts to look over one page at a time, but research materials that will be read extensively (rather than “intensively,” a distinction we book history folks make). Thus, this is the only use they can make of our materials.

The digitization of Japanese collections at Penn has invited use and a kind of library visit by virtue of being available for researchers worldwide, not just those who are at Penn (who could easily view them in person and don’t “need” a digital facsimile), or who can visit the library to “smell” the books (as the person I paraphrased put it). I think it’s more important to be able to read, research, and use these documents than to smell or witness the material artifact. Of course, there are cases in which one would want to do that, but by and large, our researchers care more about the content and visual aspects of the materials – things that can be captured and conveyed in digital images – rather than touching or handling them.

Isn’t this use, just as visiting the library in person use? Shouldn’t we be tracking visits to our digital collections, downloads, and qualitative stories about their use in research, just as we do a gate count and track circulation? I think so. As we think about the present and future of libraries, and people make comments about their not being needed because libraries are on our smartphones (like libraries of fake news, right?), we must make the argument for providing content both physically and virtually. Who do people think is providing the content for their digital libraries? Physical libraries, of course! Those collections exist in the real world and come from somewhere, with significant investments of money, time, and labor involved – and moreover, it is the skilled and knowledgable labor of professionals that is required.

On top of all of this, I feel it is most important to own up to what we can and cannot “control” online: our collections, by virtue of being able to be released at all, are largely in the public domain. Let’s not put CC licenses on them except for CC-0 (which is explicitly marking materials as public domain), pretending we can control the images when we have no legal right to (but users largely don’t know that). Let’s allow for free remixing and use without citing the digital library/archive it came from, without getting upset about posts on Tumblr. When you release public domain materials on the web (or through other services online), you are giving up your exclusive right to control the circumstances under which people use it – and as a cultural heritage institution, it is your role to perform this service for the world.

But not only should we provide this service, we should take credit for it: take credit for use, visits, and for people getting to do whatever they want with our collections. That is really meaningful and impactful use.

* Many thanks to Michael Williams for his great blog posts about our collections!

Taiyō project: first steps with data

As I begin working on my project involving Taiyō magazine, I thought I’d document what I’m doing so others can see the process of cleaning the data I’ve gotten, and then experimenting with it. This is the first part in that series: first steps with data, cleaning it, and getting it ready for analysis. If I have the Taiyō data in “plain text,” what’s there to clean? Oh, you have no idea.

taiyo_data Continue reading Taiyō project: first steps with data

website to jekyll

While my research diary has stalled out because I haven’t been researching (other than some administrative tasks like collecting and organizing article PDFs, and typing notes into Mendeley), I have made some progress on updating my website.

Specifically, I have switched over to using Jekyll, which is software that converts markdown/HTML and SASS/CSS to static web pages. Why do I want to do it? Because I want to have a consistent header and footer (navigation and that blurb at the bottom of every page) across the whole site, but don’t want to manually edit every single file every time I update one of those, or update the site structure/design. I also didn’t want to use PHP because then all my files will be .php and on top of it, it feels messier. I like static HTML a lot.

I’m just writing down my notes here for others who might want to use it too. I’ve only found tutorials that talk about how to publish your site to GitHub Pages. Obviously, I have my own hosting. I also already had a full static site coded in HTML and CSS, so I didn’t want to start all over again with markdown. (Markdown is just a different markup language from HTML; from what I can tell, you can’t get nearly the flexibility or semantic markup into your markup documents that you can with HTML, so I’m sticking with the latter.) I wondered: all these tutorials show you how to do it from scratch, but will it be difficult to convert an existing HTML/CSS site into a Jekyll-powered site?

The answer is: no. It’s really really easy. Just copy and paste from your old site into some broken-up files in the Jekyll directory, serve, and go.

I recommend following the beginning of this tutorial by Tania Rascia. This will help you get Jekyll installed and set up.

Then, if you want a website — not a blog — what you want to do is just start making “index.html”, “about.html”, folders with more .html files (or .md if you prefer), etc., in your Jekyll folder. These will all be generated as regular .html pages in the _site directory when you start the server, and will be updated as long as the server is running. It’ll all be structured how you set it up in the Jekyll folder. For my site, that means I have folders like “projects” and “guides” in addition to top-level pages (such as “index.html”).

Finally, start your server and generate all those static pages. Put your CSS file wherever the head element wants it to be on your web server. (I have to use its full URL, starting with http://, because I have multiple folders and if I just put “mollydesjardin.css” the non-top-level files will not know where to find it.) Then upload all the files from _site into your server and voilà, you have your static website.

I do not “get” Git enough yet to follow some more complicated instructions I found for automatically pushing my site to my hosting. What I’m doing, and is probably the simplest but just a little cumbersome solution, is to just manually SFTP those files to my web server as I modify them. Obviously, I do not have to upload and overwrite every file every time; I just select the ones I created or modified from the _site directory and upload those.

Hope this is helpful for someone starting out with Jekyll, converting an existing HTML/CSS site.

thinking about ‘sentiment analysis’

I just got off the phone with a researcher this morning who is interested in looking at sentiment analysis on a corpus of fiction, specifically by having some native speakers of Japanese (I think) tag adjectives as positive or negative, then look at the overall shape of the corpus with those tags in mind.

A while back, I wrote a paper about geoparsing and sentiment analysis for a class, describing a project I worked on. Talking to this researcher made me think back to this project – which I’m actually currently trying to rewrite in Python and then make work on some Japanese, rather than Victorian English, texts – and my own definition of sentiment analysis for humanistic inquiry.*

How is my definition of sentiment analysis different? How about I start with the methodology? What I did was look for salient adjectives, which I searched for by looking at most “salient” nouns (not necessarily the most frequent, but I need to refine my heuristics) and then the adjectives that appeared next to them. I also used Wordnet to look for words related to these adjectives and nouns to expand my search beyond just those specific words to ones with similar meaning that I might have missed (in particular, I looked at hypernyms (broader terms) and synonyms of nouns, and synonyms of adjectives).

My method of sentiment analysis ends up looking more like automatic summarization than a positive-negative sentiment analysis we more frequently encounter, even in humanistic work such as Matt Jockers’s recent research. I argue, of course, that my method is somewhat more meaningful. I consider all adjectives to be sentiment words, because they carry subjective judgment (even something that’s kind of green might be described by someone else as also kind of blue). And I’m more interested in the character of subjective judgment than whether it should be able to be considered ‘objectively’ as positive or negative (something I don’t think is really possible in humanistic inquiry, and even in business applications). In other words, if we have to pick out the most representative feelings of people about what they’re experiencing, what are they feeling about that experience?

After all, can you really say that weather is good or bad, that there being a lot of farm fields is good or bad? I looked at 19th-century British women’s travel narratives of “exotic” places, and I found that their sentiment was often just observations about trains and the landscape and the people. They didn’t talk about whether they were feeling positively or negatively about those things; rather, they gave us their subjective judgment of what those things were like.

My take on sentiment analysis, then, is clearly that we need to introduce human judgment to the end of the process, perhaps gathering these representative phrases and adjectives (I lean toward phrases or even whole sentences) and then deciding what we can about them. I don’t even think a human interlocutor could put down a verdict of positive or negative on these observations and judgments – sentiments – that the women had about their experiences and environments. If not even a human could do it, and humans write and train the algorithms, how can the computer do it?

Is there even a point? Does it matter if it’s possible or not? We should be looking for something else entirely.

(I really need to get cracking on this project. Stay tuned for the revised methodology and heuristics, because I hope to write more and share code here as I go along.)

* I’m also trying to write a more extensive and revised paper on this, meant for the new incarnation of LLC.

academic death squad

Are you interested in joining a supportive academic community online? A place to share ideas, brainstorming, motivation and inspiration, and if you’re comfortable, your drafts and freewriting and blogging for critique? If so, Academic Death Squad may be for you.

This is a Google group that I believe can be accessed publicly (although I’ve had some issues with signing up with non-Gmail addresses) although you appear to have to be logged in to Google to view the group’s page. Just put in a request to join and I’ll approve you. Or, if that doesn’t work, email me at mdesjardin (at) gmail.com.

Link: [Academic Death Squad]

I’m trying to get as many disciplines and geographic/chronological areas involved as possible, so all are welcome. And I especially would love to have diversity in careers, mixing in tenure-track faculty, adjuncts, grad students, staff broadly interpreted, librarians, museum curators, and independent scholars – and any other career path you can think of. Many of us not in grad student or faculty land have very little institutional support for academic research, so let’s support each other virtually.

In fact, one member has already posted a publication-ready article draft for last-minute comments, so we even have a little activity already!

Best regards and best wishes for this group. Please email me or comment on this post if you have questions, concerns, or suggestions.

よろしくお願いいたします!

*footnote: The name came originally based on a group I ran called “Creative Death Squad” but the real origin is an amazing t-shirt I used to own in Pittsburgh that read “412 Vegan Death Squad” and had a picture of a skull with a carrot driven through it. I hope the name connotates badass-ness, serious commitment to our research, and some casual levity. Take it as you will.

arsenal of research: organizing citations, PDFs, notes, brainstorming, and drafts

Post title courtesy of the tyrannical Brian Vivier.

Although I post about the content of my research quite a bit (when I do post), I thought I’d take a step back and talk about the research process today. I’m going to write about a very specific aspect: the ways in which the computer helps me organize and engage in my research.

Obviously, there are things like databases and library catalogs, which are a topic for another day. Many people I talk to don’t know the first thing about WorldCat, so it needs to be addressed! But let’s pretend I already have my sources. Now what do I do?

When I read, I’m very traditional. I take notes with pen and paper when I have a book or a photocopied source. In fact, I used to print out PDFs too, and highlight and write in the margins. Well, that turned out to be a terrible idea. Your highlights and margin notes are not very accessible when you’re coming back to the document later to brainstorm, outline, or write.

My lesson learned – learned after many difficult situations – was to take notes like I’m never going to see the source again. My advisor recommended I do this with primary sources, but if you take long notes that involve mostly direct quotes from the sources, there’s no need to buy the book or really even check it out again. There’s no need to keep binders and binders of printed-out PDFs. So that’s the kind of note-taking I do with pen and paper, first.

The next step is to get them into the computer, because I want them to be 1) stored somewhere safe (I do daily external HD backups, plus sync, more later on that), and 2) searchable, and also 3) copy and paste-able. But where to keep them? How to organize?

I have gone through several pieces of software trying to figure this out, and I’ve settled on Mendeley. I first used Scrivener even for note-taking, which is a great program, but bad for citation management. I then tried Zotero, but that turned out to be bad for PDF management. What I really wanted was a good database that would save my citations, any PDFs I happened to have (I’m currently digitizing all of my sources from my dissertation so they don’t get lost or damaged, and so I can free up my filing cabinet for other things), and ideally let me take notes and even annotate or highlight the PDFs.

Well, despite Mendeley being owned by the devil (Elsevier), it’s free and it actually does everything I need with only a few minor nitpicks, and does it in a way that makes me supremely happy. (My nitpicks are no nested bulleted lists in the notes, and no shortcut keys for bold/italics in the notes.) If you have a PDF attached to your citation and it has OCR, Mendeley’s search function will search not only your citations, notes, and annotations, but also inside the PDFs. It can be overkill at times, but it’s pretty amazing.

So step two of my research organization process is the painstaking, mindless, thankless task of typing my pen-and-paper notes into Mendeley under the appropriate citation. It’s boring but worth it. As I mentioned above, it searches all my notes, and I can copy and paste them into Scrivener, which I will address next. As I type my notes, at the very least I copy and paste them into brainstorming documents as appropriate (usually full quotes), and if I’m up to it, I do some free-writing to brainstorm how the source informs my topic and what I could write about related to it. This usually brings up new ideas I didn’t know I had.

What happens after I get all the notes typed in, PDFs organized and annotated if I have them? I next move over to Scrivener. I’ve been using it for over five years, for both research and creative writing, and can’t sing its praises enough. It’s a word processor that creates a database for your project, where you can store your reference materials, brainstorming ideas, notes, and draft. And more, if you can think of other areas you need to record notes in. Unlike old Scrivener (when I first started using it), you can now add footnotes and comments that port straight to MS Word when you compile your document for it, making the transition to final draft in Word very easy. (Sadly, publishers seem to prefer things that are not Scrivener databases when reviewing.) The typical things I store are the draft itself (of course), a research diary of brainstorming that I update periodically, brainstorming specifically about sources and particular concepts or points, and also under the “Notes” section the comments and suggestions and draft corrections I receive from others. So I keep my full writing process, except for mind mapping/concept mapping (another post), all in one place. It’s amazing.

I’m extremely happy with these two pieces of software; my only complaint is that neither of them does all of what I want, and I have to use two different things complementarily. Well, the situation is still significantly better than several years ago, when I used Mendeley Alpha and it deleted my entire library of citations multiple times. Yikes. Now its syncing works perfectly and I haven’t had a library failure yet. (Fingers crossed).

Next posts will include mind mapping software, how I take notes, how to effectively find and import source citations, and how I deal with multiple languages in my citations.

Pre-processing Japanese literature for text analysis

I recently wrote a small script to perform a couple of functions for pre-processing Aozora Bunko texts (text files of public domain, modern Japanese literature and non-fiction) to be used with Western-oriented text analysis tools, such as Voyant, other TAPoR tools, and MALLET. Whereas Japanese text analysis software focuses largely on linguistics (tagging parts of speech, lemmatizing, etc.), Western tools open up possibilities for visualization, concordances, topic modeling, and other various modes of analysis.

Why do these Aozora texts need to be processed? Well, a couple of issues.

  1. They contain ruby, which are basically glosses of Chinese characters that give their pronunciation. These can be straightforward pronunciation help, or actually different words that give added meaning and context. While I have my issues with removing ruby, it’s impossible to do straightforward tool-based analysis without removing it, and many people who want to do this kind of analysis want it to be removed.
  2. The Aozora files are not exactly plain text: they’re HTML. The HTML tags and Aozora metadata (telling where the text came from, for example) need to be removed before analysis can be performed.
  3. There are no spaces between words in Japanese, but Western text analysis tools identify words by looking at where there are spaces. Without inserting spaces, it looks like each line is one big word. So I needed to insert spaces between the Japanese words.

How did I do it? My approach, because of my background and expertise, was to create a Python script that used a couple of helpful libraries, including BeautifulSoup for ruby removal based on HTML tags, and TinySegmenter for inserting spaces between words. My script requires you to have these packages installed, but it’s not a big deal to do so. You then run the script in a command line prompt. The way it works is to look for all .html files in a directory, load them and run the pre-processing, then output each processed file with the same filename, .txt ending, a plain text UTF-8 encoded file.

The first step in the script is to remove the ruby. Helpfully, the ruby is contained in several HTML tags. I had BeautifulSoup traverse the file and remove all elements contained within these tags; it removes both the tags and content.

Next, I used a very simple regular expression to remove everything in brackets – i.e. the HTML tags. This is kind of quick and dirty, and won’t work on every file in the universe, but in Aozora texts everything inside a bracket is an HTML tag, so it’s not a problem here.

Finally, I used TinySegmenter on the resulting HTML-free text to split the text into words. Luckily for me, it returns an array of words – basically, each word is a separate element in a list like [‘word1’, ‘word2’, … ‘wordn’] for n words. This makes my life easy for two reasons. First, I simply joined the array with a space between each word, creating one long string (the outputted text) with spaces between each element in the array (words). Second, it made it easy to just remove the part of the array that contains Aozora metadata before creating that string. Again, this is quick and dirty, but from examining the files I noted that the metadata always comes at the end of the file and begins with the word 底本 (‘source text’). Remove that word and everything after it, and then you have a metadata-free file.

Write this resulting text into a plain text file, and you have a non-ruby, non-HTML, metadata-free, whitespace-delimited Aozora text! Although you have to still download all the Aozora files individually and then do what you will with the resulting individual text files, it’s an easy way to pre-process this text and get it ready for tool-based (and also your-own-program-based) text analysis.

I plan to put the script on GitHub for your perusal and use (and of course modification) but for now, check it out on my Japanese Text Analysis research guide at Penn.

#dayofDH Japanese digital resource research guides

Another “digital” thing I’ve been doing that relates to the “humanities” (but is it even remotely DH? I don’t know), is the creation of research guides for digital resources in Japanese studies of all kinds, with a focus on Japanese-language free websites and databases, and open-access publications.

So far, I’ve been working hard on creating guides for electronic Japanese studies resources, and mobile apps easily accessible in the US for both Android and iOS that relate to Japanese research or language study. The digital resources guide covers everything from general digital archives and citation indexes to literature, art, history, pop culture, and kuzushiji resources (for reading handwritten pre- and early modern documents). They range from text and image databases to dictionaries and even YouTube videos and online courseware for learning classical Japanese and how to read manuscripts.

This has been a real challenge, as you can imagine. Creating lists of stuff is one thing (and is one thing I’ve done for Japanese text analysis resources), but actually curating them and creating the equivalent of annotated bibliographies is quite another. It’s been a huge amount of research and writing – both in discovery of sources, and also in investigating and evaluating them, then describing them in plain terms to my community. I spent hours on end surfing the App and Play Stores and downloading/trying countless awful free apps – so you don’t have to!

It’s especially hard to find digital resources in ways other than word of mouth. I find that I end up linking to other librarians’ LibGuides (i.e. research guides) often because they’ve done such a fantastic job curating their own lists already. I wonder sometimes if we’re all just duplicating each other’s efforts! The NCC has a database of research guides, yes, but would it be better if we all collaboratively edited just one? Would it get overwhelming? Would there be serious disagreements about how to organize, whether to include paid resources (and which ones), and where to file things?

The answer to all these questions is probably yes, which creates problems. Logistically, we can’t have every Japanese librarian in the English-speaking world editing the same guide anyway. So it’s hard to say what the solution is – keep working in our silos? Specialize and tell our students and faculty to Google “LibGuide Japanese” + topic? (Which is what I’ve done in the past with art and art history.) Search the master NCC database? Some combination is probably the right path.

Until then, I will keep working on accumulating as many kuzushiji resources as I can for Penn’s reading group, and updating my mobile app guide if I ever find a decent まとめ!