How might we prototype a bridge between the reading experiences of paper books and their digital version?


Have you ever found the dilemma of whether to read a book on a print copy or a digital device?

Better digital reading devices aim to provide similar experiences to reading print books while delivering many other traits: quick look-up, easy annotation, and synced progress between devices. However, it probably could never replicate the tangible touch of flipping through pages on a print copy, which most readers still prefer.


What if we can have the best of both worlds by integrating digital annotation to physical touch?

Existing "Bridges"

We are not the first to ask this question. Several existing products are designed to digitize print books and physical annotation with different approaches.

Hanvon E-Pen
Hanvon E-Pen

A progressive OCR text scanner for print books, featuring vocabulary look-up and excerpt extraction.

I personally own a Hanvon E-Pen, mainly to look up words in my non-native languages. Using it has been quite pleasing, but there are two significant shortcomings:

Switching Back & Forth
between a tiny pen screen and print copies
Lack of Connectvity
to export the data to more capable devices
Livescribe 3 Smartpen
Livescribe 3 Smartpen

A real-time text scanner for handwriting featuring note synchronization to other digital devices.

Although Livescribe integrates better with other devices, it deviates much from the use case of reading but more on the notetaking side.

These defects of existing "bridges" urged us think out of the box:

The limited options for interfaces and lack of connectivity are hindering the user experience in these attempts.

Imagining AR

Our team happened to have our hands on a Microsoft Hololens. What if we utilize augmented reality to smooth out any bumps on the "bridge" in existing solutions.

Microsoft Hololens
Microsoft Hololens

We settled on two fundamental interactions to build a seamless hybrid reading experience:

AR Quick Look-Up

By overlaying information on top of a paper book, we could mitigate the hassle of going back and forth between devices – just wear Hololens while reading. To look up an obscure word, just tap your finger on it!

Holo Reader: AR Quick Look-Up
Holo Reader: AR Quick Look-Up

Cross-Platform Annotation Syncing

Handwritten annotations will be parsed and digitized to be accessible across digital reading devices, such as Kindle.

Holo Reader: Annotating Syncing
Holo Reader: Annotating Syncing

Concept Video

Introducing Holo Reader, an AR application that bridges your digital and physical reading experience.


These are technically challenging ideas, and we managed to implement a part of them during a short hackathon at NYU.

Holo Reader: In Development
Holo Reader: In Development

The client side of our prototype is built with Unity with Mixed Reality Toolkit and Vuforia. On top of that, we developed a back-end server with Flask using Python to handle data transfer and image processing.

Holo Reader: Data Flow
Holo Reader: Data Flow

Using Vuforia, the app constantly looks for user's fingertips and the word on top of it. After sending back to our server, the app receives the word definition to overlay on top of book.

For further details, check out our
→ Github Repo.

↓ more projects ↓