Focus, Shoot, Extract: Mobile App to Book Metadata

AKA: Sending screenshots from your custom mobile app to KNIME Server

Building custom mobile apps is a real money mover for businesses. They promise faster access, deeper engagement, and an intuitive interface for a more enjoyable user experience.

Let's imagine you're in the book-selling business and you've built a mobile app that will help you generate demand by anticipating your customers' reading preferences. This app encourages readers to share their favorite books. All the readers have to do is take a screenshot of the book through the app. Now, using the book metadata, the book-seller can train a recommendation engine that matches the readers' preferences with what they might like to purchase next.

But how can you make sure you’re collecting the customer data from the app efficiently enough to offer a better service? A good solution could be to build a web service hosted on a server that enables the real-time transmission, extraction, and storage of book metadata. If you don't have much experience in building and using web services or other APIs, this sounds like an overwhelming task.

In this article we'd like to show how simple it can be by using KNIME as a low-code solution.

We’re going to look at how our bookseller’s iPhone app will connect with a KNIME Server. In particular we will use a KNIME workflow, running on the KNIME Server, to fetch a screenshot of the book from the iPhone, extract the book metadata, and store this in a central repository – an SQLite database. See how this maps out in the overview below:

The steps in the process represented above are as follows:

1. User takes a screenshot of a book edition notice (containing an ISBN identifier) via a custom iPhone app.

2. Screenshot is converted into a base64-encoded image.

3. The iPhone app sends the encoded image to a web service stored on KNIME Server in the form of a Request object.

4. The web service on KNIME Server:

• Acquires the screenshot image.
• Adds a timestamp with execution date.
• Detects ISBN identifier via OCR.
• Extracts book metadata (i.e., title, author, book cover image, etc.) via Google Books API.
• Writes metadata, book cover, and timestamp to a data repository (e.g., SQLite database).
• Returns book metadata as a JSON Response object.

Without diving deeply into the technical details mobile apps can be of three types:

Web apps, which mimic the content of a website but are optimized to make them mobile-friendly, hybrid apps, which are developed to be operated on more than one platform, and native apps, which are built specifically for mobile devices, usually for either iOS or Android, and can be downloaded from the app store.

Building a native app, we need to face the everlasting dichotomy between fundamentally different operating systems (iOS vs. Android) and choose one over the other – which has pros and cons. For the purpose of this article, our “Friendly Library” mobile app is for iOS devices using Xcode. Basically, this app:

• Takes a screenshot.
• Encodes the image to a base64 format, and resizes it.
• Submits a POST Request containing the encoded image to a web service located on a KNIME Server instance.

The code that we used for this app is shown in Fig. 2 and is downloadable for free from Github.

Images are usually stored in binary format. However, since web services prefer text in Request and Response objects, we have to convert the image to a base64 format. Base64 encoding is a transformation to encode binary data in ASCII text. It's primarily used to store or transfer images, audio files, and other media online. After base64 encoding, the original image is represented with a 64-bit encoded string.

Note. Server URL and credentials (username and password) need to be updated. For security reasons, it is best not to hard-code the credentials and instead store them on the device and access them when needed. For this reason, the code above is just an example and should not be used in production.

The next step is to connect our app to a web service on KNIME Server to process, retrieve and store book metadata.

The web service to interface with the iOS app is developed on KNIME Analytics Platform and then deployed on KNIME Server.

Any workflow running on a KNIME Server can be automatically accessed as a web service. Therefore, all we need to do to create the web service is to add the nodes for the Request and the Response objects. These nodes are: the Container Input (JSON) to collect the Request object, and the Container Output (JSON) to export the Response object (Fig. 3).

Now that the web service has been created, next we have to fill the body of the service with the required operations:

• Add a timestamp with execution date.
• Import the base64-encoded image from the JSON Request object and add the timestamp.
• Transform the image from the base64 format to PNG.
• Capture ISBN identifier via OCR to retrieve book metadata using Google Books API
• Write metadata to a database and return it as a JSON Response object.

You can see each of these steps in the workflow (which will be our web service) in the workflow below (Fig. 3).

The timestamp is created by the Create Date&Time Range node. It forces the creation of only one Date&Time object and uses the current execution date and time as starting point. It is a trick we often use to generate timestamps.

In the “Add execution date” metanode, the key node is the JSON To Table. This one is responsible for extracting the 64-bit image from the JSON-formatted Request object. Then, the Column Appender node appends the timestamp to the data obtained so far.

The next step is to convert the image to a PNG format for processing. The metanode “Decode base64 image” decodes the image from the base64 format to a PNG file. Notice that the decoding function is not available as a KNIME native node yet. However, thanks to the KNIME Python Integration, we were able to use the function below in a Python Script node:

base64.decodebytes()

The “Extract ISBN” metanode performs OCR on the book edition notice using the Tess4J node, extracts the book’s ISBN identifier with the String Manipulation node, and verifies its integrity via the Rule Engine node. Notice that the Tess4J Integration currently does not work for Mac users.

Now, in order to retrieve book metadata, such as title, subtitle, authors, publishedDate, description, ISBN 10, ISBN 13, pageCount, averageRating, ratingsCount, language, and book cover, we need to query the Google Books API using the GET Request node:

https://www.googleapis.com/books/v1/volumes?q=isbn:<isbn number>

We send in the ISBN identifier and receive in response all book metadata associated with that ISBN. Book metadata, cover, and timestamp are then sent to the following metanode.

After processing, we want to store the retrieved information in a database. We chose an SQLite database just for portability of the example, but it could be any other database. KNIME offers a wide variety of Connector nodes for many flavors of databases. In the “Store metadata in database” metanode, we write retrieved data in an SQLite database using the SQLite Connector node and other DB nodes to create a dedicated table, remove duplicates, and write unique entries.

Yes, it was that simple. Download the workflow for free from the KNIME Hub and try it out yourself.

An overview of the complete application, including the screenshot taken with an iPhone, the web service to extract book metadata, and finally the web service Response can be seen in Fig. 4.

The whole application was developed in two steps:

1. An iPhone app to capture and encode an image, and transfer it to a web service as a Request object.

2. A web service running on a KNIME Server to acquire the image, process it, retrieve metadata and store it in a database.

The whole application required just one line of Python code to decode a base64 encoded string to a PNG file. Besides that, the entire web service was visually programmed using KNIME Analytics Platform.

Forget cumbersome processes, dependency conflicts and obscure data flows that are hard to debug. With a bunch of nodes and KNIME Server, building your custom web service is easy, simple to customize and to use in production. What is your use case?