Lucene Index Export Service Explained

Info: The Lucene Index Export Service requires a standard or premium Bloomreach Content license. Contact Bloomreach for details.

Info: Available since Hippo CMS 12.1.0.

Overview

This page explains how the Lucene Index Export Service operates in Bloomreach Content.

Purpose

The Lucene Index Export Service enables you to export the Lucene index from a running production repository. This feature is provided by the optional Lucene Index Export add-on. After you add the add-on to your project, it exposes a Repository JAX-RS service for on-demand Lucene index export.

How the Lucene Index Export Service Works

When you export the Lucene index from a running repository, the service generates a ZIP file with a structure similar to the following:

+ _1
+ _2
+ _3
+ ...
+ indexRevision.properties

The indexRevision.properties file records two properties:

indexRevisionBefore: The repository revision at the start of the export.
indexRevisionAfter: The repository revision at the end of the export.

Because the repository can advance during the export process, the revision may change between the start and end of the export. The exported index is guaranteed to represent a state between these two revisions. When you initialize a new repository using this exported index, the repository removes any documents added to the index after indexRevisionBefore up to indexRevisionAfter. During the remainder of the startup, the repository updates the index from the start revision to the current global revision. The newer the exported index, the less time is required for the repository to become fully up-to-date.

Example: Lucene Index Export with Revision Numbers

Assume you have a repository cluster with two nodes. The node performing the export is named node1. At the start of the export, the database contains the following information:

mysql> SELECT * FROM REPOSITORY_GLOBAL_REVISION;
+-------------+
| REVISION_ID |
+-------------+
|     2559    |
+-------------+

mysql> SELECT * FROM REPOSITORY_LOCAL_REVISIONS;
+---------------+-------------+
| JOURNAL_ID    | REVISION_ID |
+---------------+-------------+
| node1         |     2500    |
| node2         |     2559    |
+---------------+-------------+

In this scenario, node1 is 59 revisions behind node2. While node1 performs the Lucene export, the repository revision may continue to advance. At the end of the export, the state could be:

indexRevision.properties 
   - indexRevisionBefore = 2500
   - indexRevisionAfter  = 2516

And the local revisions table may look like:

mysql> SELECT * FROM REPOSITORY_LOCAL_REVISIONS;
+----------------------------------------+-------------+
| JOURNAL_ID                             | REVISION_ID |
+----------------------------------------+-------------+
| node1                                  |     2559    |
| node2                                  |     2559    |
| _HIPPO_EXTERNAL_REPO_SYNC_index-backup |     2500    |
+----------------------------------------+-------------+

During the export, the revision advanced from 2500 to 2516. After the export, node1 caught up to revision 2559. The REPOSITORY_LOCAL_REVISIONS table now includes an entry:

_HIPPO_EXTERNAL_REPO_SYNC_index-backup = 2500

This entry records the revision at the start of the index export. See below for details on its purpose.

Starting a New Repository with the Exported Index

When you use the exported Lucene index to initialize a new repository node (for example, node3), and the directory {storageRoot}/workspaces/default/index contains an indexRevision.properties file, the following steps occur during startup:

The repository uses the present Lucene index as the starting point.
The system fetches all changes from the REPOSITORY_JOURNAL table between revision 2500 (exclusive) and 2516 (inclusive) and removes these changes from the Lucene index. If some changes (such as revisions 2512 to 2516) were not yet reflected in the index, no action is required for those.
The new repository node (node3) is added to the REPOSITORY_LOCAL_REVISIONS table with REVISION_ID = 2500.
The indexRevision.properties file is deleted from {storageRoot}/workspaces/default/index.
The repository completes its normal startup, updating its index to the current global revision.

After startup, assuming the global revision has not advanced, the local revisions table may look like:

+----------------------------------------+-------------+
| JOURNAL_ID                             | REVISION_ID |
+----------------------------------------+-------------+
| node1                                  |     2559    |
| node2                                  |     2559    |
| node3                                  |     2559    |
| _HIPPO_EXTERNAL_REPO_SYNC_index-backup |     2500    |
+----------------------------------------+-------------+

Purpose of `_HIPPO_EXTERNAL_REPO_SYNC_index-backup`

The _HIPPO_EXTERNAL_REPO_SYNC_index-backup entry stores the revision ID from the start of the last successful Lucene export. After each successful export, this revision ID is updated. This mechanism prevents repository maintenance operations from cleaning the REPOSITORY_JOURNAL table beyond the revision required by the latest Lucene export. If the journal is cleaned past this revision, the Lucene export ZIP becomes unusable because the repository cannot update the index to the current state.

If you attempt to use a Lucene export ZIP where the indexRevisionBefore property is older than the earliest revision in the REPOSITORY_JOURNAL table, you cannot use that export to start a new repository. In this case, repository startup fails with an error similar to:

"Required start revision '' does NOT exist any more in the Journal table (oldest journal table record has revision '') implying the index cannot be correctly updated. Remove the index and restart to trigger a complete new index built or provide a newer index export."

To resolve this error, either use a new Lucene export or start the repository without an existing Lucene export.