Vector Store Maintenance Groovy Scripts

Overview

The AI module includes a base updater script to support vector store maintenance tasks. You can extend this script to implement custom Groovy scripts for managing vector embeddings. The base updater class is com.bloomreach.xm.ai.service.impl.vector.mgmt.VectorStoreUpdater. It provides two primary methods:

addVectorsFrom(handle)
Generates embeddings for a given handle using the configured embedding model and queues them for ingestion into the vector store. In live mode, the published variant is used; in preview mode, the unpublished variant is used. Ingestion filters—such as type, include-dirs, and exclude-dirs—are applied.
removeVectorsFor(handle)
Queues a request to remove vectors for the specified handle. No filters are applied to removals.

The base class also exposes the ingestion filter and queue for use in subclasses:

protected IngestionFilter ingestionFilter;
protected IngestionQueue ingestionQueue;

These methods operate directly on handle nodes, which makes them suitable for Groovy scripts that already support node iteration and searching.

Info: For more details, see Updater Scripts.

Implementations

The AI module provides two sample updater implementations. After installing the AI module, you can find these scripts in the Registry section of the CMS Updater Editor.

Hint: Use these examples as provided, or modify them to create more advanced updaters.

Hint: You can safely delete these scripts. The AI module does not depend on them.

VectorStoreExampleIngestor
This script queues documents for ingestion into the vector store. Select handles to ingest using any supported Groovy script node retrieval method.

class VectorStoreExampleIngestor extends VectorStoreUpdater {

    boolean doUpdate(Node node) {
        if (node.isNodeType(NT_HANDLE)) {
            log.debug "Queuing up node ${node.path} for addition to the vector store"
            if (!visitorContext.dryRun) {
                addVectorsFrom(node)
                return true
            }
        }
        return false
    }

VectorStoreExampleCleaner
This script queues documents for removal from the vector store. Select handles to remove using any supported Groovy script node retrieval method.

class VectorStoreExampleCleaner extends VectorStoreUpdater {

    boolean doUpdate(Node node) throws RepositoryException {
        if (node.isNodeType(NT_HANDLE)) {
            log.debug "Queuing up node ${node.path} for removal from vector store"
            if (!visitorContext.dryRun) {
                removeVectorsFor(node)
                return true
            }
        }
        return false
    }

Using the ingestionFilter

The base updater automatically applies the configured ingestion filter when you use addVectorsFrom. The filter is also available in your custom updaters and provides three methods:

ingestionFilter.isIndexableType.test(documentType)
ingestionFilter.isUnderIncludedDir.test(handlePath)
ingestionFilter.isUnderExcludedDir.test(handlePath)

Use Cases

Common scenarios for using these scripts include:

A document type was removed from the list of enabled types
Documents of this type remain indexed in the vector store. Run VectorStoreExampleCleaner with a query that locates all documents of the removed type to remove them from the store.
A folder with indexed content was removed from included-dirs or added to excluded-dirs
Documents under this folder and its subfolders must be removed from the vector store. Use VectorStoreExampleCleaner to process these documents.
A folder was added to included-dirs or removed from excluded-dirs
Documents under this folder and its subfolders must be added to the vector store. Use VectorStoreExampleIngestor to ingest these documents.
New content was added via import
Use VectorStoreExampleIngestor to locate and enqueue the new content for ingestion.
The vector store is out of sync with content
If ingestion failed or documents were not indexed as expected, use VectorStoreExampleIngestor to re-index the affected content.