Bloomreach XM Performance Test Suite

The Bloomreach XM performance test suite evaluates CMS editorial performance in a running Bloomreach Content environment. It combines browser-based UI latency tests and API load tests, allowing you to measure CMS responsiveness, multi-user editing behavior, and detect performance regressions before release or go-live.

You can use the suite in development, test, staging, and acceptance environments, regardless of whether Bloomreach Content is hosted in Bloomreach Cloud or self-hosted. For results that reflect production, use an environment with production-like resources. Do not run destructive setup or load tests against a live production CMS.

What the Suite Tests

The suite provides two types of tests:

Test type	Technology	What it measures
UI latency tests	Playwright and Chromium	User-perceived CMS latency for editorial workflows such as navigation, editing, saving, publishing, and concurrent editing
API load tests	Gatling	Backend Content Service performance for authenticated document editing flows, excluding browser rendering overhead

The generated report combines UI and API results into a single dashboard with latency metrics, scaling data, and optional baseline comparisons.

Where to Run the Tests

Run performance tests in an environment that closely matches your production setup:

• Use a dedicated test or acceptance environment.

• Allocate production-like resources when possible.

• Ensure the test runner can access the CMS application over the network.

• Do not store important editorial content under /content/documents/_brxmperf; the setup tool manages this folder and may recreate it.

For Bloomreach Cloud, mark an environment as Acceptance to access production-like resources without testing on the live production environment. For self-hosted deployments, use a staging or acceptance environment that mirrors production topology, database, and infrastructure settings.

Prerequisites

Before running the suite, ensure you have:

• A running Bloomreach Content instance.

• Java 17 or later.

• Maven 3.8 or later.

• Node.js 20 or later and npm 10 or later.

• Access to the CMS.

• Network access from the machine or CI runner to the CMS and to the SUT REST service at /cms/ws/qa/jcr/.

• Administrative credentials for fixture setup and cleanup.

• The performance test dependencies deployed to the CMS. See "Add the performance test dependencies to the CMS".

Add the performance test dependencies to the CMS

Note: this setup applies to 17.1.0 and later. Earlier releases do not publish bloomreach-xm-perf-cms-dependencies.

The suite needs two components deployed inside the CMS web application:

• A namespace and content type for the generated test documents.

• The SUT REST service at /cms/ws/qa/jcr/, which the setup tool uses to create and remove fixtures.

A single artifact, bloomreach-xm-perf-cms-dependencies, supplies both. Add it to the CMS module inside a dedicated Maven profile, so that no performance test tooling is packaged into a production WAR:

<profiles>
  <profile>
    <id>perf-testing</id>
    <dependencies>
      <dependency>
        <groupId>com.bloomreach.xm</groupId>
        <artifactId>bloomreach-xm-perf-cms-dependencies</artifactId>
        <version>${hippo.release.version}</version>
        <type>pom</type>
      </dependency>
    </dependencies>
  </profile>
</profiles>
copy

Build and deploy the CMS with the profile active:

mvn clean package -Pperf-testing
copy

The <type>pom</type> element is required. bloomreach-xm-perf-cms-dependencies is an aggregator POM rather than a JAR, and Maven does not resolve it without it.

After the CMS starts, verify both components:

• /hippo:namespaces/brxmperf exists in the repository console.

• curl -u <admin-user>:<admin-password> <cms-base-url>/cms/ws/qa/jcr/ returns HTTP 200.

The aggregator pulls in:

• bloomreach-xm-perf-bootstrap, which registers the brxmperf namespace, the brxmperf:brxmperfdoc document type, and editor templates for title, introduction, rich text content, and date fields.

• The SUT REST server, which exposes /cms/ws/qa/jcr/.

Omit the profile for production builds. Without -Pperf-testing, neither component is added to the WAR.

Download the Test Suite

Download the distribution archive from the enterprise Maven repository:

• Browse and download manually: https://maven.bloomreach.com/#browse/browse:enterprise-releases:com%2Fbloomreach%2Fxm%2Fbloomreach-xm-perf-test-suite

• Or fetch with Maven:

  mvn dependency:copy \
    -Dartifact=com.bloomreach.xm:bloomreach-xm-perf-test-suite:<version>:zip \
    -DoutputDirectory=. \
    -DremoteRepositories=bloomreach-maven2-enterprise::::https://maven.bloomreach.com/repository/maven2-enterprise
  

Extract the archive. This creates an xm-perf-tests directory containing the test runner.

Known issue: archives up to and including 17.2.0 fail immediately with "Child module ... does not exist". See "Maven reports that a child module does not exist" for the one-line workaround.

Prepare the Test Runner

Navigate to the performance test suite directory:

cd xm-perf-tests

Install the UI test dependencies before running the full suite:

cd ui-latency-tests
npm ci
npx playwright install chromium
cd ..

Run a Full Performance Test

To run the full test lifecycle, use:

./run-tests.sh --full --url=http://localhost:8080

This command performs these steps:

1. Installs test fixtures.

2. Runs UI latency tests at 1, 3, and 5 concurrent users.

3. Runs API load tests.

4. Cleans up test fixtures.

5. Generates the combined HTML report.

To test a remote environment, specify the CMS base URL:

./run-tests.sh --full --url=https://cms.example.com

If the CMS uses non-default administrative credentials for the SUT REST service, set them as environment variables:

SUT_ADMIN_USER=<admin-user> \
SUT_ADMIN_PASSWORD=<admin-password> \
./run-tests.sh --full --url=https://cms.example.com

Test Fixtures and perf-env.json

During setup, the suite creates a controlled set of CMS users, folders, documents, and a manifest file named perf-env.json.

By default, setup creates:

• Five author users named perf-author-01, perf-author-02, etc.

• Five editor users named perf-editor-01, perf-editor-02, etc.

• One test document per configured user.

• A test content folder at /content/documents/_brxmperf.

• A temporary folder at /content/documents/_brxmperf/temp-created for UI-created documents.

The generated perf-env.json file contains:

• Installation metadata, including the target CMS URL and content size.

• The resources created during setup, used for cleanup.

• The users and documents referenced by the UI and API tests.

Keep perf-env.json in the performance test suite root directory when running UI and API tests separately.

Common Commands

Run these commands from the performance test suite root directory.

Command	Description
./run-tests.sh --full --url=http://localhost:8080	Run setup, UI tests, API tests, cleanup, and report generation
./run-tests.sh --setup --url=http://localhost:8080	Install fixtures only
./run-tests.sh --status --url=http://localhost:8080	Check CMS connectivity, the bootstrap namespace, and fixture status
./run-tests.sh --cleanup --url=http://localhost:8080	Remove fixtures using perf-env.json
./run-tests.sh --ui --url=http://localhost:8080	Run the UI latency suite when fixtures already exist
./run-tests.sh --api --url=http://localhost:8080	Run the API load tests when fixtures already exist
./run-tests.sh --list-baselines	List stored performance baselines

To control the number of test users and documents created during setup, use --users=N:

./run-tests.sh --full --url=http://localhost:8080 --users=10

The default UI suite runs with 1, 3, and 5 concurrent users. To specify custom UI concurrency levels, run the UI suite directly from ui-latency-tests:

npm run test:suite -- --users=1,3,5,10

UI Latency Tests

The UI latency tests use Playwright and Chromium to execute real CMS editorial workflows. The tests wait for the CMS UI to become idle before recording timings, so the metrics reflect user-perceived latency.

The UI suite covers:

• Folder and document navigation.

• Document edit and save operations.

• Author and editor document actions.

• Multi-step editorial workflows.

• Concurrent editing scenarios.

Environment variables for UI tests:

Variable	Default	Description
CMS_BASE_URL	http://localhost:8080	CMS base URL
CMS_CONTEXT_PATH	/cms	CMS web application context path
HEADLESS	true	Set to false to display browser windows
WARMUP_ITERATIONS	1	Warmup iterations before measurement
MEASURED_ITERATIONS	10	Measured iterations for atomic operations
JOURNEY_ITERATIONS	3	Measured iterations for journey tests
UI_IDLE_TIMEOUT	30000	Timeout in milliseconds while waiting for UI idle
CONCURRENT_USERS	1	Number of browser sessions for direct concurrent runs
MAX_USERS	5	Maximum users for the degradation curve test

Examples:

cd ui-latency-tests

# Run the single-user suite.
npm test

# Run the standard multi-user suite.
npm run test:suite

# Show browser windows for debugging.
HEADLESS=false npm run test:suite

# Increase the UI idle timeout for slower environments.
UI_IDLE_TIMEOUT=60000 npm test

API Load Tests

API load tests use Gatling to exercise the Channel Manager Content Service at /cms/ws/content/. These tests bypass browser rendering and focus on backend and repository performance during document editing.

Each virtual user performs the following steps:

1. Authenticate with the CMS.

2. Fetch document type metadata.

3. Lock a document.

4. Update one or more fields.

5. Save the draft variant.

6. Release the lock.

The tests use documents from perf-env.json when available. For accurate results, create at least as many test documents as concurrent API users to avoid lock contention.

Examples:

cd api-load-tests

# Verify that test documents are available.
mvn gatling:test -Dgatling.simulationClass=com.bloomreach.xm.performance.DiscoverySimulation

# Run a one-user smoke test.
mvn gatling:test -Dgatling.simulationClass=com.bloomreach.xm.performance.SmokeSimulation

# Run the default Content Service simulation with custom load.
mvn gatling:test -Dusers=5 -DrampDuration=10 -DtestDuration=180

Common Gatling properties:

Property	Default	Description
baseUrl	http://localhost:8080	CMS base URL if no perf-env.json is loaded
users	1	Number of concurrent virtual users
rampDuration	10	Seconds to ramp up to full load
testDuration	120	Seconds to run at steady state
maxResponseTime	5000	Maximum response time assertion in milliseconds
minSuccessRate	95	Minimum successful request percentage

Reports

After a full run, open:

reports/index.html

The combined dashboard includes:

• An executive summary with an overall verdict.

• UI latency heatmaps for each user count.

• Metrics for journeys, journey operations, and standalone operations.

• Scaling information across user counts.

• API throughput and latency metrics.

• Baseline deltas when a baseline is available.

Detailed reports are written to:

Report	Location
Combined dashboard	reports/index.html
Machine-readable summary	reports/summary.json
Playwright report	ui-latency-tests/playwright-report/index.html
Gatling report	api-load-tests/target/gatling//index.html

Baseline Comparison

Baselines allow you to detect performance regressions between releases.

To save a baseline after a known-good run:

./run-tests.sh --full --url=http://release-server:8080 --save-baseline=release-17.0

To compare a later run against a named baseline:

./run-tests.sh --full --url=http://test-server:8080 --compare=release-17.0

To list available baselines:

./run-tests.sh --list-baselines

Baseline verdict thresholds:

Verdict	Meaning
Pass	No more than 5% slower than the baseline, or faster
Warning	More than 5% and up to 10% slower than the baseline
Regression	More than 10% slower than the baseline

Cleanup

Use cleanup after manual setup or partial test runs:

./run-tests.sh --cleanup --url=http://localhost:8080

Cleanup reads perf-env.json, deletes the generated documents, folders, and users, and removes the manifest file. If cleanup cannot find the manifest, provide it explicitly through the setup CLI:

cd perf-setup
mvn exec:java -Dexec.args="cleanup --manifest=../perf-env.json --base-url=http://localhost:8080"

Troubleshooting

perf-env.json is missing

Run setup before running UI or API tests separately:

./run-tests.sh --setup --url=http://localhost:8080

The status check cannot reach the CMS

Verify that the CMS is running and accessible from the test runner:

curl http://localhost:8080/cms/

For remote environments, check network routing, TLS configuration, and the CMS base URL passed with --url.

Maven reports that a child module does not exist

Distribution archives from 16.9.0 through 17.2.0 declare modules in xm-perf-tests/pom.xml that the archive does not contain:

[ERROR] Child module .../xm-perf-tests/perf-dependencies of .../xm-perf-tests/pom.xml does not exist
[ERROR] Child module .../xm-perf-tests/perf-cms-dependencies of .../xm-perf-tests/pom.xml does not exist

Maven stops while building the project model, so every run-tests.sh action fails, on every platform. Releases 17.1.0 and later report both modules; earlier releases report only perf-dependencies.

As a workaround, remove the offending lines from the <modules> section of xm-perf-tests/pom.xml:

<module>perf-dependencies</module>
<module>perf-cms-dependencies</module>
copy

Both are dependency-only POMs that the suite resolves from the Maven repository rather than building locally, so removing them from the module list has no further effect. Later releases ship the archive with all declared modules present.

Setup fails with "No message body reader has been found"

The failure appears while validating prerequisites:

Install failed: Cannot connect to SUT at http://localhost:8080: No message body reader has been found
for class com.onehippo.cms7.sut.rest.api.error.RestException, ContentType: text/html;charset=UTF-8

The SUT REST service is not deployed, so the CMS returns an HTML error page where the client expects JSON. The message names a content type rather than the missing service, so it is easy to misread as a connectivity problem.

Add bloomreach-xm-perf-cms-dependencies to the CMS module and redeploy. See "Add the performance test dependencies to the CMS". Confirm the fix with:

curl -u <admin-user>:<admin-password> <cms-base-url>/cms/ws/qa/jcr/
copy

A working deployment returns HTTP 200 and a JSON body. An HTTP 404 means the service is still missing.

Setup fails because the namespace is missing

Confirm that bloomreach-xm-perf-cms-dependencies is deployed in the CMS, that the CMS was built with the perf-testing profile active, and that /hippo:namespaces/brxmperf exists in the repository console.

UI tests time out while waiting for the CMS to become idle

Increase the UI idle timeout:

UI_IDLE_TIMEOUT=60000 npm test

If the timeout persists, run the tests with browser windows visible and check for dialogs, overlays, or long-running CMS requests:

HEADLESS=false npm run test:suite