Bloomreach XM Performance Test Suite

The Bloomreach XM performance test suite evaluates CMS editorial performance in a running Bloomreach Content environment. It combines browser-based UI latency tests and API load tests, allowing you to measure CMS responsiveness, multi-user editing behavior, and detect performance regressions before release or go-live.

You can use the suite in development, test, staging, and acceptance environments, regardless of whether Bloomreach Content is hosted in Bloomreach Cloud or self-hosted. For results that reflect production, use an environment with production-like resources. Do not run destructive setup or load tests against a live production CMS.

What the Suite Tests

The suite provides two types of tests:

Test type	Technology	What it measures
UI latency tests	Playwright and Chromium	User-perceived CMS latency for editorial workflows such as navigation, editing, saving, publishing, and concurrent editing
API load tests	Gatling	Backend Content Service performance for authenticated document editing flows, excluding browser rendering overhead

The generated report combines UI and API results into a single dashboard with latency metrics, scaling data, and optional baseline comparisons.

Where to Run the Tests

Run performance tests in an environment that closely matches your production setup:

• Use a dedicated test or acceptance environment.

• Allocate production-like resources when possible.

• Ensure the test runner can access the CMS application over the network.

• Do not store important editorial content under /content/documents/_brxmperf; the setup tool manages this folder and may recreate it.

For Bloomreach Cloud, mark an environment as Acceptance to access production-like resources without testing on the live production environment. For self-hosted deployments, use a staging or acceptance environment that mirrors production topology, database, and infrastructure settings.

Clustered Environments

As of 17.3.0 and 16.9.5, fixture setup works against environments that run several CMS instances behind a load balancer. The setup tool pins itself to one CMS instance by replaying the load balancer's SERVERID session-affinity cookie. It therefore never reads content that another instance has not yet synchronized.

Prerequisites

Before running the suite, ensure you have:

• A running Bloomreach Content instance.

• Java 21 or later.

• Maven 3.9 or later.

• Node.js 20 or later and npm 10 or later.

• Access to the CMS.

• Network access from the machine or CI runner to the CMS and to the SUT REST service at /cms/ws/qa/jcr/.

• Administrative credentials for fixture setup and cleanup.

• The performance test dependencies deployed to the CMS. See "Add the performance test dependencies to the CMS".

The suite runs on macOS, Linux, and Windows.

Add the performance test dependencies to the CMS

Note: this setup applies to 17.1.0 and later. Earlier releases do not publish bloomreach-xm-perf-cms-dependencies.

The suite needs two components deployed inside the CMS web application:

• A namespace and content type for the generated test documents.

• The SUT REST service at /cms/ws/qa/jcr/, which the setup tool uses to create and remove fixtures.

A single artifact, bloomreach-xm-perf-cms-dependencies, supplies both. Add it to the CMS module inside a dedicated Maven profile, so that no performance test tooling is packaged into a production WAR:

<profiles>
  <profile>
    <id>perf-testing</id>
    <dependencies>
      <dependency>
        <groupId>com.bloomreach.xm</groupId>
        <artifactId>bloomreach-xm-perf-cms-dependencies</artifactId>
        <version>${hippo.release.version}</version>
        <type>pom</type>
      </dependency>
    </dependencies>
  </profile>
</profiles>

Build and deploy the CMS with the profile active:

mvn clean package -Pperf-testing

The <type>pom</type> element is required. bloomreach-xm-perf-cms-dependencies is an aggregator POM rather than a JAR, and Maven does not resolve it without it.

After the CMS starts, verify both components:

• /hippo:namespaces/brxmperf exists in the repository console.

• curl -u <admin-user>:<admin-password> <cms-base-url>/cms/ws/qa/jcr/ returns HTTP 200.

The aggregator pulls in:

• bloomreach-xm-perf-bootstrap, which registers the brxmperf namespace, the brxmperf:brxmperfdoc document type, and editor templates for title, introduction, rich text content, and date fields.

• The SUT REST server, which exposes /cms/ws/qa/jcr/.

Omit the profile for production builds. Without -Pperf-testing, neither component is added to the WAR.

Download the Test Suite

Download the distribution archive from the enterprise Maven repository:

• Browse and download manually: https://maven.bloomreach.com/#browse/browse:enterprise-releases:com%2Fbloomreach%2Fxm%2Fbloomreach-xm-perf-test-suite

• Or fetch with Maven:

  mvn dependency:copy \
    -Dartifact=com.bloomreach.xm:bloomreach-xm-perf-test-suite:<version>:zip \
    -DoutputDirectory=. \
    -DremoteRepositories=bloomreach-maven2-enterprise::::https://maven.bloomreach.com/repository/maven2-enterprise
  

Extract the archive. This creates an xm-perf-tests directory containing the test runner.

Known issue: archives from 16.9.0 up to and including 16.9.4, and 17.0.0 up to and including 17.2.0, fail immediately with "Child module ... does not exist". This is fixed in 16.9.5, 17.2.1, and 17.3.0. For older archives, see "Maven reports that a child module does not exist" for the one-line workaround.

Prepare the Test Runner

Navigate to the performance test suite directory:

cd xm-perf-tests

Install the UI test dependencies before running the full suite:

cd ui-latency-tests
npm ci
cd ..

The test runner installs the Chromium browser automatically before the UI tests start, and skips this step when Chromium is already present. To install it in advance, for example on a CI image, run npx playwright install chromium in ui-latency-tests.

Run a Full Performance Test

To run the full test lifecycle, use:

./run-tests.sh --full --url=http://localhost:8080

On Windows, use run-tests.bat with the same options:

run-tests.bat --full --url=http://localhost:8080

This command performs these steps:

1. Installs test fixtures.

2. Runs UI latency tests at 1, 3, and 5 concurrent users.

3. Runs API load tests.

4. Cleans up test fixtures.

5. Generates the combined HTML report.

If the UI tests fail, the API load tests are skipped, because the CMS or its fixtures are then already suspect. Cleanup and the report always run, so fixtures are removed and the report explains the failure.

To test a remote environment, specify the CMS base URL:

./run-tests.sh --full --url=https://cms.example.com

If the CMS uses non-default administrative credentials for the SUT REST service, set them as environment variables:

SUT_ADMIN_USER=<admin-user> \
SUT_ADMIN_PASSWORD=<admin-password> \
./run-tests.sh --full --url=https://cms.example.com

Test Fixtures and perf-env.json

During setup, the suite creates a controlled set of CMS users, folders, documents, and a manifest file named perf-env.json.

By default, setup creates:

• Five author users named perf-author-01, perf-author-02, etc.

• Five editor users named perf-editor-01, perf-editor-02, etc.

• One test document per configured user.

• A test content folder at /content/documents/_brxmperf.

• A temporary folder at /content/documents/_brxmperf/temp-created for UI-created documents.

The generated perf-env.json file contains:

• Installation metadata, including the target CMS URL and content size.

• The resources created during setup, used for cleanup.

• The users and documents referenced by the UI and API tests.

Keep perf-env.json in the performance test suite root directory when running UI and API tests separately.

Common Commands

Run these commands from the performance test suite root directory. On Windows, replace ./run-tests.sh with run-tests.bat.

Command	Description
./run-tests.sh --full --url=http://localhost:8080	Run setup, UI tests, API tests, cleanup, and report generation
./run-tests.sh --setup --url=http://localhost:8080	Install fixtures only
./run-tests.sh --status --url=http://localhost:8080	Check CMS connectivity, the bootstrap namespace, and fixture status
./run-tests.sh --cleanup --url=http://localhost:8080	Remove fixtures using perf-env.json
./run-tests.sh --ui --url=http://localhost:8080	Run the UI latency suite when fixtures already exist
./run-tests.sh --api --url=http://localhost:8080	Run the API load tests when fixtures already exist
./run-tests.sh --list-baselines	List stored performance baselines

To control the number of test users and documents created during setup, use --users=N:

./run-tests.sh --full --url=http://localhost:8080 --users=10

The default UI suite runs with 1, 3, and 5 concurrent users. To specify custom UI concurrency levels, run the UI suite directly from ui-latency-tests:

npm run test:suite -- --users=1,3,5,10

Exit Codes

The test runner exits with a non-zero code when a phase fails or the report verdict is FAILED. A WARNING or REGRESSION verdict still exits with code 0, because those results need a person to weigh them. In a CI pipeline, a non-zero exit code therefore means the run itself is not trustworthy, not that performance got worse.

UI Latency Tests

The UI latency tests use Playwright and Chromium to execute real CMS editorial workflows. The tests wait for the CMS UI to become idle before recording timings, so the metrics reflect user-perceived latency.

The UI suite covers:

• Folder and document navigation.

• Document edit and save operations.

• Author and editor document actions.

• Multi-step editorial workflows.

• Concurrent editing scenarios.

Single-user and multi-user runs measure the same three editorial journeys: creating a document, updating a document, and publishing and unpublishing a document. Journey results therefore line up across user counts in the Scaling Overview.

Environment variables for UI tests:

Variable	Default	Description
CMS_BASE_URL	http://localhost:8080	CMS base URL
CMS_CONTEXT_PATH	/cms	CMS web application context path
HEADLESS	true	Set to false to display browser windows
WARMUP_ITERATIONS	1	Warmup iterations before measurement
MEASURED_ITERATIONS	10	Measured iterations for atomic operations
JOURNEY_ITERATIONS	3	Measured iterations for journey tests
UI_IDLE_TIMEOUT	30000	Timeout in milliseconds while waiting for UI idle
CONCURRENT_USERS	1	Number of browser sessions for direct concurrent runs
MAX_USERS	5	Maximum users for the degradation curve test

Examples:

cd ui-latency-tests

# Run the single-user suite.
npm test

# Run the standard multi-user suite.
npm run test:suite

# Show browser windows for debugging.
HEADLESS=false npm run test:suite

# Increase the UI idle timeout for slower environments.
UI_IDLE_TIMEOUT=60000 npm test

API Load Tests

API load tests use Gatling to exercise the Channel Manager Content Service at /cms/ws/content/. These tests bypass browser rendering and focus on backend and repository performance during document editing.

Each virtual user performs the following steps:

1. Authenticate with the CMS.

2. Fetch document type metadata.

3. Lock a document.

4. Update one or more fields.

5. Save the draft variant.

6. Release the lock.

The tests use documents from perf-env.json when available. For accurate results, create at least as many test documents as concurrent API users to avoid lock contention.

Examples:

cd api-load-tests

# Verify that test documents are available.
mvn gatling:test -Dgatling.simulationClass=com.bloomreach.xm.performance.DiscoverySimulation

# Run a one-user smoke test.
mvn gatling:test -Dgatling.simulationClass=com.bloomreach.xm.performance.SmokeSimulation

# Run the default Content Service simulation with custom load.
mvn gatling:test -Dusers=5 -DrampDuration=10 -DtestDuration=180

Common Gatling properties:

Property	Default	Description
baseUrl	http://localhost:8080	CMS base URL if no perf-env.json is loaded
users	1	Number of concurrent virtual users
rampDuration	10	Seconds to ramp up to full load
testDuration	120	Seconds to run at steady state
maxResponseTime	5000	Maximum response time assertion in milliseconds
minSuccessRate	95	Minimum successful request percentage

Reports

After a full run, open:

reports/index.html

The combined dashboard includes:

• An executive summary with an overall verdict.

• UI latency heatmaps for each user count.

• Metrics for journeys, journey operations, and standalone operations.

• A Scaling Overview across user counts. When the run is compared with a baseline, a second tab, Scaling Overview (baseline), shows the same view for the baseline so you can compare how both runs scale.

• API throughput and latency metrics.

• Baseline deltas when a baseline is available.

The report is generated even when a phase fails. The verdict is FAILED when any UI test failed, when no measurements were collected, or when a result file could not be read, and the report lists the reasons. Where degradation cannot be computed, for example because the baseline has no data for one of the user counts, the Scaling Overview shows N/A and names the missing user counts.

Detailed reports are written to:

Report	Location
Combined dashboard	reports/index.html
Machine-readable summary	reports/summary.json
Playwright report	ui-latency-tests/playwright-report/index.html
Gatling report	api-load-tests/target/gatling//index.html

Baseline Comparison

Baselines allow you to detect performance regressions between releases.

To save a baseline after a known-good run:

./run-tests.sh --full --url=http://release-server:8080 --save-baseline=release-17.0

To compare a later run against a named baseline:

./run-tests.sh --full --url=http://test-server:8080 --compare=release-17.0

To list available baselines:

./run-tests.sh --list-baselines

Baselines saved with earlier versions of the suite remain comparable, including their multi-user journey results.

Baseline verdict thresholds:

Verdict	Meaning
Improvement	More than 5% faster than the baseline
Pass	No more than 5% slower than the baseline, and no more than 5% faster
Warning	More than 5% and up to 10% slower than the baseline
Regression	More than 10% slower than the baseline

Cleanup

Use cleanup after manual setup or partial test runs:

./run-tests.sh --cleanup --url=http://localhost:8080

Cleanup reads perf-env.json, deletes the generated documents, folders, and users, and removes the manifest file. If cleanup cannot find the manifest, provide it explicitly through the setup CLI:

cd perf-setup
mvn exec:java -Dexec.args="cleanup --manifest=../perf-env.json --base-url=http://localhost:8080"

Troubleshooting

perf-env.json is missing

Run setup before running UI or API tests separately:

./run-tests.sh --setup --url=http://localhost:8080

The status check cannot reach the CMS

Verify that the CMS is running and accessible from the test runner:

curl http://localhost:8080/cms/

For remote environments, check network routing, TLS configuration, and the CMS base URL passed with --url.

Playwright is not installed

The UI tests stop with Playwright is not installed. Install the UI test dependencies:

cd ui-latency-tests
npm ci

The UI suite is started directly rather than through Maven, so Maven does not install these dependencies for you.

Windows refuses arguments that contain percent signs

On Windows, the test runner refuses Maven arguments that contain %NAME%, for example in a URL. The command prompt would replace such text with the value of an environment variable even inside quotes, and it cannot be escaped. Set the value through an environment variable instead, or run the suite under WSL or Git Bash.

Maven reports that a child module does not exist

Distribution archives from 16.9.0 through 16.9.4 and from 17.0.0 through 17.2.0 declare modules in xm-perf-tests/pom.xml that the archive does not contain:

[ERROR] Child module .../xm-perf-tests/perf-dependencies of .../xm-perf-tests/pom.xml does not exist
[ERROR] Child module .../xm-perf-tests/perf-cms-dependencies of .../xm-perf-tests/pom.xml does not exist

Maven stops while building the project model, so every run-tests.sh action fails, on every platform. Releases 17.1.0 and later report both modules; earlier releases report only perf-dependencies.

This is fixed in 16.9.5, 17.2.1, and 17.3.0. For older archives, remove the offending lines from the <modules> section of xm-perf-tests/pom.xml:

<module>perf-dependencies</module>
<module>perf-cms-dependencies</module>

Both are dependency-only POMs that the suite resolves from the Maven repository rather than building locally, so removing them from the module list has no further effect.

Setup fails with "No message body reader has been found"

The failure appears while validating prerequisites:

Install failed: Cannot connect to SUT at http://localhost:8080: No message body reader has been found
for class com.onehippo.cms7.sut.rest.api.error.RestException, ContentType: text/html;charset=UTF-8

The SUT REST service is not deployed, so the CMS returns an HTML error page where the client expects JSON. The message names a content type rather than the missing service, so it is easy to misread as a connectivity problem.

Add bloomreach-xm-perf-cms-dependencies to the CMS module and redeploy. See "Add the performance test dependencies to the CMS". Confirm the fix with:

curl -u <admin-user>:<admin-password> <cms-base-url>/cms/ws/qa/jcr/

A working deployment returns HTTP 200 and a JSON body. An HTTP 404 means the service is still missing.

Setup fails because the namespace is missing

Confirm that bloomreach-xm-perf-cms-dependencies is deployed in the CMS, that the CMS was built with the perf-testing profile active, and that /hippo:namespaces/brxmperf exists in the repository console.

The report verdict is FAILED

The report lists the reasons at the top of the executive summary. The most common cause is a UI test that failed, for example because the CMS did not become idle in time. Open the Playwright report for the failing test, and see the next section.

UI tests time out while waiting for the CMS to become idle

Increase the UI idle timeout:

UI_IDLE_TIMEOUT=60000 npm test

If the timeout persists, run the tests with browser windows visible and check for dialogs, overlays, or long-running CMS requests:

HEADLESS=false npm run test:suite