Skip User Agents

The Relevance Module does not target requests from web crawlers and link checker tools by default. This approach prevents automated agents—such as those used by Google, Yahoo, Bing, WatchMouse, W3C-checklink, and Xenu Link Sleuth—from affecting targeting, scoring, and normalization data. It also ensures that real-time visitor analysis does not display visits from these agents. Additionally, search engines do not benefit from indexing personalized or targeted content, such as location-specific blocks, which do not add value to search results.

The module identifies these agents by inspecting the User-Agent HTTP header. If the header contains a string that matches a known crawler or link checker, the request is excluded from targeting and related analytics.

Default Behavior

• The Relevance Module maintains a default list of user agents to skip.
• Requests from these user agents are ignored for targeting, scoring, normalization, and real-time visitor analysis.
• The module checks if the User-Agent header contains any string from the skip list.

Customizing Skipped User Agents

You can add additional user agents to the skip list in the repository. Update the multi-valued property at:

/targeting:targeting/targeting:skipUserAgents

Review your request logs to identify and exclude any additional agents that should not be targeted.

Reference

The default list of skipped user agents is defined in the hippo-addon-targeting-repository at:

src/main/resources/hcm-config/targeting-configuration.yaml

For more information about scoring and normalization, see Scoring and Normalization.