en-US

Alerts


User roles for this feature

Admin Co-admin Asset Manager Viewer
Alerts View
Alert Policies View
Add/Edit/Delete
Notification Channels View
Add/Edit/Delete


Overview

The Alerts feature helps Service Experience Insights users improve service quality and user satisfaction through proactive performance monitoring, issue identification, and faster resolution.

The Service Experience Insights Alerts feature enables users to create threshold-based Alert Policies that continuously monitor the performance of agent-to-target paths. These policies trigger Alerts and Alert Notifications when the observed metrics violate the threshold for the duration of the event. Create Alert Policies to monitor latency, jitter, loss, HTTP target availability, WiFi signal strength, and agent connectivity. Configure the trigger conditions and notification channels for Alerts triggered by the Alert Policy.

Alerts are stored in time series and listed in the Alerts table. Each Alert links to an Alert page that shows the threshold and the observed metrics before, during, and after the event—view by searching the Alerts table or clicking through from a notification.

Receive alerts for a single agent-to-target path or use Agent and Targets Groups to trigger alerts that identify issues specific to certain regions or network segments.

Subscribe to alerts via webhooks or email. Webhooks allow users to receive real-time notifications and trigger automated actions responding to an alert, such as populating a Slack channel. Subscribe to a webhook to receive a JSON file for alerts grouped by Alert Policy. Use webhooks with your existing systems to automate tasks like sending an SMS message or opening a ticket in a service management tool.

Key benefits:

  • Continuous monitoring: The feature enables users to create Alert Policies that continuously monitor the performance of agent-to-target paths, ensuring that issues are identified and addressed promptly.

  • Threshold-based alerts: Alert Policies trigger Alerts and Alert Notifications when observed metrics violate the threshold for the duration of the event. This helps to identify performance issues and take corrective actions proactively.

  • Multiple monitoring options: Users can create Alert Policies to monitor latency, jitter, loss, HTTP target availability, WiFi signal strength, and agent connectivity, providing comprehensive monitoring coverage.

  • Configurable triggers and notifications: Easily configure the trigger conditions and notification channels for Alerts triggered by the Alert Policy for specific use cases.

  • Group-based alerts: Users can receive alerts for a single agent-to-target path or use Agent and Targets Groups to trigger alerts that identify issues specific to certain regions or network segments.

  • Integration with existing systems: The feature allows users to subscribe to alerts via webhooks or email and integrate with their existing systems to automate tasks like sending an SMS message or opening a ticket in a service management tool.

  • Real-time notifications: Webhooks enable users to receive real-time notifications and trigger automated actions responding to an alert, such as populating a Slack channel, ensuring that issues are addressed promptly.


Alerts Page

Use the Alerts Page to view active and recovered alerts, access detailed information about each alert, configure threshold-based Alert Policies, and set up email and webhook distributions for alert notifications.

The Alerts Dashboard is designed to be intuitive and user-friendly, allowing users to easily access the information and tools they need to monitor their network performance effectively.

The alerts dashboard contains three important functions located in the page tabs:

  • Alerts: View and search alerts that are triggering or resolved in the Alerts > Alerts tab.

  • Alert Policies: Create Alert Policies and view alerts triggered by each Alert Policy in the Alerts > Alert Policies tab.

  • Notification Channels: Configure email distribution lists and webhook endpoints to receive alert notifications in the Alerts > Notification Channels tab.

Alerts > Alerts tab: View and search alerts that are triggering or resolved.

Click to expand

View and search alerts stored in the time series database.

  • Info panels at the top provide a summary of alerts. Click the link to sort the table to show the alerts summarized in the info panel.

    • Alerts Firing provides a count of all firing alerts.

    • New Alerts show a count of the alerts triggered in the last seven days.

    • New Firing Alerts counts alerts that began firing within the last week and have not yet recovered.

    • New Alerts Resolved gives a count of the alerts that have recovered in the past seven days.

  • Alerts are listed chronologically, with the most recent listed first.

  • Sort alerts by date and time.

  • Filter alerts by status. Is Firing or Resolved.

  • Click the Start date link to view the Alert Summary.

  • Click the Alert Policy name link to view the Alert Policy that triggered the alert and a list of Alerts for that Policy.

Table sorting

  • Start is the only sortable column in the Alerts pages.

  • Due to the volume of alerts, sorting is not practical in most projects and has been removed to improve page performance.

  • Search is not supported for the same reason.

  • Time range and filters are provided in each table.


Alert Summary Pages

Click through from the alerts list or from an Alert Notification to view a summary of each alert.

Alert summaries provide metrics and context about the monitored event that triggered the alert.

  • Firing Alerts show the metric value relative to the threshold. (above the dotted threshold line or red for connectivity alerts)
  • Recovered Alert Summary provides metrics before, during, and after the event.
  • Metrics for the time of the alert are highlighted in red. Alerts begin when an observed threshold violation that triggered last longer than the threshold duration set in the Alert Policy. Metrics are shown for this time but are not highlighted in red.

Firing Alert - Example of Connectivity Alert Firing for nine hours

Click to expand

Recovered Alert - Example of Loss Alert

Click to expand


Alert Policies

Alert Policies are rules that define when user-defined threshold-based alerts are triggered. Set threshold metrics such as latency, packet loss, and agent connectivity. When the observed metrics violate the thresholds, an alert is generated to notify the user of the anomaly.

View and create customizable alerting policies that define when threshold-based alert triggers and notifications routing.

Alerts > Alert Policies tab: Create Alert Policies and view alerts triggered by each Alert Policy.

Click to expand

Alert Policy Configuration


Name & Description Protocol Selected Agents Selected Targets Trigger Conditions Notification Channels
Name and describe the Alert Policy Choose one for each Alert Policy
  • Connectivity
  • Latency
  • Jitter
  • Loss
  • HTTP Availability
  • WiFI Signal Strength
Assign Agents
  • All in one or more agent groups
  • Individually by name
Assign Targets
  • All in one or more target groups
  • Individually by name
  • Duration
  • Reporting Interval
  • Threshold
  • Email distribution list
  • Additional emails for policy
  • Slack webhook
  • Custom webhook
Example
  • SaaS Loss SLA
  • “Send email notifications to ITOps when loss between Access Switches and SaaS services exceeds 2023 SLA.”
Loss Access Switch agent group SaaS target group Greater Than 2% for more than 5 minutes, reported every 60 seconds
  • IT-Ops email Notification Channel.
  • Level-3 ServiceNow webhook


Add Alert Policy

To create an Alert Policy, go to the Alerts > Alert Policies tab. Click the Add Alert Policy button to open the Add Alert Policies modal.

Complete these steps to begin triggering threshold-based alerts and receiving alert notifications.

Click to expand

Step 1: Name and Description

  • Enter a unique and descriptive name for the Alert Policy.
  • Describe the use case for the alert. Alerts for each policy include this Description which helps provide context to users viewing the alert.
Click to expand

Step 2: Alert Metric

  • Select the alert metric.
  • Each Alert Policy supports a single metric.
Available Metrics
Connectivity Agent down detection - no connectivity between the agent and SEI controller.
Latency The time it takes for a packet of data to travel from its source to its destination. (milliseconds)
Jitter The variation in the latency of individual packets in a data stream. (milliseconds)
Loss The failure of one or more transmitted packets to arrive at their destination. (percentage of failed packets)
HTTP Availability The availability of a web server or other network resource. (percentage of successful packets)
WiFi Signal Strength A measure of the strength of the wireless signal from a WiFi router or access point. Scale 0% to 100%. 0%, no signal - 100%, no detected signal degradation.
Click to expand

Step 3: Select Agents

Add agents to the Alert Policy using one of the following methods. Agent selection methods cannot be combined.

Agent Groups dropdown

  • Choose one or more groups from the Agent Groups dropdown.
  • The Alert Policy automatically updates when agents are added or removed from the selected groups. This eliminates the hassle of manually updating the Alert Policy whenever a new agent is deployed, or an existing agent is deactivated.
Click to expand

Agent Name

  • Click the search box to show agents nested in groups.
  • Search by agent attribute to filter agent groups. Example: Partial search of “Rasp” shows agent groups with Raspberry Pi-hosted agents.
  • Select all returned agents, all agents in a group, or click the plus icon to open a group where agents can be selected individually by name.
  • Select agents individually or all agents in the group. Unlike the agent group dropdown, selecting the agent group here does not automatically update the Alert Policy when agents are added, or removed from the group.
Click to expand

Step 4: Select Targets

Add targets to the Alert Policy using one of the following methods. Target selection methods cannot be combined.

Target Groups dropdown

  • Choose one or more groups from the Target Groups dropdown.
  • The Alert Policy automatically updates when targets are added or removed from the selected groups. This eliminates the hassle of manually updating the Alert Policy whenever a new target is deployed, or an existing target is deleted.
Click to expand

Target Name

  • Click the search box to show targets nested in target groups.
  • Search by target attribute to filter groups. Example: Partial search of “Micro” shows target groups with Microsoft Targets.
  • Select all returned targets, all targets in a group, or click the plus icon to open a group where targets can be selected individually by name.
  • Select targets individually or all targets in the group. Unlike the target group dropdown, selecting the target group here does not automatically update the Alert Policy when targets are added or removed from the group.
Click to expand

Step 5: Trigger Conditions

  • Set Threshold-based alert triggers to detect abnormalities in the monitored metric for a meaningful period of time.
  • Configure the conditions to trigger an alert and notifications for this Alert Policy.

Threshold

  • Alerts are triggered when the observed value is Less Than or Greater Than the threshold metric value.
  • Choose the Comparision Operator: Less Than or Greater Than
  • Enter a threshold metric value.
Threshold Metric Comparison Operator Threshold Value

Latency

Greater Than, Less Than - Defaults to Greater Than

Milliseconds (ms) - ex: 150ms - no default

Jitter

Greater Than, Less Than - Defaults to Greater Than

Milliseconds (ms) - ex: 75ms - no default

Loss

Greater Than, Less Than - Defaults to Greater Than

Percent(%) - ex: 2% - no default

HTTP Availability

Greater Than, Less Than - Defaults to Less Than

Percent(%) - ex: 98% no default

WiFI Signal Strength

Greater Than, Less Than - Defaults to Less Than

Percent(%) - ex: 90% - no default

Connectivity (Agent Status)

Equals, Not Equals - Defaults to Equals

On, Off - Defaults to Off

Duration

  • Set the amount of time that an event is observed before triggering an alert.
  • Duration is configurable in hours and minutes.
Click to expand
Click to expand

Step 6: Notification Channels

Notification Channels are email distribution lists and webhook endpoints to receive Alert Notifications when the Alert Policy triggers alerts.

  • Notification Channels are project-level resources.
  • Existing Notification Channels are listed by display name in the dropdown.
  • Select one more Notification Channels.
  • Notification Channels are managed in the Alerts > Notification Channel tab. See Notification Channels to learn how they are configured.
  • Use the dropdown to select a Notification Channel.
  • Click the Add Notification Channel link to select additional Notification channels.
Click to expand
Click to expand
Click to expand

Once added to an Alert Policy, the Notification Channels is listed on the Alert Policy page.

Click to expand

Notification Channels

Notification channels are email distribution lists and webhook endpoints that receive Alert Notifications when an Alert Policy triggers alerts.

Email, Slack, and custom Webhooks are supported.

Alerts > Notification Channels tab: Create Notification Channels and view and edit existing Notification channels that are available to be added to Alert Policies.

Click to expand

Click the Notification Channel link to view the notification page that lists the assigned emails or webhooks and the Alert Policies assigned to the Notification Channel.

Click to expand

Add Notification Channel

To create a Notification Channel, go to the Alerts > Notification Channels tab. Click the Add Notification Channel button to open the Add Notification Channel modal.

Complete these steps to create a Notification Channel that can be added to any Alert Policy.

Step 1: Name and Description

  • Enter a unique and descriptive name for the Notification Channel.
  • Describe the use case for the Notification Channel. Notification Channel pages include this Description.
Click to expand

Step 2: Select Type

  • Create a Notification Channel for each endpoint
  • Choose Email, Slack, or Webhook
Click to expand

Email

  • Select Email
  • Add comma-separated recipient email addresses
  • Enable and Save
  • Edit an existing Alert Policy to include this Notification Channel
Click to expand

Slack

  • Select Slack
  • Enter the Slack Incoming Webhook URL
  • Learn more about using Slack webhooks
  • Create a separate Notification Channel for each Slack Incoming Webhook URL you wish to use.
  • Alert Notification webhook payloads use the standard Slack Template
  • Enable and Save
  • Edit an existing Alert Policy to include this Notification Channel
Click to expand

Webhook

  • Select Webhook
  • Enter the Webhook URL
  • Enter authentication headers if required by the webhook endpoint
  • Add a header for each Alert Notification API result
  • Create a separate Notification Channel for each Webhook you wish to use.
  • Enable and Save
  • Edit an existing Alert Policy to include this Notification Channel
Click to expand


Common JSON Parameters

  • Use this format to create URLs that link to Alerts in the dashboard
  • dashboard.insights.edgelq.com/alert?policyId={alertingPolicy.id}&conditionId={alertingCondition.id}&alertId={alerts.id}&scope=projects/{project.id}

Name Format Description

languageCode

string

Alert Notification language, “ja-jp” or “en-us”

controllerDomain

string

Used to build links to Dashboard, if “dashboard.edgeLQ.com”, root url is https://dashboard.insights.edgelq.com/

alertingPolicy.id

string

Used to build links to Dashboard, /alert?policyId={policyId}

alertingCondition.id

string

Used to build links to Dashboard, &conditionId=<{conditionId}

alerts.id

string

Used to build links to Dashboard, &alertId={alerts.id}

projectId

string

Used to build links to Dashboard, &scope=projects/{projectId}

notification.state.alertsLifetime.startTime

string

Time alert started (if firing)

notification.state.alertsLifetime.updateTime

string

Time alert recovered (if resolved)

events.state

string

“isFiring:” true or false - (false equal recovered)

alerts.monitoredResource.labels.probe_id

string

name of agent


Testing Notification Channels

Notification Channels can be tested by clicking the Send button on the Notifications tab or in a Notification Page.

  • Email, Slack, and Webhook tests are confirmed by a pop-up success message that confirms that a test payload has been sent from the dashboard.
  • The confirmation message does not confirm that he email was received by the endpoint.
  • All Alert Notifications will be sourced from the same email address. To ensure that alerts are not being lost to a spam filter, please be sure to add noreply@monitoring.edgelq.com as a trusted email source.
Click to expand
Click to expand


Email Notification Channels

Email Alert Notifications are sent when alerts start firing and again when alerts are resolved.

Alert emails show all alerts for an alert policy every two minutes. Typically, an alert message will contain a single alert, but many could be listed.

  • Top link points to dashboard home page for the project
  • Time (UTC) link points to the alert. Note that the UTC time listed in the email is in UTC, and the alert date shown in the dashboard is in the time zone set by the user.
  • Alert Policy link points to the alert policy page that shows all alerts for the policy.
  • Notifications Channel link points to the notification channel settings of the email.
Click to expand




In This Article