Articles in this section

Dashboard Alerts Agents On-demand Troubleshooting Targets Probing Distributions User Management Activities Log Dashboard Access for New Users

Alerts

User roles for this feature

		Admin	Co-admin	Asset Manager	Viewer
Alerts	View
Alert Policies	View
	Add/Edit/Delete
Notification Channels	View
	Add/Edit/Delete

Overview

The Alerts feature helps Service Experience Insights users improve service quality and user satisfaction through proactive performance monitoring, issue identification, and faster resolution.

The Service Experience Insights Alerts feature enables users to create threshold-based Alert Policies that continuously monitor the performance of agent-to-target paths. These policies trigger Alerts and Alert Notifications when the observed metrics violate the threshold for the duration of the event. Create Alert Policies to monitor latency, jitter, loss, HTTP target availability, WiFi signal strength, and agent connectivity. Configure the trigger conditions and notification channels for Alerts triggered by the Alert Policy.

Alerts are stored in time series and listed in the Alerts table. Each Alert links to an Alert page that shows the threshold and the observed metrics before, during, and after the event—view by searching the Alerts table or clicking through from a notification.

Receive alerts for a single agent-to-target path or use Agent and Targets Groups to trigger alerts that identify issues specific to certain regions or network segments.

Subscribe to alerts via webhooks or email. Webhooks allow users to receive real-time notifications and trigger automated actions responding to an alert, such as populating a Slack channel. Subscribe to a webhook to receive a JSON file for alerts grouped by Alert Policy. Use webhooks with your existing systems to automate tasks like sending an SMS message or opening a ticket in a service management tool.

Key benefits:

Continuous monitoring: The feature enables users to create Alert Policies that continuously monitor the performance of agent-to-target paths, ensuring that issues are identified and addressed promptly.
Threshold-based alerts: Alert Policies trigger Alerts and Alert Notifications when observed metrics violate the threshold for the duration of the event. This helps to identify performance issues and take corrective actions proactively.
Multiple monitoring options: Users can create Alert Policies to monitor latency, jitter, loss, HTTP target availability, WiFi signal strength, and agent connectivity, providing comprehensive monitoring coverage.
Configurable triggers and notifications: Easily configure the trigger conditions and notification channels for Alerts triggered by the Alert Policy for specific use cases.
Group-based alerts: Users can receive alerts for a single agent-to-target path or use Agent and Targets Groups to trigger alerts that identify issues specific to certain regions or network segments.
Integration with existing systems: The feature allows users to subscribe to alerts via webhooks or email and integrate with their existing systems to automate tasks like sending an SMS message or opening a ticket in a service management tool.
Real-time notifications: Webhooks enable users to receive real-time notifications and trigger automated actions responding to an alert, such as populating a Slack channel, ensuring that issues are addressed promptly.

Alerts Page

Use the Alerts Page to view active and recovered alerts, access detailed information about each alert, configure threshold-based Alert Policies, and set up email and webhook distributions for alert notifications.

The Alerts Dashboard is designed to be intuitive and user-friendly, allowing users to easily access the information and tools they need to monitor their network performance effectively.

The alerts dashboard contains three important functions located in the page tabs:

Alerts: View and search alerts that are triggering or resolved in the Alerts > Alerts tab.
Alert Policies: Create Alert Policies and view alerts triggered by each Alert Policy in the Alerts > Alert Policies tab.
Notification Channels: Configure email distribution lists and webhook endpoints to receive alert notifications in the Alerts > Notification Channels tab.

Alerts > Alerts tab: View and search alerts that are triggering or resolved.

View and search alerts stored in the time series database.

Info panels at the top provide a summary of alerts. Click the link to sort the table to show the alerts summarized in the info panel.
- Alerts Firing provides a count of all firing alerts.
- New Alerts show a count of the alerts triggered in the last seven days.
- New Firing Alerts counts alerts that began firing within the last week and have not yet recovered.
- New Alerts Resolved gives a count of the alerts that have recovered in the past seven days.
Alerts are listed chronologically, with the most recent listed first.
Sort alerts by date and time.
Filter alerts by status. Is Firing or Resolved.
Click the Start date link to view the Alert Summary.
Click the Alert Policy name link to view the Alert Policy that triggered the alert and a list of Alerts for that Policy.

Table sorting

Start is the only sortable column in the Alerts pages.
Due to the volume of alerts, sorting is not practical in most projects and has been removed to improve page performance.
Search is not supported for the same reason.
Time range and filters are provided in each table.

Alert Summary Pages

Click through from the alerts list or from an Alert Notification to view a summary of each alert.

Alert summaries provide metrics and context about the monitored event that triggered the alert.

Firing Alerts show the metric value relative to the threshold. (above the dotted threshold line or red for connectivity alerts)
Recovered Alert Summary provides metrics before, during, and after the event.
Metrics for the time of the alert are highlighted in red. Alerts begin when an observed threshold violation that triggered last longer than the threshold duration set in the Alert Policy. Metrics are shown for this time but are not highlighted in red.

Firing Alert - Example of Connectivity Alert Firing for nine hours

Recovered Alert - Example of Loss Alert

Alert Policies

Alert Policies are rules that define when user-defined threshold-based alerts are triggered. Set threshold metrics such as latency, packet loss, and agent connectivity. When the observed metrics violate the thresholds, an alert is generated to notify the user of the anomaly.

View and create customizable alerting policies that define when threshold-based alert triggers and notifications routing.

Alerts > Alert Policies tab: Create Alert Policies and view alerts triggered by each Alert Policy.

Alert Policy Configuration


Name and describe the Alert Policy	Choose one for each Alert Policy Connectivity Latency Jitter Loss HTTP Availability WiFI Signal Strength	Assign Agents All in one or more agent groups Individually by name	Assign Targets All in one or more target groups Individually by name	Duration Reporting Interval Threshold	Email distribution list Additional emails for policy Slack webhook Custom webhook
Example
SaaS Loss SLA “Send email notifications to ITOps when loss between Access Switches and SaaS services exceeds 2023 SLA.”	Loss	Access Switch agent group	SaaS target group	Greater Than 2% for more than 5 minutes, reported every 60 seconds	IT-Ops email Notification Channel. Level-3 ServiceNow webhook

Name & Description

Protocol

Selected Agents

Selected Targets

Trigger Conditions

Notification Channels

Name and describe the Alert Policy

Choose one for each Alert Policy

Connectivity

Latency

Jitter

Loss

HTTP Availability

WiFI Signal Strength

Assign Agents

All in one or more agent groups

Individually by name

Assign Targets

All in one or more target groups

Individually by name

Duration

Reporting Interval

Threshold

Email distribution list

Additional emails for policy

Slack webhook

Custom webhook

Example

SaaS Loss SLA

“Send email notifications to ITOps when loss between Access Switches and SaaS services exceeds 2023 SLA.”

Loss

Access Switch agent group

SaaS target group

Greater Than 2% for more than 5 minutes, reported every 60 seconds

IT-Ops email Notification Channel.

Level-3 ServiceNow webhook

Add Alert Policy

To create an Alert Policy, go to the Alerts > Alert Policies tab. Click the Add Alert Policy button to open the Add Alert Policies modal.

Complete these steps to begin triggering threshold-based alerts and receiving alert notifications.

Step 1: Name and Description

Enter a unique and descriptive name for the Alert Policy.
Describe the use case for the alert. Alerts for each policy include this Description which helps provide context to users viewing the alert.

Step 2: Alert Metric

Select the alert metric.
Each Alert Policy supports a single metric.

Available Metrics
Connectivity	Agent down detection - no connectivity between the agent and SEI controller.
Latency	The time it takes for a packet of data to travel from its source to its destination. (milliseconds)
Jitter	The variation in the latency of individual packets in a data stream. (milliseconds)
Loss	The failure of one or more transmitted packets to arrive at their destination. (percentage of failed packets)
HTTP Availability	The availability of a web server or other network resource. (percentage of successful packets)
WiFi Signal Strength	A measure of the strength of the wireless signal from a WiFi router or access point. Scale 0% to 100%. 0%, no signal - 100%, no detected signal degradation.

Step 3: Select Agents

Add agents to the Alert Policy using one of the following methods. Agent selection methods cannot be combined.

Agent Groups dropdown

Choose one or more groups from the Agent Groups dropdown.
The Alert Policy automatically updates when agents are added or removed from the selected groups. This eliminates the hassle of manually updating the Alert Policy whenever a new agent is deployed, or an existing agent is deactivated.

Agent Name

Click the search box to show agents nested in groups.
Search by agent attribute to filter agent groups. Example: Partial search of “Rasp” shows agent groups with Raspberry Pi-hosted agents.
Select all returned agents, all agents in a group, or click the plus icon to open a group where agents can be selected individually by name.
Select agents individually or all agents in the group. Unlike the agent group dropdown, selecting the agent group here does not automatically update the Alert Policy when agents are added, or removed from the group.

Step 4: Select Targets

Add targets to the Alert Policy using one of the following methods. Target selection methods cannot be combined.

Target Groups dropdown

Choose one or more groups from the Target Groups dropdown.
The Alert Policy automatically updates when targets are added or removed from the selected groups. This eliminates the hassle of manually updating the Alert Policy whenever a new target is deployed, or an existing target is deleted.

Target Name

Click the search box to show targets nested in target groups.
Search by target attribute to filter groups. Example: Partial search of “Micro” shows target groups with Microsoft Targets.
Select all returned targets, all targets in a group, or click the plus icon to open a group where targets can be selected individually by name.
Select targets individually or all targets in the group. Unlike the target group dropdown, selecting the target group here does not automatically update the Alert Policy when targets are added or removed from the group.

Step 5: Trigger Conditions

Set Threshold-based alert triggers to detect abnormalities in the monitored metric for a meaningful period of time.
Configure the conditions to trigger an alert and notifications for this Alert Policy.

Threshold

Alerts are triggered when the observed value is Less Than or Greater Than the threshold metric value.
Choose the Comparision Operator: Less Than or Greater Than
Enter a threshold metric value.

Threshold Metric	Comparison Operator	Threshold Value
Latency	Greater Than, Less Than - Defaults to Greater Than	Milliseconds (ms) - ex: 150ms - no default
Jitter	Greater Than, Less Than - Defaults to Greater Than	Milliseconds (ms) - ex: 75ms - no default
Loss	Greater Than, Less Than - Defaults to Greater Than	Percent(%) - ex: 2% - no default
HTTP Availability	Greater Than, Less Than - Defaults to Less Than	Percent(%) - ex: 98% no default
WiFI Signal Strength	Greater Than, Less Than - Defaults to Less Than	Percent(%) - ex: 90% - no default
Connectivity (Agent Status)	Equals, Not Equals - Defaults to Equals	On, Off - Defaults to Off

Duration

Set the amount of time that an event is observed before triggering an alert.
Duration is configurable in hours and minutes.

Step 6: Notification Channels

Notification Channels are email distribution lists and webhook endpoints to receive Alert Notifications when the Alert Policy triggers alerts.

Notification Channels are project-level resources.
Existing Notification Channels are listed by display name in the dropdown.
Select one more Notification Channels.
Notification Channels are managed in the Alerts > Notification Channel tab. See Notification Channels to learn how they are configured.
Use the dropdown to select a Notification Channel.
Click the Add Notification Channel link to select additional Notification channels.

Notification Channels

Notification channels are email distribution lists and webhook endpoints that receive Alert Notifications when an Alert Policy triggers alerts.

Email, Slack, and custom Webhooks are supported.

Alerts > Notification Channels tab: Create Notification Channels and view and edit existing Notification channels that are available to be added to Alert Policies.

Click the Notification Channel link to view the notification page that lists the assigned emails or webhooks and the Alert Policies assigned to the Notification Channel.

Add Notification Channel

To create a Notification Channel, go to the Alerts > Notification Channels tab. Click the Add Notification Channel button to open the Add Notification Channel modal.

Complete these steps to create a Notification Channel that can be added to any Alert Policy.

Step 1: Name and Description

Enter a unique and descriptive name for the Notification Channel.
Describe the use case for the Notification Channel. Notification Channel pages include this Description.

Step 2: Select Type

Create a Notification Channel for each endpoint
Choose Email, Slack, or Webhook

Email

Select Email
Add comma-separated recipient email addresses
Enable and Save
Edit an existing Alert Policy to include this Notification Channel

Slack

Select Slack
Enter the Slack Incoming Webhook URL
Learn more about using Slack webhooks
Create a separate Notification Channel for each Slack Incoming Webhook URL you wish to use.
Alert Notification webhook payloads use the standard Slack Template
Enable and Save
Edit an existing Alert Policy to include this Notification Channel

Webhook

Select Webhook
Enter the Webhook URL
Enter authentication headers if required by the webhook endpoint
Add a header for each Alert Notification API result
Create a separate Notification Channel for each Webhook you wish to use.
Enable and Save
Edit an existing Alert Policy to include this Notification Channel

Common JSON Parameters

Use this format to create URLs that link to Alerts in the dashboard

dashboard.insights.edgelq.com/alert?policyId={alertingPolicy.id}&conditionId={alertingCondition.id}&alertId={alerts.id}&scope=projects/{project.id}

Name	Format	Description
languageCode	string	Alert Notification language, “ja-jp” or “en-us”
controllerDomain	string	Used to build links to Dashboard, if “dashboard.edgeLQ.com”, root url is https://dashboard.insights.edgelq.com/
alertingPolicy.id	string	Used to build links to Dashboard, /alert?policyId={policyId}
alertingCondition.id	string	Used to build links to Dashboard, &conditionId=<{conditionId}
alerts.id	string	Used to build links to Dashboard, &alertId={alerts.id}
projectId	string	Used to build links to Dashboard, &scope=projects/{projectId}
notification.state.alertsLifetime.startTime	string	Time alert started (if firing)
notification.state.alertsLifetime.updateTime	string	Time alert recovered (if resolved)
events.state	string	“isFiring:” true or false - (false equal recovered)
alerts.monitoredResource.labels.probe_id	string	name of agent

Testing Notification Channels

Notification Channels can be tested by clicking the Send button on the Notifications tab or in a Notification Page.

Email, Slack, and Webhook tests are confirmed by a pop-up success message that confirms that a test payload has been sent from the dashboard.
The confirmation message does not confirm that he email was received by the endpoint.
All Alert Notifications will be sourced from the same email address. To ensure that alerts are not being lost to a spam filter, please be sure to add noreply@monitoring.edgelq.com as a trusted email source.

Email Notification Channels

Email Alert Notifications are sent when alerts start firing and again when alerts are resolved.

Alert emails show all alerts for an alert policy every two minutes. Typically, an alert message will contain a single alert, but many could be listed.

Top link points to dashboard home page for the project
Time (UTC) link points to the alert. Note that the UTC time listed in the email is in UTC, and the alert date shown in the dashboard is in the time zone set by the user.
Alert Policy link points to the alert policy page that shows all alerts for the policy.
Notifications Channel link points to the notification channel settings of the email.