When incidents occur, the on-call person is typically notified through automated alerts or manual incident creation. In this article, we’ll focus on automating the process of escalating and creating incidents.
Head here if you want to read more about manually creating incidents.
Configuring On-call features
Before configuring alerts for automatic escalation and incident creation, ensure that you have:
1️⃣ Set up your teams and any necessary types in the Catalog.
2️⃣ Created schedules.
3️⃣ Established escalation paths and linked them to the appropriate teams in the Catalog.
💡 To learn more about the main steps to configure On-call, head here
Creating incidents from alerts
Alert configuration consists of configuring your
Alert sources and attributes
Alert routing including filtering, escalations, incident creation and grouping
Connecting your alert source
Go to Alert Configuration
Choose your alert source
If a direct integration isn’t available for your source, connect it via HTTP.
Follow the setup instructions to connect your source and send a test alert.
Configure alert attributes and priority with the help of our AI suggestions.
Choose to use attributes or priority from the alert payload or set them as a static field
Ensure that your alert includes an attribute specifying who should be paged. Whether you page based on Team, Service, or another criterion, the alert should be able to reference an escalation path defined in the Catalog.
💡 Attributes provide extra context to your alerts, like services, affected features, or environments.
Learn more about Attributes here and Priorities here
Routing your alerts to start escalating and creating incidents
Now set up your alert routes to escalate and create incidents automatically.
Create a new Alert route in Alert configuration
Select the sources you want to include
Filter the alerts you want or don't want to trigger incidents
Enable Escalations, choosing either
Dynamic escalation paths based on your team or service attribute (recommended), or
Static paths to use the same escalation path for all alerts
Enable incident creation
Automatically create incidents or filter which alerts should trigger them.
Configure grouping so similar alerts are handled together [Learn more]
You can also choose Mode=Test to create test incidents
💡 We recommend using only a few alert routes: One for engineering, one for the support team and maybe one for security teams if needed. Using Catalog to dynamically route escalations eases up the configuration and keeps things aligned.
Learn more about Alert configuration here
Creating incidents with a third-party paging tool
If you are still using a third-party tool like PagerDuty or OpsGenie the incident creation can still be manual, but you won't be able to dynamically route escalations as those exist in your third-party tool.
💡 Using both our On-call and Response product can create lot of benefits like unified data flow, continuous feedback on your alerts and so, better noise management all under in a single pane of glass.
Learn more about our On-call here