🌐 Overview
If you're ready to get started with your incident.io On-call program, you've come to the right place! This document will walk you through the components of setting up On-call for your organization, specifically with:
Alerts
Sources
Routes
Escalation paths
Schedules
We'd recommend you get set up in the order listed above, however, you can set these components up in whatever order is easiest for your organization.
🔥 Creating escalations
In order to page your on-call users we need to know when something has gone wrong.
You can trigger an escalation automatically through an alert route, or manually from an incident.
🚨 Escalating from an Alert
Alert sources
First, you'll need to set up the connection between incident.io and your alert source(s). We connect with a variety of sources including:
Monitoring tools such as Datadog or Prometheus
Service management tools like Zendesk
If we don’t have an integration in place for your preferred source, you can connect via HTTP. This is a useful option when we don't integrate directly with your tool, but you are able to send us an HTTP request.
Once you have selected the source you’d like to connect you'll find detailed instructions on how to set up your integration with incident.io.
Custom alert attributes
When connecting your alert sources to incident.io, you can add custom attributes to provide more context to your alerts. Examples include: team, affected customer, affected feature or environment.
These can provide helpful context for responders when they receive the alert. In addition, attributes can be used to group and filter your alerts when they're being used to create incidents.
💡 We have a more detailed end-to-end guide on getting started with Alerts here.
Alert routes
Once you have connected your alert sources, to automatically create incidents and to escalate accordingly, you’ll need to set up your alert routes. With alert routes you can:
Select one (or multiple!) alert sources to auto-create incidents
Filter any alerts you do not want to auto-create incidents for (e.g. low priority alerts)
Group alerts, so that when an alert fires, we find active incidents with similar alerts and give responders the ability to attach the alert to an incident or create a new incident from that alert (ie. grouping alerts by the Team that owns the alert)
Configure what your incidents look like (ie. populating custom fields from alert data)
Determine how and to whom you would like to escalate based on your alert configuration
You can create a new escalation path directly from your alert route or if you prefer to set it up later, you can follow the instructions below in the On-call > Escalation paths section of the dashboard.
🪄 It's worth noting that you don't need to create an alert route for each team you want to escalate to. For best practice on how to escalate to the correct team from an alert, please see this article here.
🙋 Escalating from a manually declared incident
It's not always possible to have an alert fire every time something goes wrong and so it's important to be able to page your on-call teams when someone manually declares an incident.
You can automatically escalate an incident within any given workflow using the Escalate via incident.io step. This then lets you choose an escalation path(s) or user(s) to escalate to when that workflow runs.
For example, the below configuration will escalate to my on-call team when a custom field called Escalate to engineering? is set to Yes. I can then expose this field within my incident declaration and update forms so that anyone declaring an incident can easily page an on-call engineer.
You can even go a step further here and dynamically escalate to a specific escalation path based on another custom field populated by the declarer of the incident. For a more detailed look at how to achieve this head over to this article.
💡You can also always manually escalate to an engineer using the /inc escalate command within an incident channel, or using the Escalate to Someone button within the incident homepage (shown below).
🔀 Escalation paths
Escalation paths tell us who you would like to reach out to and in what order when things are broken.
You'll be able to configure:
Who to notify on different levels of your escalation path (which can be schedules or individuals).
How long to try notifying contacts on a particular level before moving on to the next level.
How many times you'd like to repeat the path if escalations go unacknowledged.
If you do not have a schedule set up, you can create a schedule inline from the escalation path creation flow or follow the instructions below in the On-call > Schedules section of the dashboard.
📆 Schedules
Schedules determine when individuals from a team or group will be on-call. These can then be used in escalation paths.
Within your schedule creation, you will have the ability to configure:
When your handover will take place (i.e. every Monday at 9am)
When the schedule will be active (i.e. all day, only during business hours)
Which individuals will be on call, and if you want more than one person on call simultaneously
Additional rotas, which can support models such as:
Follow-the-sun (i.e. spread support across timezones)
Shadowing (i.e. onboarding new on-call individuals)
🤚 Your team will be able to create overrides for your schedules from the detail view of each schedule, from the Schedules tab under On-call, or using /inc cover in Slack.
✍️ To recap
We'd suggest running through the following checklist to get your On-call program up and running.
Add alert sources, with custom attributes
Configure your alert routes to create incidents
Create escalation paths
Set up on-call schedules
If you haven't already, connect your alert routes to an escalation path
Ensure your responders are fully set up to be on-call
🔔 Your responders will automatically receive notifications to get set-up with incident.io on-call when they are added to a schedule and/or an escalation path. And, they will have sensible defaults (i.e. notification rules) to ensure they are better prepared for when they start their first on-call shift.
If you have any questions or concerns about getting your on-call program configured, please drop us a message via your Slack Connect channel, on Intercom, or at [email protected].