Skip to main content
All CollectionsOn-call management
Getting started with On-call
Getting started with On-call

Learn more about alerts, escalation paths and schedules to help you get the most out of your on call experience.

incident.io Engineering Team avatar
Written by incident.io Engineering Team
Updated over a week ago

🌐 Overview

If you're ready to get started with your incident.io On-call program, you've come to the right place! This document will walk you through the components of setting up On-call for your organization, specifically with:

  • Alerts

    • Sources

    • Routes

  • Escalation paths

  • Schedules

We'd recommend you get set up in the order listed above, however, you can set these components up in whatever order is easiest for your organization.


πŸ”₯ Creating escalations

In order to page your on-call users we need to know when something has gone wrong.

You can trigger an escalation automatically through an alert route, or manually from an incident.

🚨 Escalating from an Alert

Alert sources

First, you'll need to set up the connection between incident.io and your alert source(s). We connect with a variety of sources including:

  • Monitoring tools such as Datadog or Prometheus

  • Service management tools like Zendesk

If we don’t have an integration in place for your preferred source, you can connect via HTTP. This is a useful option when we don't integrate directly with your tool, but you are able to send us an HTTP request.

Once you have selected the source you’d like to connect you'll find detailed instructions on how to set up your integration with incident.io.

Custom alert attributes

When connecting your alert sources to incident.io, you can add custom attributes to provide more context to your alerts. Examples include: team, affected customer, affected feature or environment.

These can provide helpful context for responders when they receive the alert. In addition, attributes can be used to group and filter your alerts when they're being used to create incidents.

πŸ’‘ We have a more detailed end-to-end guide on getting started with Alerts here.

Alert Priorities

To use priorities in the escalation path branches, you can create them in Alert Attribute settings. Priorities are now a first-class citizen in incident.io, which means that you wouldn't need to create those as a custom field in the catalog just to connect with your alert source.

πŸ’‘ More details on setting up priorities in this article.

Alert routes

Once you have connected your alert sources, to automatically create incidents and to escalate accordingly, you’ll need to set up your alert routes. With alert routes you can:

  • Select one (or multiple!) alert sources to auto-create incidents

  • Filter any alerts you do not want to auto-create incidents for (e.g. low priority alerts)

  • Group alerts, so that when an alert fires, we find active incidents with similar alerts and give responders the ability to attach the alert to an incident or create a new incident from that alert (ie. grouping alerts by the Team that owns the alert)

  • Configure what your incidents look like (ie. populating custom fields from alert data)

  • Determine how and to whom you would like to escalate based on your alert configuration

You can create a new escalation path directly from your alert route or if you prefer to set it up later, you can follow the instructions below in the On-call > Escalation paths section of the dashboard.

πŸͺ„ It's worth noting that you don't need to create an alert route for each team you want to escalate to. For best practice on how to escalate to the correct team from an alert, please see this article here.

πŸ™‹ Escalating from a manually declared incident

It's not always possible to have an alert fire every time something goes wrong and so it's important to be able to page your on-call teams when someone manually declares an incident.

You can automatically escalate an incident within any given workflow using the Escalate via incident.io step. This then lets you choose an escalation path(s) or user(s) to escalate to when that workflow runs.
​
For example, the below configuration will escalate to my on-call team when a custom field called Escalate to engineering? is set to Yes. I can then expose this field within my incident declaration and update forms so that anyone declaring an incident can easily page an on-call engineer.

You can even go a step further here and dynamically escalate to a specific escalation path based on another custom field populated by the declarer of the incident. For a more detailed look at how to achieve this head over to this article.

πŸ’‘You can also always manually escalate to an engineer using the /inc escalate command within an incident channel, or using the Escalate to Someone button within the incident homepage (shown below).


πŸ”€ Escalation paths

Escalation paths tell us who you would like to reach out to and in what order when things are broken.

You'll be able to configure:

  • Who to notify on different levels of your escalation path (which can be schedules, individuals or Slack channels).

  • How long to try notifying contacts on a particular level before moving on to the next level.

  • How many times you'd like to repeat the path if escalations go unacknowledged.

  • You can also configure your escalation path further with working hours, priorities, and round robin: read more in our Smart escalation paths article.

If you do not have a schedule set up, you can create a schedule inline from the escalation path creation flow or follow the instructions below in the On-call > Schedules section of the dashboard.


πŸ“† Schedules

Schedules determine when individuals from a team or group will be on-call. These can then be used in escalation paths.

Within your schedule creation, you will have the ability to configure:

  • When your handover will take place (i.e. every Monday at 9am)

  • When the schedule will be active (i.e. all day, only during business hours)

  • Which individuals will be on call, and if you want more than one person on call simultaneously

  • Additional rotas, which can support models such as:

    • Follow-the-sun (i.e. spread support across timezones)

    • Shadowing (i.e. onboarding new on-call individuals)

🀚 Your team will be able to create overrides for your schedules from the detail view of each schedule, from the Schedules tab under On-call, or using /inc cover in Slack.


✍️ To recap

We'd suggest running through the following checklist to get your On-call program up and running.

  • Add alert sources, with custom attributes

  • Configure your alert routes to create incidents

  • Create escalation paths

  • Set up on-call schedules

  • If you haven't already, connect your alert routes to an escalation path

  • Ensure your responders are fully set up to be on-call

πŸ”” Your responders will automatically receive notifications to get set-up with incident.io on-call when they are added to a schedule and/or an escalation path. And, they will have sensible defaults (i.e. notification rules) to ensure they are better prepared for when they start their first on-call shift.

If you have any questions or concerns about getting your on-call program configured, please drop us a message via your Slack Connect channel, on Intercom, or at [email protected].

Did this answer your question?