All Collections
Creating and running incidents
FAQs
Can we automatically create incidents?
Can we automatically create incidents?

Programatically declaring incidents

incident.io Engineering Team avatar
Written by incident.io Engineering Team
Updated over a week ago

Yes! We support automatically creating incidents from many sources including:

To see if we support the source you want, try creating an alert source to see which options are available.

βš™οΈ Can't see the source you want?
You can use our HTTP source to connect with most tools that provide webhooks.

πŸ›  Setting Up

1. Connect an alert source

1️⃣ Create an alert source to receive alerts

2️⃣ Connect your source to receive alerts

  • Follow the steps provided for your chosen source to connect it and receive your first alert

  • You'll see live alerts come in once you start receiving them, you can try sending a test alert from your source to check your configuration

3️⃣ Configure your alerts

When connecting your alert sources to incident.io, you can add custom attributes to provide more context to your alerts. Examples include: team, affected customer, affected feature or environment. This allows you to pull values from your alert payload into rich data on your incidents.

These can provide helpful clues for responders when digging into the alert issue. Plus, it will be a helpful way of grouping and filtering your alerts into incidents via alert routes.


2. Create an alert route

Now that you have an alert source that's receiving alerts, you need to connect it to an alert route to start creating incidents from your alerts.

1️⃣ Select alert sources

  • Select any alert sources which should trigger incidents

  • You can filter which specific alerts should create incidents later

2️⃣ Filter and group

  • Filter and alerts which should not auto-create incidents for (i.e. low priority alerts)

  • Group alerts, so when an alert fires, we find active incidents with similar alerts and give responders the ability to attach the alert to an incident or create a new incident from that alert (ie. grouping alerts by the Team that owns the alert). More on that later.

3️⃣ Configure incidents

  • Choose how you want the incidents created from your alerts to look

  • You can select title and summary from the alert details you have available to you

  • Add any custom fields which should be set based on details on your alert


βš™οΈ How does it work?

Once auto-creation is switched on, incidents will be automatically created in a triage state.
We'll try our best to pull the right people into the incident channel.

We'll use PagerDuty to provide an example below. When joining a triage incident, you'll be met with this:

πŸ”₯ A new incident
This will accept this triage incident as a real incident.

  • You will be asked if you'd like to rename the incident, and you'll be able to set any custom fields you have configured

  • All your workflows will kick off once the incident is accepted, and you can get to work

  • The PagerDuty incident which triggered this will become an attachment of your incident.

πŸ”— Part of an existing incident
Was this alert caused by an incident with an existing channel that you already know about?
If so, you can attach the PagerDuty incident to any open incident channel.
​
The incident that was created in triage will be declined, and the channel will be archived after a short delay.
​

❌ Not an incident
Selecting this will decline the incident, and archive the channel.
​
Declined incidents are excluded from any metrics or workflows that you have set up. It's like they never existed at all.


πŸ‘©β€πŸ‘©β€πŸ‘§β€πŸ‘§ Grouping Windows

Sometimes multiple incidents or alerts in a short space of time, are caused by the same underlying problem. To avoid creating unnecessary noise, we recommend configuring a grouping window. This allows us to check in with you before we spin up new incidents automatically.
​
We'll continue to use PagerDuty for our example, but the same applies to Opsgenie alerts and the like:

  1. An incident is auto-created from a PagerDuty incident
    ​

  2. A second PagerDuty incident is triggered within the grouping window, with a matching service
    ​

  3. Instead of creating a new incident, we will post the below message in all open incidents with matching PagerDuty services created within the grouping window.

πŸ”— Part of this incident
This will attach this PagerDuty incident to the existing incident.

  • Any other grouping suggestion messages will be deleted from other channels, it's unlikely it's linked to multiple incidents

  • The incident will be acknowledged in PagerDuty
    ​

πŸ‘Ž Not related to this incident
This will reject the PagerDuty incident from this channel, and remove all references to it.
​
If the incident is rejected from all incident channels where it was suggested, a new incident will be created in triage and you can take it from there.
​

Did this answer your question?