All Collections
Creating and running incidents
FAQs
Can we automatically create incidents?
Can we automatically create incidents?

Programatically declaring incidents

incident.io Engineering Team avatar
Written by incident.io Engineering Team
Updated over a week ago

Yes!

We support automatically creating incidents from:

We're working hard on this and plan to support other monitoring tools soon.
Have a specific tool you'd like us to work with? let us know! 🙏🏼

🛠 Setting Up

1️⃣ Connect to one of the tools above

2️⃣ Enable an incident trigger by:

  • Navigate to Settings > Integrations

  • Select the tool that you're integrating with

  • Enable the incident trigger by toggling the 'Trigger incidents from ...' button

Note that we're using Opsgenie as the example below, but all of the available tools follow the same set up flow.

You can then choose configuration including:

  • Apply conditions to determine when we'll create an incident (e.g. only for a particular service)

  • Choose whether we should auto-decline incidents (e.g. if the alert is resolved in the original system)

  • Configure what default values the automatically created incident will have (e.g. what Severity or custom fields should be set).

  • You also select a grouping window. More on that later.

⚙️ How does it work?

Once auto-creation is switched on, incidents will be automatically created in a triage state.
We'll try our best to pull the right people into the incident channel.

We'll use PagerDuty to provide an example below. When joining a triage incident, you'll be met with this:

🔥 A new incident
This will accept this triage incident as a real incident.

  • You will be asked if you'd like to rename the incident, and you'll be able to set any custom fields you have configured

  • All your workflows will kick off once the incident is accepted, and you can get to work

  • The PagerDuty incident which triggered this will become an attachment of your incident.

🔗 Part of an existing incident
Was this alert caused by an incident with an existing channel that you already know about?
If so, you can attach the PagerDuty incident to any open incident channel.

The incident that was created in triage will be declined, and the channel will be archived after a short delay.

❌ Not an incident
Selecting this will decline the incident, and archive the channel.

Declined incidents are excluded from any metrics or workflows that you have set up. It's like they never existed at all.

👩‍👩‍👧‍👧 Grouping Windows

Sometimes multiple incidents or alerts in a short space of time, are caused by the same underlying problem. To avoid creating unnecessary noise, we recommend configuring a grouping window. This allows us to check in with you before we spin up new incidents automatically.

We'll continue to use PagerDuty for our example, but the same applies to Opsgenie alerts and the like:

  1. An incident is auto-created from a PagerDuty incident

  2. A second PagerDuty incident is triggered within the grouping window, with a matching service

  3. Instead of creating a new incident, we will post the below message in all open incidents with matching PagerDuty services created within the grouping window.

🔗 Part of this incident
This will attach this PagerDuty incident to the existing incident.

  • Any other grouping suggestion messages will be deleted from other channels, it's unlikely it's linked to multiple incidents

  • The incident will be acknowledged in PagerDuty

👎 Not related to this incident
This will reject the PagerDuty incident from this channel, and remove all references to it.

If the incident is rejected from all incident channels where it was suggested, a new incident will be created in triage and you can take it from there.

What's next?

If you have another monitoring tool (Datadog, Kibana, Sentry etc.) you'd like us to create incidents from, let us know! 🙏🏼

Did this answer your question?