Opsgenie Review

Problem 1#: Fatigue and incidents taking their toll on your team

University infrastructures can be incredibly complex; with eight campuses and a large residential population, there’s a lot to monitor. Charles, a lead support analyst, is overwhelmed by connecting all the homegrown systems and solutions that are tied together to monitor the health of the university’s infrastructure. These “solutions” send hundreds of emails a day to a single inbox that he then monitors and looks through, trying to decide what’s important and what might need his immediate attention. Due to the complexity and lack of automation, Charles set alarms periodically throughout the night every night to check the university’s systems and make sure he wasn’t missing some sort of outage. Since the monitoring inbox was such a mess, sometimes outages would be missed. For example, last month, an entire portion of the campus had intermittent WiFi for one week without him knowing.

After implementing Opsgenie and getting rid of the homegrown solutions that the university was using, Charles and his colleagues are immediately notified when there is a problem by phone call, text, or mobile push notification so everyone gets more sleep. Since Opsgenie integrates with all their monitoring, ITSM, and collaboration tools easily, using homegrown, disparate methods of monitoring isn’t necessary. Everything goes through Opsgenie which streamlines incident management and enables the team to free up their time. Because of the ease of building an on-call schedule directly in Opsgenie, Charles and his teammates can now share the load of being on-call. He also uses Heartbeats, an Opsgenie feature that ensures that Opsgenie stays connected to your monitoring tools, so he can rest without worry if he doesn’t receive any alerts.

Since implementing Opsgenie everyone is getting more sleep and on-call shifts are more evenly spread out and shared amongst team members. Alert fatigue is greatly reduced because instead of slogging through a clogged inbox, the team receives alerts in real time as problems arise.

Features: alerting, Heartbeats, on-call schedules.

Problem #2. The spectre of downtime

Juan just deployed a configuration change to the server, and within minutes, alerts are created. The API monitoring tool generated alerts in Opsgenie that indicated an uptick in the average API error rate. A customer reported being unable to access his streaming account via the service desk portal, then five more customers reported the same problem. An incident was opened immediately in Opsgenie’s incident command center, and Juan, who was the developer on-call, and the team were able to determine that the recent configuration change to the server was the cause, as there was a bug in the code. Juan was then able to roll back the change and fix the problem within 30 minutes. Because of the proactive alerting in place, the team was able to respond faster and then easily collaborate with the incident command center. Using Opsgenie for alerting and incident response reduced the amount of time to solve the incident by half, giving customers a better experience, and saving the team a lot of headaches.

Features: service-aware incident management, incident command center, alerting, on-call management.

Problem #3: Missed notifications

Jen was supposed to be the senior manager on-call for the IT operations team for the third week of every month, but in the third week of March, she needed to take off for a family wedding. After confirming with her colleague that they could switch weeks, she utilized her forwarding rules to ensure that any alerts that came in for her at the time would be forwarded to her colleague, Sarah. This meant that any time Jen’s team didn’t take ownership of alerts and respond in a timely manner, Sarah would be notified so that she could stay on top of the team. Jen however, would be blissfully on vacation.

Features: on-call, routing rules, escalation policies, forwarding rules.