Looks like our well-trained server monkeys dropped the ball. Rest assured they will be dealt with. In the meantime, you probably want to head home. -
| Severity | +Description | +What To Do | +
|---|---|---|
| Major | +
+
|
+ See During an Incident. | +
| Normal | +
+
|
+ See During an Incident. | +
| Anything above this line is considered a "Major Incident". These are generally Incidents (IN). Below are service requests (SR) which are usually initiated by a human who can help with prioritizing. A call is triggered for all major incidents (indifferently of SR or IN). | +||
| Normal | +
+
|
+
+
|
+
| Normal | +
+
|
+
+
|
+
| Low | +
+
|
+
+
|
+
| #incident-chat | +https://a-voip-provider.com/incident-call | ++1 555 BIG FIRE (+1 555 244 3473) / PIN: 123456 | +
Need an IC? Do !ic page in Slack |
+ ||
| For executive summary updates only, join #executive-summary-updates. | +||
Looks like our well-trained server monkeys dropped the ball. Rest assured they will be dealt with. In the meantime, you probably want to head home. +
This site documents parts of the Spearhead Systems Issue Response process. It is a cut-down version of our internal documentation, used at Spearhead Systems for any incident or service request, and to prepare new employees for on-call responsibilities. It provides information not only on preparation but also what to do during and after.
-Few companies seem to talk about their internal processes for dealing with major incidents. We would like to change that by opening up our documentation to the community, in the hopes that it proves useful to others who may want to formalize their own processes. Additionally, it provides an opportunity for others to suggest improvements, which ends up helping everyone.
-This documentation is complementary to what is available in our existing wiki.
-A collection of pages detailing how to efficiently deal with any incident or service request that might arise, along with information on how to go on-call effectively. It provides lessons learned the hard way, along with training material for getting you up to speed quickly.
-It is intended for on-call practitioners and those involved in an operational incident or service request response process, or those wishing to enact a formal incident response process. Specifically this is for all of our Technical Support staff.
-As a service provider Spearhead Systems deals with service requests on a daily basis. The reason we exist is to deliver a service which in most cases boils down to incidents and service requests. We want to deliver a smooth and seamless experience for resolving our customers issues therefore this documentation is a guideline for how we handle these requests. This documentation will allow you give you a head start on how to deal with issues in a way which leads to the fastest possible recovery time.
-Anything from preparing to go on-call, definitions of severities, incident call etiquette, all the way to how to run a post-mortem, providing a post-mortem template and even a security incident response process.
-Lots, dig in an help us complete the picture. We can migrate most processes from Sharepoint here.
-This documentation is provided under the Apache License 2.0. In plain English that means you can use and modify this documentation and use it both commercially and for private use. However, you must include any original copyright notices, and the original LICENSE file.
-Whether you are a Spearhead Systems customer or not, we want you to have the ability to use this documentation internally at your own company. You can view the source code for all of this documentation on our GitHub account, feel free to fork the repository and use it as a base for your own internal documentation.
- - - - -For every major incident (SEV-2/1), we need to follow up with a post-mortem. A blame-free, detailed description, of exactly what went wrong in order to cause the incident, along with a list of steps to take in order to prevent a similar incident from occurring again in the future. The incident response process itself should also be included.
-
The first step is that a post-mortem owner will be designated. This is done by the IC either at the end of a major incident call, or very shortly after. You will be notified directly by the IC if you are the owner for the post-mortem. The owner is responsible for populating the post-mortem page, looking up logs, managing the followup investigation, and keeping all interested parties in the loop. Please use Slack for coordinating followup. A detailed list of the steps is available below,
-As owner of a post-mortem, you are responsible for the following,
-Once you've been designated as the owner of a post-mortem, you should start updating the page with all the relevant information.
-(If not already done by the IC) Create a new post-mortem page for the incident.
-Schedule a post-mortem meeting for within 5 business days of the incident. You should schedule this before filling in the page, just so it's on the calendar.
-Begin populating the page with all of the information you have.
-Populate the page with more detailed information.
-Perform an analysis of the incident.
-Create any followup action JIRA tickets (or note down topics for discussion if we need to decide on a direction to go before creating tickets),
-Write the external message that will be sent to customers. This will be reviewed during the post-mortem meeting before it is sent out.
-These meetings should generally last 15-30 minutes, and are intended to be a wrap up of the post-mortem process. We should discuss what happened, what we could've done better, and any followup actions we need to take. The goal is to suss out any disagreement on the facts, analysis, or recommended actions, and to get some wider awareness of the problems that are causing reliability issues for us.
-You should invite the following people to the post-mortem meeting,
-A general agenda for the meeting would be something like,
-Here are some examples of post-mortems from other companies as a reference,
-This is a standard template we use for post-mortems at PagerDuty. Each section describes the type of information you will want to put in that section.
-Guidelines
-This page is intended to be reviewed during a post-mortem meeting that should be scheduled within 5 business days of any event. -Your first step should be to schedule the post-mortem meeting in the shared calendar for within 5 business days after the incident. -Don't wait until you've filled in the info to schedule the meeting, however make sure the page is completed by the meeting.
-Post-Mortem Owner: Your name goes here.
-Meeting Scheduled For: Schedule the meeting on the "Incident Post-Mortem Meetings" shared calendar, for within 5 business days after the incident. Put the date/time here.
-Call Recording: Link to the incident call recording.
-Include a short sentence or two summarizing the root cause, timeline summary, and the impact. E.g. "On the morning of August 99th, we suffered a 1 minute SEV-1 due to a runaway process on our primary database machine. This slowness caused roughly 0.024% of alerts that had begun during this time to be delivered out of SLA."
-Include a short description of what happened.
-Include a description of the root cause. If there were any actions taken that exacerbated the issue, also include them here with the intention of learning from any mistakes made during the resolution process.
-Include a description what solved the problem. If there was a temporary fix in place, describe that along with the long-term solution.
-Be very specific here, include exact numbers.
-| Time in SEV-1 | -?mins | -
|---|---|
| Notifications Delivered out of SLA | -??% (?? of ??) | -
| Events Dropped / Not Accepted | -??% (?? of ??) Should usually be 0, but always check | -
| Accounts Affected | -?? | -
| Users Affected | -?? | -
| Support Requests Raised | -?? Include any relevant links to tickets | -
Some important times to include: (1) time the root cause began, (2) time of the page, (3) time that the status page was updated (i.e. when the incident became public), (4) time of any significant actions, (5) time the SEV-2/1 ended, (6) links to tools/logs that show how the timestamp was arrived at.
-| Time (UTC) | -Event | -Data Link | -
|---|
Each action item should be in the form of a JIRA ticket, and each ticket should have the same set of two tags: “sev1_YYYYMMDD” (such as sev1_20150911) and simply “sev1”. Include action items such as: (1) any fixes required to prevent the root cause in the future, (2) any preparedness tasks that could help mitigate the problem if it came up again, (3) remaining post-mortem steps, such as the internal email, as well as the status-page public post, (4) any improvements to our incident response process.
-This is a follow-up for employees. It should be sent out right after the post-mortem meeting is over. It only needs a short paragraph summarizing the incident and a link to this wiki page.
---Briefly summarize what happened and where the post-mortem page (this page) can be found.
-
This is what will be included on the status.pagerduty.com website regarding this incident. What are we telling customers, including an apology? (The apology should be genuine, not rote.)
--- - - - -Summary
-What Happened?
-What Are We Doing About This?
-
You've just joined an incident call, and you've never been on one before. You have no idea what's going on, or what you're supposed to be doing. This page will help you through your first time on an incident call, and will provide a reference for future calls you may be a part of.
-
-Credit: Official White House Photo by Pete Souza
Use clear terminology, and avoid using acronyms or abbreviations during a call. Clear and accurate communication is more important than quick communication.
-
Standard radio voice procedure does not need to be followed on calls. However, you should familiarize yourself with the terms, as you may hear them on a call (or need to use them yourself). The ones in more active use on major incident calls are,
-Do not invent new abbreviations, and always favor being explicit of implicit. It is better to make things clearer than to try and save time by abbreviating, only to have a misunderstanding because others didn't know the abbreviation.
-The Incident Commander (IC) is the leader of the incident response process, and is responsible for bringing the incident to resolution. They will announce themselves at the start of the call, and will generally be doing most of the talking.
-Ask on the call if an IC is present. If you have no response, type !ic page in Slack. This will page the primary and backup IC to the call.
You're welcome to join only one of the channels, however you should not actively participate in the incident response if so, as it causes disjoined communication. Liaise with someone who is both in Slack and on the call to provide any input you may have so that they can raise it.
- - - - -There are several roles for our incident response teams at Spearhead Systems. Certain roles only have one person per incident (e.g. support engineer), whereas other roles can have multiple people (e.g. Sysadmins, Solution Architects, etc.). It's all about coming together as a team, working the problem, and getting a solution quickly.
-Here is a rough outline of our role hierarchy, with each role discussed in more detail on the rest of this page.
-
A Team Leader acts as the single source of truth of what is currently happening and what is going to happen during an major incident. They come in all shapes, sizes, and colors. TL's are also the key elements in a project (boards in DoIT).
-As any system grows in size and complexity, things break and cause incidents. The TL is needed to help drive major incidents to resolution by organizing his team towards a common goal.
-Anyone on the TL on-call schedule. Trainees are typically on the TL Shadow schedule.
-Take a look at our Team Leader training guide.
-A Sysadmin is a direct support role for the Team Leader. This is not a shadow where the person just observes, the Sysadmin is expected to perform important tasks during an incident.
-It's important for the TL to focus on the problem at hand, rather than worrying about documenting the steps or monitoring timers. The Sysadmin helps to support the TL and keep them stay focussed on the incident.
-The Sysadmin is expected to:
-Any Team Leader can act as a Sysadmin. Sysadmins need to be trained as an Team Leader as they may be required to take over command.
-Take a look at our Sysadmin training guide. Sysadmins also need to be trained as an Team Leaders.
-TODO:::move scribe responsibilities to TL and Sysadmin -::: or assign this to our juniors?
-A Scribe documents the timeline of an incident as it progresses, and makes sure all important decisions and data are captured for later review.
-The incident commander will need to focus on the problem at hand, and the subject matter experts will need to focus on resolving the incident. It is important to capture a timeline of events as they happen so that they can be reviewed during the post-mortem to determine how well we performed, and so we can accurate determine any additional impact that we might not have noticed at the time.
-The Scribe is expected to:
-Anyone can act as a scribe during an incident, and are chosen by the Incident Commander at the start of the call. Typically the Deputy will act as the Scribe, but that doesn't necessarily need to happen, and for larger incidents may not be possible.
-Follow our Scribe training guide, and then notify the Incident Commanders that you would like to be considered for scribing for the next incident.
-TODO::: END move scribe responsibilities to TL and Sysadmin
-A Subject Matter Expert (SME), sometimes called a "Resolver" or "Architect", is a domain expert or designated owner of a component or service that is part of the Spearhead Systems service delivery concept.
-The TL and Sysadmins are not all-knowing super beings. When there is a problem with a service or a particular system, an expert in that service is needed to be able to quickly help identify and fix issues.
-Anyone who is considered a "domain expert" can act as a resolver for an incident. Typically the service's primary on-call will act as the SME for that service.
-Take a look at our Subject Matter Expert training guide. You should also discuss with your team and service owner to determine what the requirements are for your particular service.
-A person responsible for interacting with customers, either directly, or via our public communication channels. Typically a member of the Customer Support team.
-All of the other roles will be actively working on identifying the cause and resolving the issue, we need a role which is focused purely on the customer interaction side of things so that it can be done properly, with the due care and attention it needs.
-Any member of the Support Team can act as a customer liaison.
-Discuss with the Support Team about becoming our next customer liaison.
- - - - -The first step in any incident response process is to determine what actually constitutes an incident. We have two high level categories for classifying incidents: this is done using "SR" or "IN" defintions with an attached priority of "Minor", "Normal" or "Major". "SR" are "Service requests" initiated by a customer and usually do not constitute a critical issue (there are exceptions) and "IN" are "incidents" which are generally "urgent".
-All of our operational issues are to be classified as either a Service Request or an Incident. Incidents have priority over Service Requests provided that there are no Service Requests with a higher priority. In general you will want to resolve a higher severity SR or IN than a lower one (a "Major" priority gets a more intensive response than a "Normal" incident for example).
-Always Assume The Worst
-If you are unsure which level an incident is (e.g. not sure if IN is Major or Normal), treat it as the higher one. During an incident is not the time to discuss or litigate severities, just assume the highest and review during a post-mortem.
-| Severity | -Description | -What To Do | -
|---|---|---|
| Major | -
-
|
- See During an Incident. | -
| Normal | -
-
|
- See During an Incident. | -
| Anything above this line is considered a "Major Incident". These are generally Incidents (IN). Below are service requests (SR) which are usually initiated by a human who can help with prioritizing. A call is triggered for all major incidents (indifferently of SR or IN). | -||
| Normal | -
-
|
-
-
|
-
| Normal | -
-
|
-
-
|
-
| Low | -
-
|
-
-
|
-
Be Specific
-When creating Cards in Doit, be as specific as possible and include all necessary details. Include relevant details regarding when the issue started, what may have triggered it, etc.. Document your efforts through worklogs and be specific there as well.
-| Severity | +Description | +What To Do | +
|---|---|---|
| Major | +
+
|
+ See During an Incident. | +
| Normal | +
+
|
+ See During an Incident. | +
| Anything above this line is considered a "Major Incident". These are generally Incidents (IN). Below are service requests (SR) which are usually initiated by a human who can help with prioritizing. A call is triggered for all major incidents (indifferently of SR or IN). | +||
| Normal | +
+
|
+
+
|
+
| Normal | +
+
|
+
+
|
+
| Low | +
+
|
+
+
|
+
| #incident-chat | +https://a-voip-provider.com/incident-call | ++1 555 BIG FIRE (+1 555 244 3473) / PIN: 123456 | +
Need an IC? Do !ic page in Slack |
+ ||
| For executive summary updates only, join #executive-summary-updates. | +||
Information on what to do during a major incident. See our severity level descriptions for what constitutes a major incident.
-Documentation
-For your own internal documentation, you should make sure that this page has all of the necessary information prominently displayed. Such as: phone bridge numbers, Slack rooms, important chat commands, etc. Here is an example,
-| #incident-chat | -https://a-voip-provider.com/incident-call | -+1 555 BIG FIRE (+1 555 244 3473) / PIN: 123456 | -
Need an IC? Do !ic page in Slack |
- ||
| For executive summary updates only, join #executive-summary-updates. | -||
Security Incident?
-If this is a security incident, you should follow the Security Incident Response process.
-Join the incident call and chat (see links above).
-Follow along with the call/chat, add any comments you feel are appropriate, but keep the discussion relevant to the problem at hand.
-Follow instructions from the Incident Commander.
-
!ic page in Slack. This will page the primary and backup IC's at the same time.Resolve the incident as quickly and as safely as possible, use the Deputy to assist you. Delegate any tasks to relevant experts at your discretion.
-Announce on the call and in Slack that you are the incident commander, who you have designated as deputy (usually the backup IC), and scribe.
-Identify if there is an obvious cause to the incident (recent deployment, spike in traffic, etc.), delegate investigation to relevant experts,
-Identify investigation & repair actions (roll back, rate-limit services, etc) and delegate actions to relevant service experts. Typically something like this (obviously not an exhaustive list),
-Listen for prompts from your Deputy regarding severity escalations, decide whether we need to announce publicly, and instruct customer liaison accordingly.
-Once incident has recovered or is actively recovering, you can announce that the incident is over and that the call is ending. This usually indicates there's no more productive work to be done for the incident right now.
-(After call ends) Create the post-mortem page from the template, and assign an owner to the post-mortem for the incident.
-(After call ends) Send out an internal email explaining that we had a major incident, provide a link to the post-mortem.
-You are there to support the IC in whatever they need.
-Monitor the status, and notify the IC if/when the incident escalates in severity level,
-!status - Will tell you the current status.!status stalk - Will continually monitor the status and report it to the room every 30s.Be prepared to page other people as directed by the Incident Commander.
-Provide regular status updates in Slack (roughly every 30mins) to the executive team, giving an executive summary of the current status. Keep it short and to the point, and use @here.
-Follow instructions from the Incident Commander.
-You are there to document the key information from the incident in Slack.
-Update the Slack room with who the IC is, who the Deputy is, and that you're the scribe (if not already done).
-You should add notes to Slack when significant actions are taken, or findings are determined. You don't need to wait for the IC to direct this - use your own judgment.
-TODO notes to the Slack room that indicate follow-ups slated for later.Follow instructions from the Incident Commander.
-You are there to support the incident commander in identifying the cause of the incident, suggesting and evaluation repair actions, and following through on the repair actions.
-Investigate the incident by analyzing any graphs or logs at your disposal. Announce all findings to the incident commander.
-Announce all suggestions for resolution to the incident commander, it is their decision on how to proceed, do not follow any actions unless told to do so!
-Follow instructions from the incident commander.
-(Optional) Once the call is over and post-mortem is created, add any notes you think are relevant to the post-mortem page.
-Be on stand-by to post public facing messages regarding the incident.
-You will typically be required to update the status page and to send Tweets from our various accounts at certain times during the call.
-Follow instructions from the Incident Commander.
-Incident Commander Required
-As with all major incidents at PagerDuty, security ones will also involve an Incident Commander, who will delegate the tasks to relevant resolvers. Tasks may be performed in parallel as assigned by the IC. Page one at the earliest possible opportunity.
-Details for each of these items are available in the next section.
-Stop the attack as quickly as you can, via any means necessary. Shut down servers, network isolate them, turn off a data center if you have to. Some common things to try,
-kill -9 any active session you think is an attacker.sudo shutdown nowIdentify the likely attack vectors and path/fix them so they cannot be re-exploited immediately after stopping the attack.
-Identify the key responders for the security incident, and keep them all in the loop. Set up a secure method of communicating all information associated with the incident. Details on the incident (or even the fact that an incident has occurred) should be kept private to the responders until you are confident the attack is not being triggered internally.
-Any instances which were affected by the attack should be immediately isolated from any other instances. As soon as possible, an image of the system should be taken and put into a read-only cold storage for later forensic analysis.
-Work with all tools at your disposal to identify the timeline of the attack, along with exactly what the attacker did.
-Using forensic analysis of log files, time-series graphs, and any other information/tools at your disposal, attempt to identify what information was compromised (if any),
-Based on the data that was compromised, assess the risk to other systems.
-Start applying mitigations to other parts of your system.
-Once you are confident the systems are secured, and enough monitoring is in place to detect another attack, you can move onto the forensic analysis stage.
-Delegate to: VP or Director of Engineering
-Communicate internally only once you are confident (via forensic analysis) that the attack was not sourced internally.
-Delegate to: VP or Director of Engineering
-Work with law enforcement to identify the source of the attack, letting any system owners know that systems under their control may be compromised, etc.
-Delegate to: Marketing Team
-Once you have validated all of the information you have is accurate, have a timeline of events, and know exactly what information was compromised, how it was compromised, and sure that it won't happen again. Only then should you prepare and release a public statement to customers informing them of the compromised information and any steps they need to take.
-This documentation covers parts of the Spearhead Systems Issue Response process. It is a copy of PagerDuty's documentation and furthermore a cut-down version of our own internal documentation, used at Spearhead Systems for any issue (incident or service request), and to prepare new employees for on-call responsibilities. It provides information not only on preparing for an incident, but also what to do during and after. It is intended to be used by on-call practitioners and those involved in an operational incident response process (or those wishing to enact a formal incident response process). See the about page for more information on what this documentation is and why it exists. This documentation is complementary to what is available in our existing wiki and may not yet be open sourced.
-Issue, Incident and Service Request
-At Spearhead we use the term issue to define any request from our customers. Issues fall into two categories: "Service Requests (SR)" and "Incidents (IN)". Note that we use the term Incident to describe both a service request as well as incidents. For brevity we will use SR and IN throughout this documentation.
-A "service request" is usually initiated by a human and is generally not critical for the normal functioning of the business while an "incident" is an issue that is or can cause interruption to normal business functions.
-
If you've never been on-call before, you might be wondering what it's all about. These pages describe what the expectations of being on-call are, along with some resources to help you.
-Reading material for things you probably want to know before an incident occurs. You likely don't want to be reading these during an actual incident.
-Information and processes during an incident.
-Our followup processes, how we make sure we don't repeat mistakes and are always improving.
-So, you want to learn about incident response? You've come to the right place.
-Useful material and resources from external parties that are relevant to incident response.
-{{summary}}
-No results found
"); - } - - if(jQuery){ - /* - * We currently only automatically hide bootstrap models. This - * requires jQuery to work. - */ - jQuery('#mkdocs_search_modal a').click(function(){ - jQuery('#mkdocs_search_modal').modal('hide'); - }); - } - - }; - - var search_input = document.getElementById('mkdocs-search-query'); - - var term = getSearchTerm(); - if (term){ - search_input.value = term; - search(); - } - - search_input.addEventListener("keyup", search); - -}); diff --git a/mkdocs/js/text.js b/mkdocs/js/text.js deleted file mode 100644 index 17921b6..0000000 --- a/mkdocs/js/text.js +++ /dev/null @@ -1,390 +0,0 @@ -/** - * @license RequireJS text 2.0.12 Copyright (c) 2010-2014, The Dojo Foundation All Rights Reserved. - * Available via the MIT or new BSD license. - * see: http://github.com/requirejs/text for details - */ -/*jslint regexp: true */ -/*global require, XMLHttpRequest, ActiveXObject, - define, window, process, Packages, - java, location, Components, FileUtils */ - -define(['module'], function (module) { - 'use strict'; - - var text, fs, Cc, Ci, xpcIsWindows, - progIds = ['Msxml2.XMLHTTP', 'Microsoft.XMLHTTP', 'Msxml2.XMLHTTP.4.0'], - xmlRegExp = /^\s*<\?xml(\s)+version=[\'\"](\d)*.(\d)*[\'\"](\s)*\?>/im, - bodyRegExp = /We manage how we get alerted based on many factors such as the customers contractual SLA, the urgency of their request or incident, etc.. an alert or notification is something which requires a human to perform an action. Based on the severity of the issue (service request or incident) we prioritize accordingly in DoIT.
-Major Priority Alerts
-Anything that wakes up a human in the middle of the night should be immediately human actionable. If it is none of those things, then we need to adjust the alert to not page at those times.
-| Priority | -Alerts | -Response | -
|---|---|---|
| Major | -Major-Priority Spearhead Alert 24/7/365. | -Requires immediate human action. | -
| Normal | -Normal-Priority Spearhead Alert during business hours only. | -Requires human action that same working day. | -
| Minor | -Minor-Priority Spearhead Alert 24/7/365. | -Requires human action at some point. | -
| Notification | -Suppressed Events. No response required. | -Informational only. We do not need these to clutter out ticketing or inboxes. If they are enabled they should be sent only to required/specific people, not groups. | -
Both IN and SR (incidents, service requests) share the same priorities. The actual response / resolution times vary and are based upon contractual agreements with the customer. These details (SLA) are available in DoIT on the organization page of the respective customer.
-If you're setting up a new alert/notification, consider the chart above for how you want to alert people. Be mindful of not creating new high-priority alerts if they don't require an immediate response, for example.
-Alert Channels
-Presently we use email as the only notification method. This means keeping an eye on your email is essential! -SMS and Push notifications are in the pipeline for DoIT.
-This would be a Major priority IN, requiring immediate human action to resolve.
-
This would be a Normal priority SR, requiring human action soon, but not immediately.
-
This would be a Minor priority SR, requiring human action some time soon.
-
A summary of expectations and helpful information for being on-call.
-
Being on-call means that you are able to be contacted at any time in order to investigate and fix issues that may arise. For example, if you are on-call, should any alarms be triggered by our monitoring solution, you will receive a "page" (an alert on your mobile device, email, phone call, or SMS, etc.) giving you details on what has broken. You will be expected to take whatever actions are necessary in order to resolve the issue and return your service to a normal state.
-At Spearhead Systems we consider you are on-call during normal working hours in which case you are proactively working with DoIT and looking over your assigned cards/boards as well as when you are formally "on-call" and issues are being redirected to you.
-On-call responsibilities extend beyond normal office hours, and if you are on-call you are expected to be able to respond to issues, even at 2am. This sounds horrible (and it can be), but this is what our customers go through, and is the problem that the Spearhead Systems professional services is trying to fix!
-Prepare
-Triage
-Fix
-Improve
-Support
-No expectation to be the first to acknowledge all of the alerts during the on-call period.
-No expectation to fix all issues by yourself.
-If your team is starting its own on-call rotation, here are some scheduling recommendations from the Operations team.
-Always have a backup schedule. Yes, this means two people being on-call at the same time, however it takes a lot of the stress off of the primary if they know they have a specific backup they can contact, rather than trying to chose a random member of the team.
-The third-level of your escalation (after backup schedule) should probably be your entire team. This should hopefully never happen (it's happened once in the history of the Support team), but when it does, it's useful to be able to just get the next available person.
-
Team managers can (and should) be part of your normal rotation. It gives a better insight into what has been going on.
-New members of the team should shadow your on-call rotation during the first few weeks. They should get all alerts, and should follow along with what you are doing. (All new employees shadow the Support team for one week of on-call, but it's useful to have new team members shadow your team rotations also. Just not at the same time).
-We recommend you set your escalation timeout to 5 minutes. This should be plenty of time for someone to acknowledge the incident if they're able to. If they're not able to within 5 minutes, then they're probably not in a good position to respond to the incident anyway.
-When going off-call, you should provide a quick summary to the next on-call about any issues that may come up during their shift. A service has been flapping, an issue is likely to re-occur, etc. If you want to be formal, this can be a written report via email, but generally a verbal summary is sufficient.
-You are free to set up your notification rules as you see fit, to match how you would like to best respond to incidents. If you're not sure how to configure them, the Support team has some recommendations,
-
If the current on-call comes into the office at 12pm looking tired, it's not because they're lazy. They probably got paged in the night. Cut them some slack and be nice.
-Don't acknowledge an incident out from under someone else. If you didn't get paged for the incident, then you shouldn't be acknowledging it. Add a comment with your notes instead.
-
If you are testing something, or performing an action that you know will cause a page (notification, alert), it's customary to "take the pager" for the time during which you will be testing. Notify the person on-call that you are taking the pager for the next hour while you test.
-"Never hesitate to escalate" - Never feel ashamed to rope in someone else if you're not sure how to resolve an issue. Likewise, never look down on someone else if they ask you for help.
-Always consider covering an hour or so of someone else's on-call time if they request it and you are able. We all have lives which might get in the way of on-call time, and one day it might be you who needs to swap their on-call time in order to have a night out with your friend from out of town.
-If an issue comes up during your on-call shift for which you got paged, you are responsible for resolving it. Even if it takes 3 hours and there's only 1 hour left of your shift. You can hand over to the next on-call if they agree, but you should never assume that's possible.
-Looks like our well-trained server monkeys dropped the ball. Rest assured they will be dealt with. In the meantime, you probably want to head home. +
So you want to be a deputy? You've come to the right place!
-
-Credit: oregondot @ Flickr
The purpose of the Deputy is to support the IC by keeping track of timers, notifying the IC of important information, and paging other people as directed by the IC.
-It's important for the IC to focus on the problem at hand, rather than worrying about monitoring timers. The deputy is there to help support the IC and keep them focussed on the incident.
-As a Deputy, you will be expected to take over command from the IC if they request it.
-You should not be performing any remediations, checking graphs, or investigating logs. Those tasks will be delegated to the resolvers by the IC.
-Before you can be a Deputy, it is expected that you meet the following criteria. Don't worry if you don't meet them all yet, you can still continue with training!
-Read up on our Different Roles for Incidents to see what is expected from a Deputy, as well as what we expect from the other roles you'll be interacting with.
-The training process for a Deputy is quite simple.
-The Steps for Deputy provide a detailed description of what you should be doing during an incident.
-Here are some examples of phrases and patterns you should use during incident calls.
-As you listen to the call, you should keep track of the responders to the call as you hear them speak. Make a note on a piece of paper, or use the !ic responders to see who they are. The IC may ask you who is on-call for a particular system, and you should know the answer, and be able to page them.
--Do we have a representative from [X] on the call?
-(pause)
-Deputy, can you go ahead and page the [X] on-call please.
-
You can page them however you see fit, phone call, etc.
-Provide regular status updates on Slack (roughly every 30mins), giving an executive summary of the current status during SEV-1 incidents. Keep it short and to the point, and use @here. Mention the current state, the actions in progress, customer impact, and expected time remaining. It's OK to miss out some of those if the information isn't known.
---@here: We are in SEV-1 due to X. Current actions in progress are to do Y. Expecting 3 mins to complete that action. Once action is complete, system should recover on its own within 5 minutes.
-
You are expected to keep track of how long the incident has been running for, and provide callouts to the IC every 10 minutes so they can take actions such as increasing the severity, or asking Support to Tweet out. This is as simple as telling the IC on the call,
---IC, be advised the incident is now at the 10 minute mark.
-
Similarly, when the IC asks for someone to get back to them in X minutes, you are expected to keep track of that. You should remind the IC when that time has been reached.
--- - - - -IC, be advised the timer for [TEAM]'s investigation is up.
-
Ever wonder what all of those strange words you sometimes see in our documentation mean? This page is here to help.
-| Term | -Description | -
|---|---|
| IC / Incident Commander | -The incident commander is the person responsible for bringing any major incident to resolution. They are the highest ranking individual on any major incident call, regardless of their day-to-day rank. Their decisions made as commander are final. More info. | -
| Deputy | -Typically the backup IC. The deputy's job is to support the IC during the call, providing them with any help they need. More info. | -
| Scribe | -The scribe's job is to keep a log of all activities performed during the call in a written chat log on Slack. More info. | -
| Resolver | -A person on the incident call who is able to help resolve issues within a particular system. Also referred to as an SME (see below). More info. | -
| SME | -"Subject Matter Expert", someone who is an expert in a particular service or subject who can provide information to the IC, and perform resolution actions for a particular system. More info. | -
| CAN Report | -CAN stands for "Conditions" "Actions" "Needs", if an IC asks you for a CAN report, you should provide the current state of your service (condition), what actions need to be taken to return it to a healthy state (actions), and what support you need in order to perform the actions (needs). | -
| Sev / Severity | -How severe the incident is. The "sev" of an incident determines the type of response we give. The higher the severity, the higher the likelihood of making risky actions to resolve the situation. More info. | -
| Span of Control | -Refers to the number of direct reports you have. For example, if the IC has 10 people as direct reports on a call, they have a large span of control. We aim to make the span of control as minimal as we can while still being productive. | -
| Grenade Thrower | -Someone who joins the call at a late time in the game, and provides information that completely derails the current thinking. They then leave almost immediately. | -
| Executive Swoop | -When an executive comes on the call and drops some sort of bombshell. A version of grenade throwing. | -
So you want to be an incident commander? You've come to the right place! You don't need to be a senior team member to become an IC, anyone can do it providing you have the requisite knowledge (yes, even an intern)!
-
-Credit: NASA
If you could boil down the definition of an Incident Commander to one sentence, it would be,
---Take whatever actions are necessary to protect PagerDuty systems and customers.
-
The purpose of the Incident Commander is to be the decision maker during an major incident; Delegating tasks and listening to input from subject matter experts in order to bring the incident to resolution.
-The Incident Commander becomes the highest ranking individual on any major incident call, regardless of their day-to-day rank. Their decisions made as commander are final.
-Your job as an IC is to listen to the call and to watch the incident Slack room in order to provide clear coordination, recruiting others to gather context/details. You should not be performing any actions or remediations, checking graphs, or investigating logs. Those tasks should be delegated.
-Before you can be an Incident Commander, it is expected that you meet the following criteria. Don't worry if you don't meet them all yet, you can still continue with training!
-Read up on our Different Roles for Incidents to see what is expected from an Incident Commander, as well as what we expect from the other roles you'll be interacting with.
-Some qualities we expect from an effective leader include being able to:
-As a leader, you should try to:
-The process is fairly loose for now. Here's a list of things you can do to train though,
-Read the rest of this page, particularly the sections below.
-Participate in Failure Friday (FF).
-Play a game of "Keep Talking and Nobody Explodes" with other people in the office.
-Shadow a current incident commander for at least a full week shift.
-Reverse shadow a current incident commander for at least a full week shift.
-What's the difference between an IC in training, and an IC? (This isn't the set up to a joke). Simple, an IC puts themselves on the schedule.
-Every incident is different (we're hopefully not repeating the same issue multiple times!), but there's a common process you can apply to each one.
-Identify the symptoms.
-Size-up the situation.
-Stabilize the incident.
-Provide regular updates.
-The deputy for an incident is generally the backup Incident Commander. However, as an Incident Commander, you may appoint one or more Deputies. Note that Deputy Incident Commanders must be as qualified as the Incident Commander, and that if a Deputy is assigned, he or she must be fully qualified to assume the Incident Commander’s position if required.
-Sharing information during an incident is a critical process. As an Incident Commander (or Deputy), you should be prepared to brief others as necessary. You will also be required to communicate your intentions and decisions clearly so that there is no ambiguity in your commands.
-When given information from a responder, you should clearly acknowledge that you have received and understood their message, so that the responder can be confident in moving on to other tasks.
-After an incident, you should communicate with other training Incident Commanders on any debrief actions you feel are necessary.
-The Steps for Incident Commander provide a detailed description of what you should be doing during an incident.
-Additionally, aside from following the usual incident call etiquette, there a few extra etiquette guidelines you should follow as IC:
-Here are some examples of phrases and patterns you should use during incident calls.
-At the start of any major incident call, the incident commander should announce the following,
---This is [NAME], I am the Incident Commander for this call.
-
This establishes to everyone on the call what your name is, and that you are now the commander. You should state "Incident Commander" and not "IC", as newcomers may not be familiar with the terminology yet. The word "commander" makes it very clear that you're in charge.
-If you are trained to be an IC and have joined a call, even if you aren't the IC on-call, you should do the following,
---Is there an IC on the call?
-(pause)
-Hearing no response, this is [NAME], and I am now the Incident Commander for this call.
-
If the on-call IC joins later, you may hand over to them at your discretion (see below for the hand-off procedure)
-During a call, you will want to know who is available from the various teams in order to resolve the incident. Etiquette dictates that people should announce themselves, but sometimes you may be joining late to the call. If you need a representative from a team, just ask on the call. Your deputy can page one if no one answers.
---Do we have a representative from [X] on the call?
-(pause)
-Deputy, can you go ahead and page the [X] on-call please.
-
When you need to give out an assignment or task, give it to a person directly, never say "can someone do..." as this leads to the bystander effect. Instead, all actions should be assigned to a specific person, and time-boxed with a specific number of minutes.
---IC: Bob, please investigate the high latency on web app boxes. I'll come back to you for an answer in 3 minutes.
-Bob: Understood
-
Keep track of how many minutes you assigned, and check in with that person after that time. You can get help from your deputy to help track the timings.
-If a decision needs to be made, it comes down to the IC. Once the IC makes a decision, it is final. But it's important that no one can come later and object to the plan, saying things like "I knew that would happen". An IC will use very specific language to be sure that doesn't happen.
---The proposal is to [EXPLAIN PROPOSAL]
-Are there any strong objections to this plan?
-(pause)
-Hearing no objects, we are proceeding with this proposal.
-
If you were to ask "Does everyone agree?", you'd get people speaking over each other, you'd have quiet people not speaking up, etc. Asking for any STRONG objections gives people the chance to object, but only if they feel strongly on the matter.
-It's important to maintain a cadence during a major incident call. Whenever there is a lull in the proceedings, usually because you're waiting for someone to get back to you, you can fill the gap by explaining the current situation and the actions that are outstanding. This makes sure everyone is on the same page.
---While we wait for [X], here's an update of our current situation.
-We are currently in a SEV-1 situation, we believe to be caused by [X]. There's an open question to [Y] who will be getting back to us in 2 minutes. In the meantime, we have Tweeted out that we are experiencing issues. Our next Tweet will be in 10 minutes if the incident is still ongoing at that time.
-Are there any additional actions or proposals from anyone else at this time?
-
Transfer of command, involves (as the name suggests) transferring command to another Incident Commander. There are multiple reasons why a transfer of command might take place,
-Never feel like you are not doing your job properly by handing over. Handovers are encouraged. In order to handover, out of band from the main call (via Slack for example), notify the other IC that you wish to transfer command. Update them with anything you feel appropriate. Then announce on the call,
---Everyone on the call, be advised, at this time I am handing over command to [X].
-
The new IC should then announce on the call as if they were joining a new call (see above), so that everyone is aware of the new commander.
-Note that the arrival of a more qualified person does NOT necessarily mean a change in incident command.
-Often times on a call people will be talking over one another, or an argument on the correct way to proceed may break out. As Incident Commander it's important that order is maintained on a call. The Incident Commander has the power to remove someone from the call if necessary (even if it's the CEO). But often times you just need to remind people to speak one at a time. Sometimes the discussion can be healthy even if it starts as an argument, but you shouldn't let it go on for too long.
---(noise)
-Ok everyone, can we all speak one at a time please. So far I'm hearing two options to proceed: 1) [X], 2) [Y].
-Are there any other proposals someone would like to make at this time?
-...etc
-
You may ask a question as IC and receive an answer that doesn't actually answer your question. This is generally when you ask for a yes/no answer but get a more detailed explanation. This can often times be because the person doesn't understand the call etiquette. But if it continues, you need to take action in order to proceed.
---IC: Is this going to disable the service for everyone?
-SME: Well... for some people it....
-IC: Stop. I need a yes/no answer. Is this going to disable the service for everyone?
-SME: Well... it might not do...
-IC: Stop. I'm going to ask again, and the only two words I want to hear from you are "yes" or "no. If this going to disable the service for everyone?
-SME: Well.. like I was saying..
-IC: Stop. Leave the call. Backup IC can you please page the backup on-call for [service] so that we can get an answer.
-
You may get someone who would be senior to you during peacetime come on the call and start overriding your decisions as IC. This is unacceptable behaviour during wartime, as the IC is in command. While this is rare, you can get things back on track with the following,
---Executive: No, I don't want us doing that. Everyone stop. We need to rollback instead.
-IC: Hold please. [EXECUTIVE], do you wish to take over command?
-Executive: Yes/No
-(If yes) IC: Understood. Everyone on the call, be advised, at this time I am handling over command to [EXECUTIVE]. They are now the incident commander for this call.
-(If no) IC: In that case, please cause no further interruptions or I will remove you from the call.
-
This makes it clear to the executive that they have the option of being in charge and making decisions, but in order to do so they must continue as an Incident Commander. If they refuse, then remind them that you are in charge and disruptive interruptions will not be tolerated. If they continue, remove them from the call.
-At the end of an incident, you should announce to everyone on the call that you are ending the call at this time, and provide information on where followup discussion can take place. It's also customary to thank everyone.
---Ok everyone, we're ending the call at this time. Please continue any followup discussion on Slack. Thanks everyone.
-
PagerDuty employees have access to all previous incident calls, and can listen to them at their discretion. We can't release these calls, so for everyone else, here are some short examples from popular culture to show the techniques at work.
-Here's a clip from the movie Apollo 13, where Gene Kranz (Flight Director / Incident Commander) shows some great examples of Incident Command. Here are some things to note:
-Another clip from Apollo 13. Things to note:
-Learning about the Spearhead Systems incident response process is an important part of being an effective on-call engineer at Spearhead Systens. This section goes over our training material for the various roles that are involved in our incident response, along with some additional information and training material from government agencies.
-Our training guides are split up by role, however you are encouraged to read through the training guides even for roles you don't belong to, as it can give you some good insight into how those people will be behaving during major incidents.
-Our incident response process is loosely based on the US National Incident Management System (NIMS), which is described as,
-A systematic, proactive approach to guide departments and agencies at all levels of government, nongovernmental organizations, and the private sector to work together seamlessly and manage incidents involving all threats and hazards—regardless of cause, size, location, or complexity—in order to reduce loss of life, property and harm to the environment.
-While it might not initially seem that this would be applicable to an IT operations environment, we've found that many of the lessons learned from major incidents in these situations can be directly applied to our industry too. The principles are the same and span many different environments.
-
If you want to learn more about NIMS, we recommend the ICS-100 and ICS-700 online training courses, which go over NIMS and the Incident Command System (You can also take an online examination after training in order to get a certificate from FEMA). There is also a wealth of additional training material and courses from FEMA on NIMS, which I would encourage you to look at.
-If you're based in the US and interested in taking a more active incident response role in your community, we recommend investigating your local CERT programs (Community Emergency Response Teams). Many cities offer CERT training, after which you can volunteer as a CERT contributor within your community. Not only is it an opportunity to get real world experience with disaster response, but the skills you learn can be applied to everyday life too.
-Also take a look at the Additional Reading section on the home page.
- - - - -So you want to be a scribe? You've come to the right place! You don't need to be a senior team member to become a deputy or scribe, anyone can do it providing you have the requisite knowledge!
-
-Credit: Holly Chaffin
The purpose of the Scribe is to maintain a timeline of key events during an incident. Documenting actions, and keeping track of any followup items that will need to be addressed.
-It's important for the rest of the command staff to be able to focus on the problem at hand, rather than worrying about documenting the steps.
-Your job as Scribe is to listen to the call and to watch the incident Slack room, keeping track of context and actions that need to be performed, documenting these in Slack as you go. You should not be performing any remediations, checking graphs, or investigating logs. Those tasks will be delegated to the subject matter experts (SME's) by the Incident Commander.
-Before you can be a Scribe, it is expected that you meet the following criteria. Don't worry if you don't meet them all yet, you can still continue with training!
-Read up on our Different Roles for Incidents to see what is expected from a Scribe, as well as what we expect from the other roles you'll be interacting with.
-There is no formal training process for this role, reading this page should be sufficient for most tasks. Here's a list of things you can do to train though,
-Read the rest of this page, particularly the sections below.
-Participate in Failure Friday (FF).
-Scribing is more art than science. The objective is to keep an accurate record of important events that occurred on the call, so that we can look back at the timeline to see what happened. But what exactly is important? There's no overwhelming answer, and it really comes down the judgement and experience. But here are some general things you most definitely want to capture as scribe.
-The Steps for Scribe provide a detailed description of what you should be doing during an incident.
-Here are some examples of phrases and patterns you should use during incident calls.
-At the start of any major incident call, you should start our status stalking bot, so that it will post to the room an update automatically.
---!status stalk
-
This will provide the update and allow the IC to see the status without having to keep asking.
-During a call, you will hear lots of discussion happening, you should not be documenting all of this in the chat room. You only want to document things which will be important for the final timeline. It's not always obvious what this might be, and it's usually a matter of judgement. You generally want to note any actions the IC has asked someone to perform, along with the result of any polling decisions.
---Polled for decision on whether to perform rolling restart. We are proceeding with restart. [USER_A] to execute.
-
Some actions might seem important at the time, but end up not being. That's OK. It's better to have more info than not enough, but don't go overboard.
-Sometimes during the call, someone will either mention something we "should fix", or the IC will specifically ask you to note a followup item. You can do this in Slack by simply prefixing with "TODO", this will make it easier to search for later.
---TODO: Why did we not get paged for the fall in traffic on [X] cluster?
-
The post-mortem owner will find these after and raise tasks for them.
-When the IC ends the call, you should post a message into Slack to let everyone know the call is over, and that they should continue discussion elsewhere.
---Call is over, thanks everyone. Follow up in Slack.
-
Don't forget to also stop the status stalking.
--- - - - -!status unstalk
-
If you are on-call for any team at PagerDuty, you may be paged for a major incident and will be expected to respond as a subject matter expert (SME) for your service. This page details everything you need to know in order to be prepared for that responsibility. If you are interested in becoming an Incident Commander, take a look at the Incident Commander Training page.
-
-Credit: oregondot @ Flickr
If you are on-call for your team, there are certain expectations of you as that on-call. This applies to both the primary and secondary on-calls. Getting paged about a SEV-3 or SEV-4 in your system comes with different expectations than getting paged with a major SEV-2.
-When an incident occurs, you must be mobilized or assigned to become part of the incident response. In other words, until you are mobilized to the incident via a page or being directly asked by someone else on the incident, you remain in your everyday role. After being mobilized, your first task is to check in and receive an assignment. While it's tempting to see an incident happening and want to jump in and help, when resources show up that have not been requested, the management of the incident can be compromised.
-If you're not sure about something, it is perfectly acceptable to bring in other SMEs from your team that you believe know a given system better than you. Don't let your ego keep you from bringing in additional help. Our motto is "Never hesitate to escalate", you will never be looked down upon for escalating something because you didn't know how to handle it.
-There will be incidents. Some will be caused by you, some will be caused by others... some will just happen. Our entire incident response process is completely blameless. Blaming people is counter productive and just distracts from the problem at hand. No matter how an incident started, they all need to get solved as quickly as possible.
-Behavior during a major incident is very different to any other alert you may have received in the past. We call a major incident "wartime", and make a distinction between that and normal everyday operations ("peacetime").
-The organizational structure is generally based on seniority. The more senior members of a team will lead discussions, and managers or team leads will have the final say. Decisions are made after careful consideration of all options, and to minimize potential risk to customers.
-Wartime is different, and you will notice on our major incident calls that there's a different organizational structure.
-