mpana/spearhead-issue-response
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

refactored to sph specific use

Browse Source
This commit is contained in:
Marius Pana 2017-01-21 13:46:47 +02:00
parent d4511d9de6
commit 5b4f9a148f
1 changed files with 36 additions and 35 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

71
docs/during/during_an_incident.md
View File

@ -1,19 +1,19 @@
Information on what to do during a major incident. See our [severity level descriptions](/before/severity_levels.md) for what constitutes a major incident. Information on what to do during a major incident. See our [severity level descriptions](/before/severity_levels.md) for what constitutes a major incident.
!!! note "Documentation" !!! note "Documentation"
For your own internal documentation, you should make sure that this page has all of the necessary information prominently displayed. Such as: phone bridge numbers, Slack rooms, important chat commands, etc. Here is an example, Always document your activities. Keep a detailed worklog of your actions in DoIT and communicate verbosely on Slack or other channels (email, etc.).
<table class="custom-table" id="contact-summary"> <table class="custom-table" id="contact-summary">
<thead> <thead>
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><a href="#">#incident-chat</a></td> <td><a href="#">#support</a></td>
<td><a href="#">https://a-voip-provider.com/incident-call</a></td> <td><a href="#">http://response.spearhead.systems</a></td>
<td><a href="#">+1 555 BIG FIRE</a> (+1 555 244 3473) / PIN: 123456</td> <td><a href="#">+40728 005 263</a> </td>
</tr> </tr>
<tr> <tr>
<td colspan="3" class="centered">Need an IC? Do <code>!ic page</code> in Slack</td> <td colspan="3" class="centered">Need an TL? Do <code>!tl page</code> in Slack</td>
</tr> </tr>
<tr> <tr>
<td colspan="3"><em>For executive summary updates only, join <a href="#">#executive-summary-updates</a>.</em></td> <td colspan="3"><em>For executive summary updates only, join <a href="#">#executive-summary-updates</a>.</em></td>
@ -33,18 +33,22 @@ Information on what to do during a major incident. See our [severity level descr
1. Follow along with the call/chat, add any comments you feel are appropriate, but keep the discussion relevant to the problem at hand. 1. Follow along with the call/chat, add any comments you feel are appropriate, but keep the discussion relevant to the problem at hand.
* If you are not an SME, try to filter any discussion through the primary SME for your service. Too many people discussing at once get become overwhelming, so we should try to maintain a hierarchical structure to the call if possible. * If you are not an SME, try to filter any discussion through the primary SME for your service. Too many people discussing at once get become overwhelming, so we should try to maintain a hierarchical structure to the call if possible.
1. Follow instructions from the Incident Commander. 1. Follow instructions from the Team Leader.
* **Is there no IC on the call?** * **Is there no TL on the call?**
* Manually page them via Slack, with `!ic page` in Slack. This will page the primary and backup IC's at the same time. * Manually page them via Slack, with `!tl page` in Slack. This will page the primary and backup TL's at the same time.
* Never hesitate to page the IC. It's much better to have them and not need them than the other way around. * Never hesitate to page the TL. It's much better to have them and not need them than the other way around.
## Steps for Incident Commander !!! info "Not a call?"
Resolve the incident as quickly and as safely as possible, use the Deputy to assist you. Delegate any tasks to relevant experts at your discretion. Not all issues begin with a formal call. Some issues are self-explanatory and automatically generated via our monitoring platforms, a customer logging on to our portal, etc. In these scenarios [DoIT](http://doit.sphs.ro) is the definitive source. If that is not sufficient ask your TL.
1. Announce on the call and in Slack that you are the incident commander, who you have designated as deputy (usually the backup IC), and scribe.
## Steps for the Team Leader
Resolve the incident as quickly and as safely as possible, use the Sysadmin to assist you. Delegate any tasks to relevant experts at your discretion.
1. Announce on the call, in DoIT and in Slack that you are the team leader, who you have designated as sysadmin (usually the backup TL), and scribe/juniors if any.
1. Identify if there is an obvious cause to the incident (recent deployment, spike in traffic, etc.), delegate investigation to relevant experts, 1. Identify if there is an obvious cause to the incident (recent deployment, spike in traffic, etc.), delegate investigation to relevant experts,
* Use the service experts on the call to assist in the analysis. They should be able to quickly provide confirmation of the cause, but not always. It's the call of the IC on how to proceed in cases where the cause is not positively known. Confer with service owners and use their knowledge to help you. * Use the service experts on the call to assist in the analysis. They should be able to quickly provide confirmation of the cause, but not always. It's the call of the TL on how to proceed in cases where the cause is not positively known. Confer with service owners and use their knowledge to help you.
1. Identify investigation & repair actions (roll back, rate-limit services, etc) and delegate actions to relevant service experts. Typically something like this (obviously not an exhaustive list), 1. Identify investigation & repair actions (roll back, rate-limit services, etc) and delegate actions to relevant service experts. Typically something like this (obviously not an exhaustive list),
* **Bad Deployment:** Roll it back. * **Bad Deployment:** Roll it back.
@ -53,8 +57,8 @@ Resolve the incident as quickly and as safely as possible, use the Deputy to ass
* **Data Center Outage:** Validate automation has removed bad data center. Force it to do so if not. * **Data Center Outage:** Validate automation has removed bad data center. Force it to do so if not.
* **Degraded Service Behavior without load:** Gather forensic data (heap dumps, etc), and consider doing a rolling restart. * **Degraded Service Behavior without load:** Gather forensic data (heap dumps, etc), and consider doing a rolling restart.
1. Listen for prompts from your Deputy regarding severity escalations, decide whether we need to announce publicly, and instruct customer liaison accordingly. 1. Listen for prompts from your Sysadmin regarding severity escalations, decide whether we need to announce publicly, and instruct customer liaison accordingly.
* Announcing publicly is at your discretion as IC. If you are unsure, then announce publicly ("If in doubt, tweet it out"). * Announcing publicly is at your discretion as TL. If you are unsure, then announce publicly ("If in doubt, tweet it out").
1. Once incident has recovered or is actively recovering, you can announce that the incident is over and that the call is ending. This usually indicates there's no more productive work to be done for the incident right now. 1. Once incident has recovered or is actively recovering, you can announce that the incident is over and that the call is ending. This usually indicates there's no more productive work to be done for the incident right now.
* Move the remaining, non-time-critical discussion to Slack. * Move the remaining, non-time-critical discussion to Slack.
@ -66,46 +70,43 @@ Resolve the incident as quickly and as safely as possible, use the Deputy to ass
1. (After call ends) Send out an internal email explaining that we had a major incident, provide a link to the post-mortem. 1. (After call ends) Send out an internal email explaining that we had a major incident, provide a link to the post-mortem.
## Steps for Deputy ## Steps for Sysadmin
You are there to support the IC in whatever they need. You are there to support the TL in whatever they need.
1. Monitor the status, and notify the IC if/when the incident escalates in severity level, 1. Monitor the status, and notify the TL if/when the incident escalates in severity level.
* OfficerURL can help you to monitor the status on Slack,
* `!status` - Will tell you the current status.
* `!status stalk` - Will continually monitor the status and report it to the room every 30s.
1. Be prepared to page other people as directed by the Incident Commander. 1. Be prepared to page other people as directed by the Team Leader.
1. Provide regular status updates in Slack (roughly every 30mins) to the executive team, giving an executive summary of the current status. Keep it short and to the point, and use @here. 1. Provide regular status updates in Slack (roughly every 30mins) to the executive team, giving an executive summary of the current status. Keep it short and to the point, and use @here.
1. Follow instructions from the Incident Commander. 1. Follow instructions from the Team Leader.
## Steps for Scribe ## Steps for Scribe
You are there to document the key information from the incident in Slack. You are there to document the key information from the incident in Slack, DoIT, our WiKi, etc.
1. Update the Slack room with who the IC is, who the Deputy is, and that you're the scribe (if not already done). 1. Update the apropriate channel with who the TL is, who the Sysadmin is, and that you're the scribe (if not already done).
* e.g. "IC: Bob Boberson, Deputy: Deputy Deputyson, Scribe: Writer McWriterson" * e.g. "TL: Bob Boberson, Sysadmin: Gigi Con, Scribe: Writer Writerson"
1. You should add notes to Slack when significant actions are taken, or findings are determined. You don't need to wait for the IC to direct this - use your own judgment. 1. You should add notes to the proper channels when significant actions are taken, or findings are determined. You don't need to wait for the TL to direct this - use your own judgment.
* You should also add `TODO` notes to the Slack room that indicate follow-ups slated for later. * You should also add `TODO` notes to the proper channel that indicate follow-ups slated for later.
1. Follow instructions from the Incident Commander. 1. Follow instructions from the Team Leader.
## Steps for Subject Matter Experts ## Steps for Subject Matter Experts
You are there to support the incident commander in identifying the cause of the incident, suggesting and evaluation repair actions, and following through on the repair actions. You are there to support the team leader in identifying the cause of the incident, suggesting and evaluation repair actions, and following through on the repair actions.
1. Investigate the incident by analyzing any graphs or logs at your disposal. Announce all findings to the incident commander. 1. Investigate the incident by analyzing any graphs or logs at your disposal. Announce all findings to the incident commander.
* If you are unsure of the cause, that's fine, state that you are investigating and provide regular updates to the IC. * If you are unsure of the cause, that's fine, state that you are investigating and provide regular updates to the TL.
1. Announce all suggestions for resolution to the incident commander, it is their decision on how to proceed, do not follow any actions unless told to do so! 1. Announce all suggestions for resolution to the team leader, it is their decision on how to proceed, do not follow any actions unless told to do so!
1. Follow instructions from the incident commander. 1. Follow instructions from the team leader.
1. (Optional) Once the call is over and post-mortem is created, add any notes you think are relevant to the post-mortem page. 1. (Optional) Once the call is over and post-mortem is created, add any notes you think are relevant to the post-mortem page.
## Steps for Customer Liaison ## Steps for Customer Liaison
Be on stand-by to post public facing messages regarding the incident. Be on stand-by to post public facing messages regarding the incident.
1. You will typically be required to update the status page and to send Tweets from our various accounts at certain times during the call. 1. You will typically be required to update the status page and to send Tweets or other communications from our various accounts at certain times during the call.
1. Follow instructions from the Incident Commander. 1. Follow instructions from the Team Leader.