<metaproperty="og:title"content="Severity Levels - Spearhead Systems Incident Response Documentation"/>
<metaproperty="og:site_name"content="Spearhead Systems Incident Response Documentation"/>
<metaproperty="og:description"content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work."/>
<metaname="twitter:title"content="Severity Levels - Spearhead Systems Incident Response Documentation"/>
<metaname="twitter:description"content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work."/>
<p>The first step in any incident response process is to determine what actually constitutes an incident. We have two high level categories for classifying incidents: these are "SR" or "IN" defintions with an attached priority of "Minor", "Normal" or "Major". "SR" are "Service requests" initiated by a customer and usually do not constitute a critical issue (there are exceptions) while "IN" are "incidents" which are generally "urgent".</p>
<p>All issues reported to Spearhead are to be classified as either a Service Request or an Incident. Incidents have priority over Service Requests provided that there are no Service Requests with a higher priority. In general you will want to resolve a higher severity SR or IN than a lower one (a "Major" priority gets a more intensive response than a "Normal" incident for example).</p>
<pclass="admonition-title">Always Assume The Worst</p>
<p>If you are unsure which level an incident is (e.g. not sure if IN is Major or Normal), <strong>treat it as the higher one</strong>. During an incident is not the time to discuss or litigate severities, just assume the highest and review during a post-mortem.</p>
</div>
<tableclass="custom-table">
<thead>
<tr>
<th>Severity</th>
<th>Description</th>
<th>What To Do</th>
</tr>
</thead>
<tbody>
<tr>
<tdclass="sev-1">Major</td>
<td>
<ul>
<li>The system is in a critical state and is actively impacting a large number of customers.</li>
<li>Functionality has been severely impaired for a long time, breaking SLA.</li>
<li>Customer-data-exposing security vulnerability has come to our attention.</li>
</ul>
</td>
<td>See <ahref="/during/during_an_incident">During an Incident</a>.</td>
<li>Partial loss of functionality, only affecting minority of customers.</li>
<li>Something that has the likelihood of becoming Major if nothing is done.</li>
<li>No redundancy in a service (failure of 1 more node will cause outage).</li>
</ul>
</td>
<td>
<ul>
<li>Work on issue as your top priority.</li>
<li>Liaise with engineers of affected systems to identify cause.</li>
<li>If related to recent deployment, rollback.</li>
<li>Monitor status and notice if/when it escalates.</li>
<li>Mention on Slack if you think it has the potential to escalate.</li>
</ul>
</td>
</tr>
<tr>
<tdclass="sev-2">Normal</td>
<td>
<ul>
<li>Performance issues (delays, etc). Tasks that require non-immediate attention.</li>
<li>Job failure (not impacting alerting).</li>
</ul>
</td>
<td>
<ul>
<li>Work on the issue as your first priority (above "Low" tasks).</li>
<li>Monitor status and notice if/when it escalates.</li>
</ul>
</td>
</tr>
<tr>
<tdclass="sev-5">Low</td>
<td>
<ul>
<li>Normal issues which aren't impacting system use, cosmetic issues, etc.</li>
</ul>
</td>
<td>
<ul>
<li>Create a DoIT card and assign to owner of affected system.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<divclass="admonition note">
<pclass="admonition-title">Be Specific</p>
<p>When creating Cards in Doit, be as specific as possible and include all necessary details. Include relevant details regarding when the issue started, what may have triggered it, etc.. Document your efforts through worklogs and be specific there as well.</p>
