<metaproperty="og:title"content="Severity Levels - Spearhead Systems Incident Response Documentation"/>
<metaproperty="og:site_name"content="Spearhead Systems Incident Response Documentation"/>
<metaproperty="og:description"content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work."/>
<metaname="twitter:title"content="Severity Levels - Spearhead Systems Incident Response Documentation"/>
<metaname="twitter:description"content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work."/>
<p>The first step in any incident response process is to determine what actually constitutes an incident. We have two high level categories for classifying incidents: this is done using "SR" or "IN" defintions with an attached priority of "Minor", "Normal" or "Major". "SR" are "Service requests" initiated by a customer and usually do not constitute a critical issue (there are exceptions) and "IN" are "incidents" which are generally "urgent".</p>
<p>All of our operational issues are to be classified as either a Service Request or an Incident. Incidents have priority over Service Requests provided that there are no Service Requests with a higher priority. In general you will want to resolve a higher severity SR or IN than a lower one (a "Major" priority gets a more intensive response than a "Normal" incident for example).</p>
<divclass="admonition note">
<pclass="admonition-title">Always Assume The Worst</p>
<p>If you are unsure which level an incident is (e.g. not sure if IN is Major or Normal), <strong>treat it as the higher one</strong>. During an incident is not the time to discuss or litigate severities, just assume the highest and review during a post-mortem.</p>
</div>
<tableclass="custom-table">
<thead>
<tr>
<th>Severity</th>
<th>Description</th>
<th>What To Do</th>
</tr>
</thead>
<tbody>
<tr>
<tdclass="sev-1">Major</td>
<td>
<ul>
<li>The system is in a critical state and is actively impacting a large number of customers.</li>
<li>Functionality has been severely impaired for a long time, breaking SLA.</li>
<li>Customer-data-exposing security vulnerability has come to our attention.</li>
</ul>
</td>
<td>See <ahref="/during/during_an_incident">During an Incident</a>.</td>
</tr>
<tr>
<tdclass="sev-2">Normal</td>
<td>
<ul>
<li>Functionality of virtualization platform is severely impaired.</li>
<li>E-mail system is offline.</li>
</ul>
</td>
<td>See <ahref="/during/during_an_incident">During an Incident</a>.</td>
</tr>
<tr>
<tdclass="warning"colspan="3">Anything above this line is considered a "Major Incident". These are generally Incidents (IN). Below are service requests (SR) which are usually initiated by a human who can help with prioritizing. A call is triggered for all major incidents (indifferently of SR or IN).</td>
</tr>
<tr>
<tdclass="sev-2">Normal</td>
<td>
<ul>
<li>Partial loss of functionality, only affecting minority of customers.</li>
<li>Something that has the likelihood of becoming Major if nothing is done.</li>
<li>No redundancy in a service (failure of 1 more node will cause outage).</li>
</ul>
</td>
<td>
<ul>
<li>Work on issue as your top priority.</li>
<li>Liaise with engineers of affected systems to identify cause.</li>
<li>If related to recent deployment, rollback.</li>
<li>Monitor status and notice if/when it escalates.</li>
<li>Mention on Slack if you think it has the potential to escalate.</li>
</ul>
</td>
</tr>
<tr>
<tdclass="sev-2">Normal</td>
<td>
<ul>
<li>Performance issues (delays, etc). Tasks that require non-immediate attention.</li>
<li>Job failure (not impacting alerting).</li>
</ul>
</td>
<td>
<ul>
<li>Work on the issue as your first priority (above "Low" tasks).</li>
<li>Monitor status and notice if/when it escalates.</li>
</ul>
</td>
</tr>
<tr>
<tdclass="sev-5">Low</td>
<td>
<ul>
<li>Normal bugs which aren't impacting system use, cosmetic issues, etc.</li>
</ul>
</td>
<td>
<ul>
<li>Create a DoIT ticket and assign to owner of affected system.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<divclass="admonition note">
<pclass="admonition-title">Be Specific</p>
<p>When creating Cards in Doit, be as specific as possible and include all necessary details. Include relevant details regarding when the issue started, what may have triggered it, etc.. Document your efforts through worklogs and be specific there as well. </p>
