<metaproperty="og:title"content="Alerting Principles - Spearhead Systems Incident Response Documentation"/>
<metaproperty="og:site_name"content="Spearhead Systems Incident Response Documentation"/>
<metaproperty="og:description"content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work."/>
<metaname="twitter:title"content="Alerting Principles - Spearhead Systems Incident Response Documentation"/>
<metaname="twitter:description"content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work."/>
<p>We manage how we get alerted based on many factors such as the customers contractual SLA, the urgency of their request or incident, etc.. <strong>an alert or notification is something which requires a human to perform an action</strong>. Based on the severity of the issue (service request or incident) we prioritize accordingly in <ahref="http://doit.sphs.ro">DoIT</a>.</p>
<p>Anything that wakes up a human in the middle of the night should be <strong>immediately human actionable</strong>. If it is none of those things, then we need to adjust the alert to not page at those times.</p>
</div>
<table>
<thead>
<tr>
<th>Priority</th>
<th>Alerts</th>
<th>Response</th>
</tr>
</thead>
<tbody>
<tr>
<td>Major</td>
<td>Major-Priority Spearhead Alert 24/7/365.</td>
<td>Requires <strong>immediate human action</strong>.</td>
</tr>
<tr>
<td>Normal</td>
<td>Normal-Priority Spearhead Alert during <strong>business hours only</strong>.</td>
<td>Requires human action that same working day.</td>
<td>Informational only. We do not need these to clutter out ticketing or inboxes. If they are enabled they should be sent only to required/specific people, not groups.</td>
<p>Both IN and SR (incidents, service requests) share the same priorities. The actual response / resolution times vary and are based upon contractual agreements with the customer. These details (SLA) are available in DoIT on the organization page of the respective customer.</p>
<p>If you're setting up a new alert/notification, consider the chart above for how you want to alert people. Be mindful of not creating new high-priority alerts if they don't require an immediate response, for example.</p>
<divclass="admonition info">
<pclass="admonition-title">Alert Channels</p>
<p>Presently we use email as the only notification method. This means keeping an eye on your email is essential!
SMS and Push notifications are in the pipeline for DoIT. </p>
<h4id="production-service-is-failing-for-75-of-requests-automation-is-unable-to-resolve_">"Production service is failing for 75% of requests, automation is unable to resolve."_<aclass="headerlink"href="#production-service-is-failing-for-75-of-requests-automation-is-unable-to-resolve_"title="Permanent link">#</a></h4>
<p>This would be a <strong>Major</strong> priority IN, requiring immediate human action to resolve.</p>
<h4id="a-customer-sends-an-email-stating-that-production-server-disk-space-is-filling-expected-to-be-full-in-48-hours-log-rotation-is-insufficient-to-resolve">"A customer sends an email stating that "Production server disk space is filling, expected to be full in 48 hours. Log rotation is insufficient to resolve."<aclass="headerlink"href="#a-customer-sends-an-email-stating-that-production-server-disk-space-is-filling-expected-to-be-full-in-48-hours-log-rotation-is-insufficient-to-resolve"title="Permanent link">#</a></h4>
<p>This would be a <strong>Normal</strong> priority SR, requiring human action soon, but not immediately.</p>
<h4id="an-ssl-certificate-is-due-to-expire-in-one-week">"An SSL certificate is due to expire in one week."<aclass="headerlink"href="#an-ssl-certificate-is-due-to-expire-in-one-week"title="Permanent link">#</a></h4>
<p>This would be a <strong>Minor</strong> priority SR, requiring human action some time soon.</p>
