Spearhead Systems Incident Response Documentation

This documentation covers parts of the Spearhead Systems Incident Response process. It is a copy of PagerDuty's documentation and furthermore a cut-down version of our own internal documentation, used at Spearhead Systems for any issue (incident or service request), and to prepare new employees for on-call responsibilities. It provides information not only on preparing for an incident or service request, but also what to do during and after. It is intended to be used by those involved in our operational technical support response process (or those wishing to become part of our support team). See the about page for more information on what this documentation is and why it exists. This documentation is complementary to what is available in our existing wiki and other systems that have not been open sourced.

Being On-Call#

If you've never been on-call before or part of a support delivery team, you might be wondering what it's all about. These pages describe what the expectations are, along with some resources to help you.

• Being On-Call - A guide to being on-call, both what your responsibilities are, and what they are not.
• Alerting Principles - The principles we use to determine what things notify an engineer, and what time of day they do so.

Before an Incident#

Reading material for things you probably want to know before an incident occurs. You likely don't want to be reading these during an actual incident.

• Severity Levels - Information on our severity level classification. What constitutes a Low issue? What's a "Major Incident"?, etc.
• Different Roles for Incidents - Information on the roles during an incident; Team Leader, Sysadmin, etc.
• Incident Call Etiquette - Our etiquette guidelines for incident calls, before you find yourself in one.

During an Incident#

Information and processes during an incident.

• During an Incident - Information on what to do during an incident, and how to constructively contribute.
• Security Incident Response - Security incidents are handled differently to normal operational incidents.

After an Incident#

Our followup processes, how we make sure we don't repeat mistakes and are always improving.

• Post-Mortem Process - Information on our post-mortem process; what's involved and how to write or run a post-mortem.
• Post-Mortem Template - The template we use for writing our post-mortems for major incidents.

Training#

So, you want to learn about incident response? You've come to the right place.

• Training Overview - An overview of our training guides and additional training material from third-parties.
• Incident Commander Training - A guide to becoming our next Incident Commander.
• Deputy Training - How to be a deputy and back up the Incident Commander.
• Scribe Training - A guide to scribing.
• Subject Matter Expert Training - A guide on responsibilities and behavior for all participants in a major incident.
• Glossary of Incident Response Terms - A collection of terms that you may hear being used, along with their definition.

Additional Reading#

Useful material and resources from external parties that are relevant to incident response.

• Incident Management for Operations (O'Reilly)
• Incident Response (O'Reilly)
• Debriefing Facilitation Guide (Etsy)
• US National Incident Management System (NIMS) (FEMA)
• Every Minute Counts: Leading Heroku's Incident Response (Blake Gentry)