# Post-Mortem Template - Spearhead Systems Incident Response Documentation
A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work.
Author: Spearhead Systems, Inc.
Keywords: spearhead, incident, response

# Post-Mortem Template

This is a standard template for post-mortems. Each section describes the type of information you will want to put in that section.

Guidelines

This page is intended to be reviewed during a post-mortem meeting that should be scheduled within 5 business days of any event. Your first step should be to schedule the post-mortem meeting in the shared calendar for within 5 business days after the incident. Don't wait until you've filled in the info to schedule the meeting, however make sure the page is completed by the meeting.

Post-Mortem Owner: Your name goes here.

Meeting Scheduled For: Schedule the meeting on the "Incident Post-Mortem Meetings" shared calendar, for within 5 business days after the incident. Put the date/time here.

Call Recording: Link to the incident call recording / slack transcript or DoIT card.

## Overview

Include a short sentence or two summarizing the root cause, timeline summary, and the impact. E.g. "On the morning of August 99th, we suffered a 1 minute IN-3 due to a runaway process on our primary database machine. This slowness caused roughly 0.024% of alerts that had begun during this time to be delivered out of SLA."

## What Happened

Include a short description of what happened.

## Root Cause

Include a description of the root cause. If there were any actions taken that exacerbated the issue, also include them here with the intention of learning from any mistakes made during the resolution process.

## Resolution

Include a description what solved the problem. If there was a temporary fix in place, describe that along with the long-term solution.

## Impact

Be very specific here, include exact numbers.

Time in SR-3 | ?mins
--- | ---
Notifications Delivered out of SLA | ??% (?? of ??)
Events Dropped / Not Accepted | ??% (?? of ??) Should usually be 0, but always check
Accounts Affected | ??
Users Affected | ??
Support Requests Raised | ?? Include any relevant links to tickets

## Responders

- Who was the TL?
- Who was the scribe?
- Who else was involved?
- Who else was involved?

## Timeline

Some important times to include: (1) time the root cause began, (2) time of the page, (3) time that the status page was updated (i.e. when the incident became public), (4) time of any significant actions, (5) time the IN-3 ended, (6) links to tools/logs that show how the timestamp was arrived at.

Time (UTC) | Event | Data Link
--- | --- | ---

## How'd We Do?

### What Went Well?

- List anything you did well and want to call out. It's OK to not list anything.

### What Didn't Go So Well?

- List anything you think we didn't do very well. The intent is that we should follow up on all points here to improve our processes.

## Action Items

Each action item should be in the form of a DoIT card respectiv GTD next actions principle: "a clear and concise single action to move things forward”. Include action items such as: (1) any fixes required to prevent the root cause in the future, (2) any preparedness tasks that could help mitigate the problem if it came up again, (3) remaining post-mortem steps, such as the internal email, as well as the status-page public post, (4) any improvements to our incident response process.

## Messaging

### Internal Email

This is a follow-up for employees. It should be sent out right after the post-mortem meeting is over. It only needs a short paragraph summarizing the incident and a link to this wiki page.

> Briefly summarize what happened and where the post-mortem page (this page) can be found.

### External Message

This is what will be included on the public facing status website (status.spearhead.systems) regarding this incident. What are we telling customers, including an apology? (The apology should be genuine, not rote.)

> Summary

> What Happened?

> What Are We Doing About This?

Copyright © Spearhead Systems, Inc. – Documentation built with MkDocs using the Material theme.