592 lines
18 KiB
HTML
592 lines
18 KiB
HTML
<!DOCTYPE html>
|
|
<!--[if lt IE 7 ]><html class="no-js ie6"><![endif]-->
|
|
<!--[if IE 7 ]><html class="no-js ie7"><![endif]-->
|
|
<!--[if IE 8 ]><html class="no-js ie8"><![endif]-->
|
|
<!--[if IE 9 ]><html class="no-js ie9"><![endif]-->
|
|
<!--[if (gt IE 9)|!(IE)]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
|
|
<head>
|
|
<meta charset="utf-8">
|
|
|
|
<title>Sysadmin - Spearhead Systems Incident Response Documentation</title>
|
|
|
|
<!-- Author and License -->
|
|
<meta name="author" content="Spearhead Systems, Inc." />
|
|
<meta name="dcterms.license" content="http://www.apache.org/licenses/LICENSE-2.0" />
|
|
|
|
<!-- Page Description -->
|
|
|
|
<meta name="keywords" content="spearhead, incident, response" />
|
|
<meta name="robots" content="index, follow, noarchive" />
|
|
|
|
<!-- Mobile -->
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0" />
|
|
<meta name="theme-color" content="#1f293a" />
|
|
|
|
<!-- Canonical Link -->
|
|
<link rel="canonical" href="https://response.spearhead.systems/training/sysadmin/">
|
|
|
|
<!-- Favicon -->
|
|
|
|
<link rel="shortcut icon" type="image/x-icon" href="../../assets/img/icon.png" />
|
|
<link rel="icon" type="image/x-icon" href="../../assets/img/icon.png" />
|
|
|
|
<!-- Apple -->
|
|
<meta name="apple-mobile-web-app-title" content="Spearhead Systems Incident Response Documentation" />
|
|
<meta name="apple-mobile-web-app-capable" content="yes" />
|
|
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
|
<link rel="apple-touch-icon" href="../../assets/img/icon.png">
|
|
|
|
<!-- Open Graph -->
|
|
<meta property="og:url" content="https://response.spearhead.systems/training/sysadmin/" />
|
|
<meta property="og:title" content="Sysadmin - Spearhead Systems Incident Response Documentation" />
|
|
<meta property="og:site_name" content="Spearhead Systems Incident Response Documentation" />
|
|
<meta property="og:description" content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work." />
|
|
<meta property="og:image" content="https://response.spearhead.systems/assets/img/cover.png" />
|
|
<meta property="og:locale" content="en_US" />
|
|
<meta property="og:type" content="website" />
|
|
|
|
<!-- Twitter -->
|
|
<meta name="twitter:card" content="summary_large_image" />
|
|
<meta name="twitter:title" content="Sysadmin - Spearhead Systems Incident Response Documentation" />
|
|
<meta name="twitter:description" content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work." />
|
|
<meta name="twitter:image" content="https://response.spearhead.systems/assets/img/cover.png" />
|
|
|
|
<!-- Style -->
|
|
<style>
|
|
@font-face {
|
|
font-family: 'Icon';
|
|
src: url('../../assets/fonts/icon.eot?52m981');
|
|
src: url('../../assets/fonts/icon.eot?#iefix52m981')
|
|
format('embedded-opentype'),
|
|
url('../../assets/fonts/icon.woff?52m981')
|
|
format('woff'),
|
|
url('../../assets/fonts/icon.ttf?52m981')
|
|
format('truetype'),
|
|
url('../../assets/fonts/icon.svg?52m981#icon')
|
|
format('svg');
|
|
font-weight: normal;
|
|
font-style: normal;
|
|
}
|
|
</style>
|
|
<link rel="stylesheet" href="../../assets/stylesheets/application-a422ff04cc.css">
|
|
|
|
<link rel="stylesheet" href="../../assets/stylesheets/palettes-05ab2406df.css">
|
|
|
|
|
|
|
|
|
|
|
|
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:400,700|Roboto+Mono">
|
|
<style>
|
|
body, input {
|
|
font-family: 'Roboto', Helvetica, Arial, sans-serif;
|
|
}
|
|
pre, code {
|
|
font-family: 'Roboto Mono', 'Courier New', 'Courier', monospace;
|
|
}
|
|
</style>
|
|
|
|
|
|
<link rel="stylesheet" href="../../assets/css/extra.css">
|
|
|
|
|
|
<!-- Scripts -->
|
|
<script src="../../assets/javascripts/modernizr-4ab42b99fd.js"></script>
|
|
|
|
</head>
|
|
|
|
|
|
|
|
<body class="palette-primary-green palette-accent-blue-grey">
|
|
|
|
|
|
|
|
|
|
<div class="backdrop">
|
|
<div class="backdrop-paper"></div>
|
|
</div>
|
|
<input class="toggle" type="checkbox" id="toggle-drawer">
|
|
<input class="toggle" type="checkbox" id="toggle-search">
|
|
<label class="toggle-button overlay" for="toggle-drawer"></label>
|
|
<header class="header">
|
|
<nav aria-label="Header">
|
|
<div class="bar default">
|
|
<div class="button button-menu" role="button" aria-label="Menu">
|
|
<label class="toggle-button icon icon-menu" for="toggle-drawer">
|
|
<span></span>
|
|
</label>
|
|
</div>
|
|
<div class="stretch">
|
|
<div class="mainlogo">
|
|
<a href="/" title="Go to homepage.">
|
|
<img src="../../assets/img/logo.png" title="Spearhead Systems" />
|
|
</a>
|
|
</div>
|
|
<div class="title">
|
|
<span class="path">
|
|
Incident Response
|
|
<i class="icon icon-link"></i>
|
|
</span>
|
|
|
|
<span class="path">
|
|
|
|
|
|
Training <i class="icon icon-link"></i>
|
|
|
|
|
|
</span>
|
|
|
|
Sysadmin
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<div class="button button-twitter" role="button" aria-label="Twitter">
|
|
<a href="https://twitter.com/spearhead_sys" title="@spearhead_sys on Twitter" target="_blank" class="toggle-button icon icon-twitter"></a>
|
|
</div>
|
|
|
|
|
|
|
|
<div class="button button-github" role="button" aria-label="GitHub">
|
|
<a href="https://github.com/spearheadsys" title="@spearheadsys on GitHub" target="_blank" class="toggle-button icon icon-github"></a>
|
|
</div>
|
|
|
|
<div class="button button-search" role="button" aria-label="Search">
|
|
<label class="toggle-button icon icon-search" title="Search" for="toggle-search"></label>
|
|
</div>
|
|
</div>
|
|
<div class="bar search">
|
|
<div class="button button-close" role="button" aria-label="Close">
|
|
<label class="toggle-button icon icon-back" for="toggle-search"></label>
|
|
</div>
|
|
<div class="stretch">
|
|
<div class="field">
|
|
<input class="query" type="text" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false">
|
|
</div>
|
|
</div>
|
|
<div class="button button-reset" role="button" aria-label="Search">
|
|
<button class="toggle-button icon icon-close" id="reset-search"></button>
|
|
</div>
|
|
</div>
|
|
</nav>
|
|
</header>
|
|
<main class="main">
|
|
|
|
<div class="drawer">
|
|
<nav aria-label="Navigation">
|
|
|
|
<a href="https://github.com/spearheadsys/issue-response-docs" class="project">
|
|
<div class="banner">
|
|
|
|
<div class="logo">
|
|
<img src="../../assets/img/icon.png">
|
|
</div>
|
|
|
|
<div class="name">
|
|
<strong>
|
|
Spearhead Systems Incident Response Documentation
|
|
<span class="version">
|
|
|
|
</span>
|
|
</strong>
|
|
|
|
<br>
|
|
spearheadsys/issue-response-docs
|
|
|
|
</div>
|
|
</div>
|
|
</a>
|
|
<div class="scrollable">
|
|
<div class="wrapper">
|
|
|
|
<ul class="repo">
|
|
<li class="repo-download">
|
|
|
|
|
|
<a href="https://github.com/spearheadsys/issue-response-docs/archive/master.zip" target="_blank" title="Download" data-action="download">
|
|
<i class="icon icon-download"></i> Download
|
|
</a>
|
|
|
|
</li>
|
|
<li class="repo-stars">
|
|
<a href="https://github.com/spearheadsys/issue-response-docs/stargazers" target="_blank" title="Stargazers" data-action="star">
|
|
<i class="icon icon-star"></i> Stars
|
|
<span class="count">–</span>
|
|
</a>
|
|
</li>
|
|
</ul>
|
|
<hr/>
|
|
|
|
<div class="toc">
|
|
<ul>
|
|
|
|
|
|
<li>
|
|
<a class="" title="Home" href="../..">
|
|
Home
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<span class="section">On-Call</span>
|
|
<ul>
|
|
|
|
|
|
<li>
|
|
<a class="" title="Being On-Call" href="../../oncall/being_oncall/">
|
|
Being On-Call
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Alerting Principles" href="../../oncall/alerting_principles/">
|
|
Alerting Principles
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<span class="section">Before an Incident</span>
|
|
<ul>
|
|
|
|
|
|
<li>
|
|
<a class="" title="Severity Levels" href="../../before/severity_levels/">
|
|
Severity Levels
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Different Roles" href="../../before/different_roles/">
|
|
Different Roles
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Call Etiquette" href="../../before/call_etiquette/">
|
|
Call Etiquette
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<span class="section">During an Incident</span>
|
|
<ul>
|
|
|
|
|
|
<li>
|
|
<a class="" title="During An Incident" href="../../during/during_an_incident/">
|
|
During An Incident
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Security Incident" href="../../during/security_incident_response/">
|
|
Security Incident
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<span class="section">After an Incident</span>
|
|
<ul>
|
|
|
|
|
|
<li>
|
|
<a class="" title="Post-Mortem Process" href="../../after/post_mortem_process/">
|
|
Post-Mortem Process
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Post-Mortem Template" href="../../after/post_mortem_template/">
|
|
Post-Mortem Template
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<span class="section">Training</span>
|
|
<ul>
|
|
|
|
|
|
<li>
|
|
<a class="" title="Overview" href="../overview/">
|
|
Overview
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Team Leader" href="../team_leader/">
|
|
Team Leader
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="current" title="Sysadmin" href="./">
|
|
Sysadmin
|
|
</a>
|
|
|
|
|
|
|
|
<ul>
|
|
|
|
<li class="anchor">
|
|
<a title="Purpose" href="#purpose">
|
|
Purpose
|
|
</a>
|
|
</li>
|
|
|
|
<li class="anchor">
|
|
<a title="Prerequisites" href="#prerequisites">
|
|
Prerequisites
|
|
</a>
|
|
</li>
|
|
|
|
<li class="anchor">
|
|
<a title="Responsibilities" href="#responsibilities">
|
|
Responsibilities
|
|
</a>
|
|
</li>
|
|
|
|
<li class="anchor">
|
|
<a title="Training Process" href="#training-process">
|
|
Training Process
|
|
</a>
|
|
</li>
|
|
|
|
<li class="anchor">
|
|
<a title="Incident Call Procedures and Lingo" href="#incident-call-procedures-and-lingo">
|
|
Incident Call Procedures and Lingo
|
|
</a>
|
|
</li>
|
|
|
|
</ul>
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Scribe" href="../scribe/">
|
|
Scribe
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Subject Matter Expert" href="../subject_matter_expert/">
|
|
Subject Matter Expert
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="Glossary" href="../glossary/">
|
|
Glossary
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
</li>
|
|
|
|
|
|
|
|
<li>
|
|
<a class="" title="About" href="../../about/">
|
|
About
|
|
</a>
|
|
|
|
</li>
|
|
|
|
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</nav>
|
|
</div>
|
|
<article class="article">
|
|
<div class="wrapper">
|
|
|
|
<h1>Sysadmin</h1>
|
|
|
|
<p>So you want to be a Sysadmin? You've come to the right place!</p>
|
|
<p><img alt="Deputy" src="../../assets/img/headers/incident_command_support.jpg" />
|
|
<em>Credit: <a href="https://www.flickr.com/photos/oregondot/8743801731/in/album-72157633494644719/">oregondot @ Flickr</a></em></p>
|
|
<h2 id="purpose">Purpose<a class="headerlink" href="#purpose" title="Permanent link">#</a></h2>
|
|
<p>The purpose of the Sysadmin is to support the TL by keeping track of timers, notifying the TL of important information, and paging other people as directed by the TL.</p>
|
|
<p>It's important for the TL to focus on the problem at hand, rather than worrying about monitoring timers. The deputy is there to help support the TL and keep them focussed on the incident.</p>
|
|
<p>As a Sysadmin, you will be expected to take over command from the TL if they request it.</p>
|
|
<p><strong>You should be performing any remediations, checking graphs, or investigating logs</strong> unless otherwise delegated by the TL.</p>
|
|
<h2 id="prerequisites">Prerequisites<a class="headerlink" href="#prerequisites" title="Permanent link">#</a></h2>
|
|
<p>Before you can be a Sysadmin, it is expected that you meet the following criteria. Don't worry if you don't meet them all yet, you can still continue with training!</p>
|
|
<ul>
|
|
<li>Be trained as an <a href="../team_leader/">Team Leader</a>.</li>
|
|
</ul>
|
|
<h2 id="responsibilities">Responsibilities<a class="headerlink" href="#responsibilities" title="Permanent link">#</a></h2>
|
|
<p>Read up on our <a href="../../before/different_roles/">Different Roles for Incidents</a> to see what is expected from a Sysadmin, as well as what we expect from the other roles you'll be interacting with.</p>
|
|
<h2 id="training-process">Training Process<a class="headerlink" href="#training-process" title="Permanent link">#</a></h2>
|
|
<p>The training process for a Sysadmin is quite simple.</p>
|
|
<ul>
|
|
<li>Follow our <a href="../team_leader/">Team Leader Training</a>.</li>
|
|
<li>Read this page.</li>
|
|
</ul>
|
|
<h2 id="incident-call-procedures-and-lingo">Incident Call Procedures and Lingo<a class="headerlink" href="#incident-call-procedures-and-lingo" title="Permanent link">#</a></h2>
|
|
<p>The <a href="../../during/during_an_incident/">Steps for Sysadmin</a> provide a detailed description of what you should be doing during an incident.</p>
|
|
<p>Here are some examples of phrases and patterns you should use during incident calls.</p>
|
|
<h3 id="keep-track-of-responders">Keep Track of Responders<a class="headerlink" href="#keep-track-of-responders" title="Permanent link">#</a></h3>
|
|
<p>As you listen to the call, you should keep track of the responders to the call as you hear them speak. Make a note on a piece of paper and add them to the Watchers in DoIT. The TL may ask you who is on-call for a particular issue, and you should know the answer, and be able to page them.</p>
|
|
<blockquote>
|
|
<p>Do we have a representative from [X] on the call?</p>
|
|
<p>(pause)</p>
|
|
<p>Sysadmin, can you go ahead and page the [X] on-call please.</p>
|
|
</blockquote>
|
|
<p>You can page them however you see fit, phone call, etc.</p>
|
|
<h3 id="provide-executive-status-updates">Provide Executive Status Updates<a class="headerlink" href="#provide-executive-status-updates" title="Permanent link">#</a></h3>
|
|
<p>Provide regular status updates on Slack (roughly every 30mins), giving an executive summary of the current status during IN-3 incidents. Keep it short and to the point, and use @here. Mention the current state, the actions in progress, customer impact, and expected time remaining. It's OK to miss out some of those if the information isn't known.</p>
|
|
<blockquote>
|
|
<p>@here: We are in IN-3 due to X. Current actions in progress are to do Y. Expecting 3 mins to complete that action. Once action is complete, system should recover on its own within 5 minutes.</p>
|
|
</blockquote>
|
|
<h3 id="alert-tl-to-timers">Alert TL to Timers<a class="headerlink" href="#alert-tl-to-timers" title="Permanent link">#</a></h3>
|
|
<p>You are expected to keep track of how long the incident has been running for, and provide callouts to the TL every 10 minutes so they can take actions such as increasing the severity, or asking Support to Tweet out. This is as simple as telling the TL on the call,</p>
|
|
<blockquote>
|
|
<p>TL, be advised the incident is now at the 10 minute mark.</p>
|
|
</blockquote>
|
|
<p>Similarly, when the TL asks for someone to get back to them in X minutes, you are expected to keep track of that. You should remind the TL when that time has been reached.</p>
|
|
<blockquote>
|
|
<p>TL, be advised the timer for [TEAM]'s investigation is up.</p>
|
|
</blockquote>
|
|
<aside class="copyright" role="note">
|
|
|
|
Copyright © Spearhead Systems, Inc. –
|
|
|
|
Documentation built with
|
|
<a href="http://www.mkdocs.org" target="_blank">MkDocs</a>
|
|
using the
|
|
<a href="http://squidfunk.github.io/mkdocs-material/" target="_blank">
|
|
Material
|
|
</a>
|
|
theme.
|
|
</aside>
|
|
|
|
<footer class="footer">
|
|
|
|
<nav class="pagination" aria-label="Footer">
|
|
<div class="previous">
|
|
|
|
<a href="../team_leader/" title="Team Leader">
|
|
<span class="direction">
|
|
Previous
|
|
</span>
|
|
<div class="page">
|
|
<div class="button button-previous" role="button" aria-label="Previous">
|
|
<i class="icon icon-back"></i>
|
|
</div>
|
|
<div class="stretch">
|
|
<div class="title">
|
|
Team Leader
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</a>
|
|
|
|
</div>
|
|
<div class="next">
|
|
|
|
<a href="../scribe/" title="Scribe">
|
|
<span class="direction">
|
|
Next
|
|
</span>
|
|
<div class="page">
|
|
<div class="stretch">
|
|
<div class="title">
|
|
Scribe
|
|
</div>
|
|
</div>
|
|
<div class="button button-next" role="button" aria-label="Next">
|
|
<i class="icon icon-forward"></i>
|
|
</div>
|
|
</div>
|
|
</a>
|
|
|
|
</div>
|
|
</nav>
|
|
|
|
</footer>
|
|
|
|
</div>
|
|
</article>
|
|
<div class="results" role="status" aria-live="polite">
|
|
<div class="scrollable">
|
|
<div class="wrapper">
|
|
<div class="meta"></div>
|
|
<div class="list"></div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</main>
|
|
<script>
|
|
var base_url = '../..';
|
|
var repo_id = 'spearheadsys/issue-response-docs';
|
|
</script>
|
|
<script src="../../assets/javascripts/application-997097ee0c.js"></script>
|
|
|
|
|
|
</body>
|
|
</html> |