spearhead-issue-response/during/security_incident_response/index.html

742 lines
28 KiB
HTML

<!DOCTYPE html>
<!--[if lt IE 7 ]><html class="no-js ie6"><![endif]-->
<!--[if IE 7 ]><html class="no-js ie7"><![endif]-->
<!--[if IE 8 ]><html class="no-js ie8"><![endif]-->
<!--[if IE 9 ]><html class="no-js ie9"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<title>Security Incident - Spearhead Systems Incident Response Documentation</title>
<!-- Author and License -->
<meta name="author" content="Spearhead Systems, Inc." />
<meta name="dcterms.license" content="http://www.apache.org/licenses/LICENSE-2.0" />
<!-- Page Description -->
<meta name="keywords" content="spearhead, incident, response" />
<meta name="robots" content="index, follow, noarchive" />
<!-- Mobile -->
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0" />
<meta name="theme-color" content="#1f293a" />
<!-- Canonical Link -->
<link rel="canonical" href="https://response.spearhead.systems/during/security_incident_response/">
<!-- Favicon -->
<link rel="shortcut icon" type="image/x-icon" href="../../assets/img/icon.png" />
<link rel="icon" type="image/x-icon" href="../../assets/img/icon.png" />
<!-- Apple -->
<meta name="apple-mobile-web-app-title" content="Spearhead Systems Incident Response Documentation" />
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<link rel="apple-touch-icon" href="../../assets/img/icon.png">
<!-- Open Graph -->
<meta property="og:url" content="https://response.spearhead.systems/during/security_incident_response/" />
<meta property="og:title" content="Security Incident - Spearhead Systems Incident Response Documentation" />
<meta property="og:site_name" content="Spearhead Systems Incident Response Documentation" />
<meta property="og:description" content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work." />
<meta property="og:image" content="https://response.spearhead.systems/assets/img/cover.png" />
<meta property="og:locale" content="en_US" />
<meta property="og:type" content="website" />
<!-- Twitter -->
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content="Security Incident - Spearhead Systems Incident Response Documentation" />
<meta name="twitter:description" content="A collection of information about the Spearhead Systems incident response process. Not only how to prepare new employees for on-call responsibilities, but also how to handle major incidents, both in preparation and after-work." />
<meta name="twitter:image" content="https://response.spearhead.systems/assets/img/cover.png" />
<!-- Style -->
<style>
@font-face {
font-family: 'Icon';
src: url('../../assets/fonts/icon.eot?52m981');
src: url('../../assets/fonts/icon.eot?#iefix52m981')
format('embedded-opentype'),
url('../../assets/fonts/icon.woff?52m981')
format('woff'),
url('../../assets/fonts/icon.ttf?52m981')
format('truetype'),
url('../../assets/fonts/icon.svg?52m981#icon')
format('svg');
font-weight: normal;
font-style: normal;
}
</style>
<link rel="stylesheet" href="../../assets/stylesheets/application-a422ff04cc.css">
<link rel="stylesheet" href="../../assets/stylesheets/palettes-05ab2406df.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Colfax Regular:400,700|Roboto+Mono">
<style>
body, input {
font-family: 'Colfax Regular', Helvetica, Arial, sans-serif;
}
pre, code {
font-family: 'Roboto Mono', 'Courier New', 'Courier', monospace;
}
</style>
<link rel="stylesheet" href="../../assets/css/extra.css">
<!-- Scripts -->
<script src="../../assets/javascripts/modernizr-4ab42b99fd.js"></script>
</head>
<body class="palette-primary-green palette-accent-blue-grey">
<div class="backdrop">
<div class="backdrop-paper"></div>
</div>
<input class="toggle" type="checkbox" id="toggle-drawer">
<input class="toggle" type="checkbox" id="toggle-search">
<label class="toggle-button overlay" for="toggle-drawer"></label>
<header class="header">
<nav aria-label="Header">
<div class="bar default">
<div class="button button-menu" role="button" aria-label="Menu">
<label class="toggle-button icon icon-menu" for="toggle-drawer">
<span></span>
</label>
</div>
<div class="stretch">
<div class="mainlogo">
<a href="/" title="Go to homepage.">
<img src="../../assets/img/logo.png" title="PagerDuty" />
</a>
</div>
<div class="title">
<span class="path">
Incident Response
<i class="icon icon-link"></i>
</span>
<span class="path">
During an Incident <i class="icon icon-link"></i>
</span>
Security Incident
</div>
</div>
<div class="button button-twitter" role="button" aria-label="Twitter">
<a href="https://twitter.com/spearhead_sys" title="@spearhead_sys on Twitter" target="_blank" class="toggle-button icon icon-twitter"></a>
</div>
<div class="button button-github" role="button" aria-label="GitHub">
<a href="https://github.com/spearheadsys" title="@spearheadsys on GitHub" target="_blank" class="toggle-button icon icon-github"></a>
</div>
<div class="button button-search" role="button" aria-label="Search">
<label class="toggle-button icon icon-search" title="Search" for="toggle-search"></label>
</div>
</div>
<div class="bar search">
<div class="button button-close" role="button" aria-label="Close">
<label class="toggle-button icon icon-back" for="toggle-search"></label>
</div>
<div class="stretch">
<div class="field">
<input class="query" type="text" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false">
</div>
</div>
<div class="button button-reset" role="button" aria-label="Search">
<button class="toggle-button icon icon-close" id="reset-search"></button>
</div>
</div>
</nav>
</header>
<main class="main">
<div class="drawer">
<nav aria-label="Navigation">
<a href="https://github.com/spearheadsys/issue-response-docs" class="project">
<div class="banner">
<div class="logo">
<img src="../../assets/img/icon.png">
</div>
<div class="name">
<strong>
Spearhead Systems Incident Response Documentation
<span class="version">
</span>
</strong>
<br>
spearheadsys/issue-response-docs
</div>
</div>
</a>
<div class="scrollable">
<div class="wrapper">
<ul class="repo">
<li class="repo-download">
<a href="https://github.com/spearheadsys/issue-response-docs/archive/master.zip" target="_blank" title="Download" data-action="download">
<i class="icon icon-download"></i> Download
</a>
</li>
<li class="repo-stars">
<a href="https://github.com/spearheadsys/issue-response-docs/stargazers" target="_blank" title="Stargazers" data-action="star">
<i class="icon icon-star"></i> Stars
<span class="count">&ndash;</span>
</a>
</li>
</ul>
<hr/>
<div class="toc">
<ul>
<li>
<a class="" title="Home" href="../..">
Home
</a>
</li>
<li>
<span class="section">On-Call</span>
<ul>
<li>
<a class="" title="Being On-Call" href="../../oncall/being_oncall/">
Being On-Call
</a>
</li>
<li>
<a class="" title="Alerting Principles" href="../../oncall/alerting_principles/">
Alerting Principles
</a>
</li>
</ul>
</li>
<li>
<span class="section">Before an Incident</span>
<ul>
<li>
<a class="" title="Severity Levels" href="../../before/severity_levels/">
Severity Levels
</a>
</li>
<li>
<a class="" title="Different Roles" href="../../before/different_roles/">
Different Roles
</a>
</li>
<li>
<a class="" title="Call Etiquette" href="../../before/call_etiquette/">
Call Etiquette
</a>
</li>
</ul>
</li>
<li>
<span class="section">During an Incident</span>
<ul>
<li>
<a class="" title="During An Incident" href="../during_an_incident/">
During An Incident
</a>
</li>
<li>
<a class="current" title="Security Incident" href="./">
Security Incident
</a>
<ul>
<li class="anchor">
<a title="Checklist" href="#checklist">
Checklist
</a>
</li>
<li class="anchor">
<a title="Attack Mitigation" href="#attack-mitigation">
Attack Mitigation
</a>
</li>
<li class="anchor">
<a title="Cut Off Attack Vector" href="#cut-off-attack-vector">
Cut Off Attack Vector
</a>
</li>
<li class="anchor">
<a title="Assemble Response Team" href="#assemble-response-team">
Assemble Response Team
</a>
</li>
<li class="anchor">
<a title="Isolate Affected Instances" href="#isolate-affected-instances">
Isolate Affected Instances
</a>
</li>
<li class="anchor">
<a title="Identify Timeline of Attack" href="#identify-timeline-of-attack">
Identify Timeline of Attack
</a>
</li>
<li class="anchor">
<a title="Compromised Data" href="#compromised-data">
Compromised Data
</a>
</li>
<li class="anchor">
<a title="Assess Risk" href="#assess-risk">
Assess Risk
</a>
</li>
<li class="anchor">
<a title="Apply Additional Mitigations" href="#apply-additional-mitigations">
Apply Additional Mitigations
</a>
</li>
<li class="anchor">
<a title="Forensic Analysis" href="#forensic-analysis">
Forensic Analysis
</a>
</li>
<li class="anchor">
<a title="Internal Communication" href="#internal-communication">
Internal Communication
</a>
</li>
<li class="anchor">
<a title="Liaise With Law Enforcement / External Actors" href="#liaise-with-law-enforcement-external-actors">
Liaise With Law Enforcement / External Actors
</a>
</li>
<li class="anchor">
<a title="External Communication" href="#external-communication">
External Communication
</a>
</li>
<li class="anchor">
<a title="Additional Reading" href="#additional-reading">
Additional Reading
</a>
</li>
</ul>
</li>
</ul>
</li>
<li>
<span class="section">After an Incident</span>
<ul>
<li>
<a class="" title="Post-Mortem Process" href="../../after/post_mortem_process/">
Post-Mortem Process
</a>
</li>
<li>
<a class="" title="Post-Mortem Template" href="../../after/post_mortem_template/">
Post-Mortem Template
</a>
</li>
</ul>
</li>
<li>
<span class="section">Training</span>
<ul>
<li>
<a class="" title="Overview" href="../../training/overview/">
Overview
</a>
</li>
<li>
<a class="" title="Incident Commander" href="../../training/incident_commander/">
Incident Commander
</a>
</li>
<li>
<a class="" title="Deputy" href="../../training/deputy/">
Deputy
</a>
</li>
<li>
<a class="" title="Scribe" href="../../training/scribe/">
Scribe
</a>
</li>
<li>
<a class="" title="Subject Matter Expert" href="../../training/subject_matter_expert/">
Subject Matter Expert
</a>
</li>
<li>
<a class="" title="Glossary" href="../../training/glossary/">
Glossary
</a>
</li>
</ul>
</li>
<li>
<a class="" title="About" href="../../about/">
About
</a>
</li>
</ul>
</div>
</div>
</div>
</nav>
</div>
<article class="article">
<div class="wrapper">
<h1>Security Incident</h1>
<div class="admonition note">
<p class="admonition-title">Incident Commander Required</p>
<p>As with all major incidents at PagerDuty, security ones will also involve an Incident Commander, who will delegate the tasks to relevant resolvers. Tasks may be performed in parallel as assigned by the IC. Page one at the earliest possible opportunity.</p>
</div>
<h2 id="checklist">Checklist<a class="headerlink" href="#checklist" title="Permanent link">#</a></h2>
<p>Details for each of these items are available in the next section.</p>
<ol>
<li>Stop the attack in progress.</li>
<li>Cut off the attack vector.</li>
<li>Assemble the response team.</li>
<li>Isolate affected instances.</li>
<li>Identify timeline of attack.</li>
<li>Identify compromised data.</li>
<li>Assess risk to other systems.</li>
<li>Assess risk of re-attack.</li>
<li>Apply additional mitigations, additions to monitoring, etc.</li>
<li>Forensic analysis of compromised systems.</li>
<li>Internal communication.</li>
<li>Involve law enforcement.</li>
<li>Reach out to external parties that may have been used as vector for attack.</li>
<li>External communication.</li>
</ol>
<hr />
<h2 id="attack-mitigation">Attack Mitigation<a class="headerlink" href="#attack-mitigation" title="Permanent link">#</a></h2>
<p>Stop the attack as quickly as you can, via any means necessary. Shut down servers, network isolate them, turn off a data center if you have to. Some common things to try,</p>
<ul>
<li>Shutdown the instance from the provider console (do not delete or terminate if you can help it, as we'll need to do forensics).</li>
<li>If you happen to be logged into the box you can try to,<ul>
<li>Re-instate our default iptables rules to restrict traffic.</li>
<li><code>kill -9</code> any active session you think is an attacker.</li>
<li>Change root password, and update /etc/shadow to lock out all other users.</li>
<li><code>sudo shutdown now</code></li>
</ul>
</li>
</ul>
<h2 id="cut-off-attack-vector">Cut Off Attack Vector<a class="headerlink" href="#cut-off-attack-vector" title="Permanent link">#</a></h2>
<p>Identify the likely attack vectors and path/fix them so they cannot be re-exploited immediately after stopping the attack.</p>
<ul>
<li>If you suspect a third-party provider is compromised, delete all accounts except your own (and those of others who are physically present) and immediately rotate your password and MFA tokens.</li>
<li>If you suspect a service application was an attack vector, disable any relevant code paths, or shut down the service entirely.</li>
</ul>
<h2 id="assemble-response-team">Assemble Response Team<a class="headerlink" href="#assemble-response-team" title="Permanent link">#</a></h2>
<p>Identify the key responders for the security incident, and keep them all in the loop. Set up a secure method of communicating all information associated with the incident. Details on the incident (or even the fact that an incident has occurred) should be kept private to the responders until you are confident the attack is not being triggered internally.</p>
<ul>
<li>The security and site-reliability teams should usually be involved.</li>
<li>A representative for any affected services should be involved.</li>
<li>An Incident Commander (IC) should be appointed, who will also appoint the usual incident command roles. The incident command team will be responsible for keeping documentation of actions taken, and for notifying internal stakeholders as appropriate.</li>
<li>Do not communicate with anyone not on the response team about the incident until forensics has been performed. The attack could be happening internally.</li>
<li>Give the project an innocuous codename that can be used for chats/documents so if anyone overhears they don't realize it's a security incident. (e.g. sapphire-unicorn).</li>
<li>Prefix all emails, and chat topics with "Attorney Work Project".</li>
</ul>
<h2 id="isolate-affected-instances">Isolate Affected Instances<a class="headerlink" href="#isolate-affected-instances" title="Permanent link">#</a></h2>
<p>Any instances which were affected by the attack should be immediately isolated from any other instances. As soon as possible, an image of the system should be taken and put into a read-only cold storage for later forensic analysis.</p>
<ul>
<li>Blacklist the IP addresses for any affected instances from all other hosts.</li>
<li>Turn off and shutdown the instances immediately if you didn't do that to stop the attack.</li>
<li>Take a disk image for any disks attached to the instances, and ship them to an off-site cold storage location. You should make sure these images are read-only and cannot be tampered with.</li>
</ul>
<h2 id="identify-timeline-of-attack">Identify Timeline of Attack<a class="headerlink" href="#identify-timeline-of-attack" title="Permanent link">#</a></h2>
<p>Work with all tools at your disposal to identify the timeline of the attack, along with exactly what the attacker did.</p>
<ul>
<li>Any reconnaissance the attacker performed on the system before the attack started.</li>
<li>When the attacker gained access to the system.</li>
<li>What actions the attacker performed on the system, and when.</li>
<li>Identify how long the attacker had access to the system before they were detected, and before they were kicked out.</li>
<li>Identify any queries the attacker ran on databases.</li>
<li>Try to identify if the attacker still has access to the system via another back door. Monitor logs for unusual activity, etc.</li>
</ul>
<h2 id="compromised-data">Compromised Data<a class="headerlink" href="#compromised-data" title="Permanent link">#</a></h2>
<p>Using forensic analysis of log files, time-series graphs, and any other information/tools at your disposal, attempt to identify what information was compromised (if any),</p>
<ul>
<li>Identify any data that was compromised during the attack.<ul>
<li>Was any data exfiltrated from a database?</li>
<li>What keys were on the system that are now considering compromised?</li>
<li>Was the attacker able to identify other components of the system (map out the network, etc).</li>
</ul>
</li>
<li>Find exactly what customer data has been compromised, if any.</li>
</ul>
<h2 id="assess-risk">Assess Risk<a class="headerlink" href="#assess-risk" title="Permanent link">#</a></h2>
<p>Based on the data that was compromised, assess the risk to other systems.</p>
<ul>
<li>Does the attacker have enough information to find another way in?</li>
<li>Were any passwords or keys stored on the host? If so, they should be considered compromised, regardless of how they were stored.</li>
<li>Any user accounts that were used in the initial attack should rotate all of their keys and passwords on every other system they have an account.</li>
</ul>
<h2 id="apply-additional-mitigations">Apply Additional Mitigations<a class="headerlink" href="#apply-additional-mitigations" title="Permanent link">#</a></h2>
<p>Start applying mitigations to other parts of your system.</p>
<ul>
<li>Rotate any compromised data.</li>
<li>Identify any new alerting which is needed to notify of a similar breach.</li>
<li>Block any IP addresses associated with the attack.</li>
<li>Identify any keys/credentials that are compromised and revoke their access immediately.</li>
</ul>
<h2 id="forensic-analysis">Forensic Analysis<a class="headerlink" href="#forensic-analysis" title="Permanent link">#</a></h2>
<p>Once you are confident the systems are secured, and enough monitoring is in place to detect another attack, you can move onto the forensic analysis stage.</p>
<ul>
<li>Take any read-only images you created, any access logs you have, and comb through them for more information about the attack.</li>
<li>Identify exactly what happened, how it happened, and how to prevent it in future.</li>
<li>Keep track of all IP addresses involved in the attack.</li>
<li>Monitor logs for any attempt to regain access to the system by the attacker.</li>
</ul>
<h2 id="internal-communication">Internal Communication<a class="headerlink" href="#internal-communication" title="Permanent link">#</a></h2>
<p><strong>Delegate to:</strong> VP or Director of Engineering</p>
<p>Communicate internally only once you are confident (via forensic analysis) that the attack was not sourced internally.</p>
<ul>
<li>Don't go into too much detail.</li>
<li>Overview the timeline.</li>
<li>Discuss mitigation steps taken.</li>
<li>Follow up with more information once it is known.</li>
</ul>
<h2 id="liaise-with-law-enforcement-external-actors">Liaise With Law Enforcement / External Actors<a class="headerlink" href="#liaise-with-law-enforcement-external-actors" title="Permanent link">#</a></h2>
<p><strong>Delegate to:</strong> VP or Director of Engineering</p>
<p>Work with law enforcement to identify the source of the attack, letting any system owners know that systems under their control may be compromised, etc.</p>
<ul>
<li>Contact local law enforcement.</li>
<li>Contact FBI.</li>
<li>Contact operators for any systems used in the attack, their systems may also have been compromised.</li>
<li>Contact security companies to help in assessing risk and any PR next steps.</li>
</ul>
<h2 id="external-communication">External Communication<a class="headerlink" href="#external-communication" title="Permanent link">#</a></h2>
<p><strong>Delegate to:</strong> Marketing Team</p>
<p>Once you have validated all of the information you have is accurate, have a timeline of events, and know exactly what information was compromised, how it was compromised, and sure that it won't happen again. Only then should you prepare and release a public statement to customers informing them of the compromised information and any steps they need to take.</p>
<ul>
<li>Include the date in the title of any announcement, so that it's never confused for a potential new breach.</li>
<li>Don't say "We take security very seriously". It makes everyone cringe when they read it.</li>
<li>Be honest, accept responsibility, and present the facts, along with exactly how we plan to prevent such things in future.</li>
<li>Be as detailed as possible with the timeline.</li>
<li>Be as detailed as possible in what information was compromised, and how it affects customers. If we were storing something we shouldn't have been, be honest about it. It'll come out later and it'll be much worse.</li>
<li>Don't name and shame any external parties that might have caused the compromise. It's bad form. (Unless they've already publicly disclosed, in which case we can link to their disclosure).</li>
<li>Release the external communication as soon as possible, preferably within a few days of the compromise. The longer we wait, the worse it will be.</li>
<li>Figure out if there is a way to get in touch with customers' internal security teams before the general public notice is sent.</li>
</ul>
<hr />
<h2 id="additional-reading">Additional Reading<a class="headerlink" href="#additional-reading" title="Permanent link">#</a></h2>
<ul>
<li><a href="http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r2.pdf">Computer Security Incident Handling Guide</a> (NIST)</li>
<li><a href="https://www.sans.org/reading-room/whitepapers/incident/incident-handlers-handbook-33901">Incident Handler's Handbook</a> (SANS)</li>
<li><a href="https://technet.microsoft.com/en-us/library/cc700825.aspx">Responding to IT Security Incidents</a> (Microsoft)</li>
<li><a href="http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=7153">Defining Incident Management Processes for CSIRTs: A Work in Progress</a> (CMU)</li>
<li><a href="https://www.first.org/conference/2008/papers/killcrece-georgia-slides.pdf">Creating and Managing Computer Security Incident Handling Teams (CSIRTS)</a> (CERT)</li>
</ul>
<aside class="copyright" role="note">
Copyright &copy; Spearhead Systems, Inc. &ndash;
Documentation built with
<a href="http://www.mkdocs.org" target="_blank">MkDocs</a>
using the
<a href="http://squidfunk.github.io/mkdocs-material/" target="_blank">
Material
</a>
theme.
</aside>
<footer class="footer">
<nav class="pagination" aria-label="Footer">
<div class="previous">
<a href="../during_an_incident/" title="During An Incident">
<span class="direction">
Previous
</span>
<div class="page">
<div class="button button-previous" role="button" aria-label="Previous">
<i class="icon icon-back"></i>
</div>
<div class="stretch">
<div class="title">
During An Incident
</div>
</div>
</div>
</a>
</div>
<div class="next">
<a href="../../after/post_mortem_process/" title="Post-Mortem Process">
<span class="direction">
Next
</span>
<div class="page">
<div class="stretch">
<div class="title">
Post-Mortem Process
</div>
</div>
<div class="button button-next" role="button" aria-label="Next">
<i class="icon icon-forward"></i>
</div>
</div>
</a>
</div>
</nav>
</footer>
</div>
</article>
<div class="results" role="status" aria-live="polite">
<div class="scrollable">
<div class="wrapper">
<div class="meta"></div>
<div class="list"></div>
</div>
</div>
</div>
</main>
<script>
var base_url = '../..';
var repo_id = 'spearheadsys/issue-response-docs';
</script>
<script src="../../assets/javascripts/application-997097ee0c.js"></script>
</body>
</html>