The value of the True Positive
As originally published on Vectra's Unfiltered at https://www.unfilteredcxo.com/
Cybersecurity is afflicted with the duty of “proving a negative” all the way up to the Board room. We can learn some tricks from incident response and threat intelligence to tackle the art of distinguishing the lucky from the good.
When it comes to incident response, it is challenging – but essential – to define criteria for closing an investigation. Enter the true positive. When someone says that they did not see any sign of a particular activity, it is essential for them to first validate a true positive. This means mimicking a similar event to the one being sought and verifying that the tools, processes, and data needed to identify it are available. If you are trying to confirm that data was not stolen, for example, the absence of evidence is not adequate. You should reproduce the exfiltration of harmless data through the suspected path to ensure that it would have been detected and evidenced by your process. If you are trying to determine if a specific IP address touched your environment during a specific time frame, first validate that you can locate evidence of other IP addresses touching the same environment during the same time window. Seasoned incident responders will know the trick of finding logs by “forging” an event with an identifiable marker. For example, try logging into your public-facing web site with the username findmeifyoucan and then searching for that string in every tool possible an hour later (since log shipping delays are unfortunately real).
Moving up to a tactical level, this concept is useful in responding to regulatory inquiries. Operators of highly regulated critical infrastructure will be familiar with any operational glitch being immediately chased by regulators asking if it was related to cyber. The concept of “reproducibility” can be valuable here. Recreate your issue in a non-production environment to establish ground for ruling out a cyber cause.
Moving up again - this time to a strategic level - the concept of using a true positive to validate conclusions can be a core tenet of an effective organizational culture in cyber and beyond. This is an evolution of the classic trust but verify mantra we've been hearing in security for years (I personally leave out "trust but"). That can be easier said than done when you are the one receiving information and deciding whether or not to trust it. Many times, though, we are the source. The tone we use to communicate is crucial when it comes to inviting criticism and “verification” from the listener. The science of threat intelligence deals with this concept every day. Intelligence documents begin stanzas with qualifiers such as “we assess with moderate confidence that”. Qualifying your statements not only reveals the amount of data and facts relied upon to reach a conclusion, but invites the listener to re-perform the analysis without fear of hurting anyone's feelings. This is particularly important for senior practitioners with many years of experience. It is important for us to remember that not only does the landscape and technology change over time, but it's also always possible that we were wrong to begin with back when we were cutting our teeth. It is far preferable to make a statement such as “ you should re-perform this, but when I looked into this in the past I did not find anything interesting.” This is a healthier way to approach important technical assertions than “don't waste your time - there is nothing there!” The latter phrasing is intimidating and sets up junior practitioners to feel like they may be insulting you if they perform an investigation.
Finally, self-criticism should be the foundation of a security strategy itself in the form of frequent testing. This practice enables a CISO to evolve from “we had no incidents this quarter” to “we saw no incidents, but synthesized an attack via a red team, confirmed most of our detections were effective, and improved the ones we found lacking.”
Culture is king when it comes to a security organization, and inviting self-criticism is a core component at all levels.