Always A Bad Day For Adversaries

Tag: quality

CART: The 4 Qualities of Good Threat Intelligence

I write often of poor quality threat intelligence which pervades the security community.  Poor quality threat intelligence not only has a heavy cost on its consumers, it also threatens the confidence threat intelligence consumers place in their providers.  Confidence is the cornerstone of threat intelligence.  Nobody will take intelligence from an untrustworthy source and act – at least they shouldn’t.  It is important that the producer and consumer trust each other.  That trust needs to be based on transparency and verification.

However, how does one appropriately assess threat intelligence?  The first step must be to identify the qualities which define “good” threat intelligence.  However, these are not binary qualities – there is a clear gradient based on use case.  Timeliness is a good example of this gradient as some intelligence (likely more strategic) has a more fluid timeliness requirement while tactical threat intelligence has stricter requirements.

Further, one single threat intelligence source will not likely be able to satisfy all qualities simultaneously.  For instance, it is unlikely any one provider will have complete visibility across Diamond elements or Kill Chain phases and consumers will have to rely on more than one to achieve satisfactory completeness.

The four qualities are (CART): Completeness, Accuracy, Relevance, and Timeliness.

Completeness

Threat intelligence must be sufficiently complete to provide effective detection and (hopefully) prevention.  For instance, providing a domain indicator used in the exploitation of only one victim is not sufficient for other victims and therefore the intelligence is effectively incomplete and unhelpful.

Accuracy

Threat intelligence must save organizations more in success than it costs them in errors and mistakes.

Relevance

Threat intelligence must address a threat to the organization in a method that allows for effective action.  Intelligence addressing threats not faced by the organization is of no value.  Further, intelligence delivered in a type or method not usable by the organization is also unhelpful.

Timeliness

Threat intelligence must be received and operationalized fast enough to make an impact more valuable than the cost of the threat intelligence itself.

The Cost of Bad Threat Intelligence

There is no doubt that threat intelligence is now “a thing.” At RSA 2015 I couldn’t help but notice how many vendor booths were hawking their relevance to threat intelligence.  I hear about a threat intelligence start-up almost weekly.  That is not surprising given venture capital is flowing and C-suite customers are now investing in “threat intelligence.”  Everyone wants a piece of the pie.

While market growth for threat intelligence produces innovations it also produces negative by-products (welcome to capitalism).  The most concerning by-product is the reduction in threat intelligence quality.

A growing number of published threat intelligence reports contain inaccuracies and poor analysis.  A growing number of indicators across a variety of producers are either stale, irrelevant, or generate so many false positives to be useless.

What so many fail to realize is the cost of poor quality intelligence.  Here are some of the costs:

  • If a single threat intelligence-sourced alert generates $1000 worth of time to investigate a false positive, it is easy to see how that relatively small amount can multiple within an organization and across enterprises worldwide.
  • If an intelligence producer reports incorrectly categorizes a threat as APT (say instead of cyber crime) an organization’s security response to the threat will be (and should be) different likely involving a deeper investigation.  Again, this additional, and likely unnecessarily deep, investigation is costly in both time and resources.
  • Every poor quality report costs time to read and digest.  Time that could be spent understanding a high-quality report.
  • Every poor association or correlation derails an analytic effort at an organization.

Because organizational security resources are finite and already stretched thin these mistakes, errors, and poor practices consume critical resources which could be spent on other problems and reduces the security of an organization.

Two market elements have caused this quality reduction:

  • A need to garner attention in the growing cacophony of the threat intelligence market feeding a “first to publish” mentality which usually results in a “rush to publish.”
  • A lack of customer education resulting in a poor evaluation of providers thereby incentivizing the wrong aspects of threat intelligence – such as volume of indicators over their quality or relevance

Obviously, only threat intelligence providers can solve the problem, but what pressures can help drive effective change?  Here are some:

  • Threat intelligence customers armed with evaluation criteria (particularly quality metrics) which helps them leverage threat intelligence effectively without generating unnecessary costs – this will help create market drivers for higher quality
  • Industry must self-police bad intelligence by being honest with ourselves and each other.
  • Threat intelligence aggregation platforms should have quality assessment capabilities informing the intelligence consumer of potential problems (likewise they are also be in a position to highlight timely, relevant, and unique intelligence of great value)
  • Threat intelligence analysts trained in analytic tradecraft stressing quality and accepting an ethical duty

Security professionals practicing threat intelligence must understand the implications of mistakes and poor analysis.  Bad intelligence can and does decrease the security effectiveness of an organization. Therefore it is an ethical duty of the threat intelligence practitioner to reduce errors. Threat intelligence is difficult – intelligence by definition attempts to illuminate the unknown and works by making judgments with imperfect data – errors are natural to the domain.  But, with proper practices and procedures bad intelligence can, and must, be minimized.

Powered by WordPress & Theme by Anders Norén