Always A Bad Day For Adversaries

Tag: evaluation

Questions for Evaluating an External Threat Intelligence Source

I’ve spoken before on the cost of poor threat intelligence and its risk to an organization.  I’ve also spoken about the 4 qualities of good intelligence: relevance, timeliness, accuracy, and completeness. To better evaluate threat intelligence sources – DRIVE FOR TRANSPARENCY!  If you treat threat intelligence like a black box you’re going to lose.

Here are questions to use when evaluating an external source. These are just a starting point or additions to your own list based on your unique needs.

[Relevance] Why do I need threat intelligence?

Before you go out evaluating threat intelligence sources, you need to know what you’re looking for.  This is best done using a threat model for your organization and asking where threat intelligence supports visibility and decision making within that model.  Remember, your own threat intelligence is almost ALWAYS better than that produced by an external source.  External intelligence should complement your own visibility and reduce gaps.

Kudos: Thanks to Stephen Ramage for his comment highlighting the exclusion of such a critical question.

[Relevance] What types of intelligence are available?

Strategic country-level reporting? Cyber threats mixed with political threats?  Technical indicators?  Campaign behaviors?  Written context?  These all determine how useful, actionable, and relevant the intelligence will be for your organization.

[Relevance] Give me your context!

Make sure you understand the context provided with any data.  There is a difference between threat data and threat intelligence.  Intelligence helps drive effective decision-making.  Context makes data relevant.

[Relevance] Which threat types?

Is it limited to botnet C2 nodes?  Commodity threats in general?  Does it cover targeted threats?  Does the threat intelligence provide insight into your threat model?

Related Questions: How many unique threats are distinguishable in the intelligence?

[Relevance] How many direct threats to my organization or those in my industry has your intelligence identified?

Has the source ever shown direct success in highlighting threats in your industry?

[Relevance] How is the intelligence made available to consumers?

If the intelligence is not provided in a usable form, it will not be successful.

[Relevance] What types of use-cases produce the best experience/feedback?  In which use cases has your intelligence failed?

This is a soft-ball question but one which should provoke a good question-answer session.  The answers will illuminate their decisions developing the intelligence and highlight where the intelligence may fit best (or not fit at all).

Related question: What threat model is this intelligence attempting to address?

[Completeness/Relevance] What is the source of the intelligence?

Is this intelligence derived from human sources crawling the dark-web?  Global network apertures?  VirusTotal diving?  This question should frame their visibility into threats and inform the types of intelligence expected.  This also highlights any natural biases in the collection.  Look for sources of external intelligence which complement your own internal threat intelligence capabilities.

[Completeness] What phases of the kill-chain does the intelligence illuminate?

Understand how wide, against any single threat, the intelligence goes.  Does it only show C2, or will it also illuminate pre-exploitation activities as well.  The wider the intelligence, the greater the likelihood of it being useful.

[Completeness] What is the volume and velocity of the intelligence?

“How much” intelligence is actually produced?  Numbers don’t matter that much – but if the number is ridiculously small or ridiculously large, it is an indicator of possible issues.

[Accuracy] How is the intelligence classified and curated?

Drive for transparency in their process which helps improve your evaluation on accuracy. Be wary of “silver bullet” buzz-word answers such as “machine learning” or “cloud.”

[Accuracy] How is the intelligence validated?

Do you want to track down false positives all day?  No!  Do you want to rely on poor analysis? No! Make sure this question gets enough attention.

Related questions: How often is it re-validated?  How are false positives handled?  How can customers report false positives?  What is your false positive rate?  How many times in the last month have you had to recall or revise an intelligence report?

[Accuracy] Does the intelligence expire?

Expiration of intelligence is key.  Is there a process which continuously validates the intelligence?

[Timeliness] How quickly is the intelligence made available to customers after detection?

Related questions: What part of your process delays intelligence availability?  What is the slowest time to availability from initial detection?

CART: The 4 Qualities of Good Threat Intelligence

I write often of poor quality threat intelligence which pervades the security community.  Poor quality threat intelligence not only has a heavy cost on its consumers, it also threatens the confidence threat intelligence consumers place in their providers.  Confidence is the cornerstone of threat intelligence.  Nobody will take intelligence from an untrustworthy source and act – at least they shouldn’t.  It is important that the producer and consumer trust each other.  That trust needs to be based on transparency and verification.

However, how does one appropriately assess threat intelligence?  The first step must be to identify the qualities which define “good” threat intelligence.  However, these are not binary qualities – there is a clear gradient based on use case.  Timeliness is a good example of this gradient as some intelligence (likely more strategic) has a more fluid timeliness requirement while tactical threat intelligence has stricter requirements.

Further, one single threat intelligence source will not likely be able to satisfy all qualities simultaneously.  For instance, it is unlikely any one provider will have complete visibility across Diamond elements or Kill Chain phases and consumers will have to rely on more than one to achieve satisfactory completeness.

The four qualities are (CART): Completeness, Accuracy, Relevance, and Timeliness.

Completeness

Threat intelligence must be sufficiently complete to provide effective detection and (hopefully) prevention.  For instance, providing a domain indicator used in the exploitation of only one victim is not sufficient for other victims and therefore the intelligence is effectively incomplete and unhelpful.

Accuracy

Threat intelligence must save organizations more in success than it costs them in errors and mistakes.

Relevance

Threat intelligence must address a threat to the organization in a method that allows for effective action.  Intelligence addressing threats not faced by the organization is of no value.  Further, intelligence delivered in a type or method not usable by the organization is also unhelpful.

Timeliness

Threat intelligence must be received and operationalized fast enough to make an impact more valuable than the cost of the threat intelligence itself.

Powered by WordPress & Theme by Anders Norén