Quality Control and Effectiveness in Threat Prevention

Forta is the largest network of security intel in Web3. The decentralized Forta Network leverages machine learning and a community of security researchers to detect exploits, scams and other threats.

In the world of threat detection and prevention, particularly in Web3, the accuracy and reliability of security solutions are paramount. A false positive from Forta’s Attack Detector, for instance, could have significant repercussions, including revenue loss, increased user friction, and a decline in trust towards the solution. This is especially critical for protocols that might pause operations due to an incorrect alert.

A crucial aspect in assessing the quality of these solutions is transparency, as highlighted in the "Proof of Quality" blog post on Forta's website. This transparency becomes even more vital when considering tools like the Attack Detector, a leading solution in the Web3 threat prevention landscape. The Attack Detector functions by real-time monitoring of blockchain activities, covering all stages of protocol exploits – funding, preparation, exploitation, and money laundering. Detailed information on these stages can be found in the blog post "Web3 Kill Chain" and the Attack Detector's documentation.

In October 2023, the Attack Detector issued 450 alerts. Given that the number of actual protocol exploits was significantly lower, this initially suggests a high level of noise. However, a deeper analysis reveals a different picture. The key lies in the context: protocols primarily focus on alerts directly related to their operations. An alert is pertinent only if it involves the protocol's address. Consequently, while the overall number of alerts might be high, the relevant alerts for a specific protocol could be significantly lower, or even zero, unless it's under attack.

A study involving random sampling of protocols from DeFiLama, assessing the contextual precision of alerts, resulted in a 100% precision (with a 95%-100% confidence interval at 90% confidence value). Further, an analysis of publicly disclosed attacks in October showed a contextual recall of 70% for the Attack Detector.

Another critical factor is the timeliness of these alerts. It's essential that the alert triggers during the preparation phase of an attack as opposed to at or after exploitation. While this narrows down the number of relevant alerts, thereby increasing the contextual precision, it also reduces the contextual recall. For October 2023, early contextual precision was at 100%, whereas the early contextual recall was 40%.

This data leads to the conclusion that early contextual recall and precision are vital metrics for a threat prevention solution. The Attack Detector successfully identified almost half of all attacks early, with no false positives, demonstrating its efficacy in providing timely and relevant security alerts.

For those evaluating threat prevention solutions in the Web3 space, it's crucial to inquire about the solution's recall and precision, and how these metrics are defined. For the Attack Detector, these metrics are transparent and publicly disclosed monthly.

To experience the benefits of this advanced threat detection system, interested parties are encouraged to trial Forta’s Attack Detector, available here. This cutting-edge solution not only provides precise and contextually relevant alerts but also proves effective in early-stage threat detection, a critical aspect in securing digital assets and operations in the dynamic world of Web3.