How Forta’s Predictive ML Models Detect Attacks Before Exploitation

Community Spotlight: This article was guest authored by Forta community member Mariko Wakabayashi, machine learning engineer at OpenZeppelin.

Hi I’m Mariko, a machine learning engineer at OpenZeppelin focusing on helping the Forta community use machine learning to detect Web3 security threats. Recently, I created a ML based Forta detection bot that began to successfully detect incoming hacks since launching in early October 2022. Using Forta’s constant scanning of 7 different blockchains, the model was able to detect over 4 recent hacks worth $23.5M by mid November 2022, in real-time, before they occurred. 

This is a large milestone not just for the Forta Network but also for Web3 security, further demonstrating that predictive threat detection is possible using ML. In this blog post I describe how this became possible and share lessons learned alongside resources. I hope this can serve as an invitation to a broader group of developers and data scientists to explore even more ML opportunities on Forta improving Web3’s security. 

Why Malicious Contract Detection?

In early October this year, I created a detection bot that uses a machine learning model to detect malicious smart contract creations. Some of you may be wondering, how does detecting malicious contracts help with early threat detection? Well, most hackers in Web3 use smart contracts to execute an attack.

According to the Web3 kill chain, there are two distinct stages that can happen before an exploitation: funding and preparation stages. In the funding stage, the hacker first gathers funds for the attack. Next, in the preparation stage, the hacker deploys the exploit contract, followed by the exploitation stage in which the contract is invoked to execute the attack. This means we can mitigate or prevent attacks if we can catch any suspicious contract as it deploys. That also includes exploits executed in a single atomic transaction such as flashloan attacks, which requires a contract deployment first to initiate this type of attack. 

According to the “SoK: Decentralized Finance (DeFi) Attacks” paper that analyzed close to 200 real-world incidents, malicious smart contract bytecode analysis has the potential to prevent attacks. 

“While the adversarial smart contract bytecode is already publicly available in the rescue time frame, the incident has not yet occurred. As such, defensive tools can theoretically reverse engineer the contract bytecode and determine its strategy using methods such as symbolic analysis, static analysis, and fuzzing, potentially mitigating or preventing harm.” The authors of the paper investigated past incidents’ rescue time frame (time duration between contract deployment and invocation) and observed sufficient time to react in certain cases. 

“The average rescue time frame for protocol layers is 1±4.1 hours, with the longest rescue time frame being 26.5 hours.” This suggests that, given enough examples of malicious smart contracts, a machine learning model may be able to differentiate malicious from benign contracts. 

In order to test this theory out, I trained a machine learning model to analyze past hacks and benign Ethereum smart contracts, learn common hack patterns, and detect new malicious activity. More specifically, the model looks at the contract’s disassembled EVM opcodes to decide whether it’s malicious. In the Web2 world, analyzing opcodes has been common for malware detection where the source code is not available.  

After deploying the bot, I observed that the ML bot was indeed able to detect exploits before it happened. For example, here are a few this ML bot detected before exploitation:

- The $15.8M Team Finance Hack 1 hr before exploitation: bot alert
- The $7.5M DFX Finance Hack 6 minutes before exploitation: bot alert
- The $300K Olympus Dao Hack 2 minutes before exploitation: bot alert

Had these protocols been subscribed to Forta’s attack detector feed, and taken timely action, it could have prevented over $23.5 million of assets from getting exploited. 

How did the ML model detect hacks?

For a machine learning model to detect new malicious contracts, it first needs to learn from past malicious and benign smart contracts to be able to differentiate the two. 

So for malicious examples, I gathered around 100 contracts that had the Etherscan labels “exploit” or “heist” from Luabase, a SQL data platform for querying blockchain data. 

Since there are not any contracts explicitly labeled as benign, I treated Etherscan verified contracts as benign. In most cases, verified contracts are safe (note that phishing contracts go through verification to impersonate legitimate token contracts and trick users) because their source code and ABIs are made public for anyone to interact with, so I gathered around 130,000 verified smart contracts.

After gathering enough past examples, the question was what smart contract data to use for training. There were three options: 

1.  Contract source code
2.  Contract bytecode
3.  Contract opcodes disassembled from bytecode

Contract code is commonly written in a high-level language like Solidity and can be made public  if a smart contract was verified. However, most exploit contracts are not verified because exploiters want to hide their activities, so this was not an option. 

The next option was contract bytecode. Solidity code is compiled down to bytecode before deploying the contract on the Ethereum network, and this bytecode is retrievable as a string of bytes from all contract creation transactions. This data can be found in the contract creation tx’s input data field if an EOA initiated the creation. If a contract deployed a new contract, this new contract’s bytecode can be found in the EVM trace. 

A ML model can learn from the bytecode and detect new malicious contracts. However, the challenge with this data is that it can be hard for humans to comprehend, making it difficult to debug and understand the ML model’s predictions.

So the last option was contract opcodes. Opcodes are machine code the Ethereum Virtual Machine can interpret and execute. This data can be obtained with tools like evmdasm that disassemble contract bytecode down to opcode instructions.

According to this paper that analyzed more than 30,000 verified smart contracts, they were able to find meaningful patterns and identify common and rare opcodes. 

This means a machine learning model has the potential to learn common opcode patterns found in not only verified benign contracts, but also non-verified exploit contracts. 

So after collected this data, the next question was what ML algorithm and features to use. Input for a ML model – typically a processed piece of data a ML model needs during training and serving (e.g. total number of approval transactions can be a feature for a phishing scam detection model).

In the natural language processing domain, there is a well studied task of identifying sentiment in documents. This is a supervised classification problem where machine learning models are trained to tell apart positive from negative text by reviewing past examples. 

Malicious contract detection can also be treated as a supervised classification problem where the model is taught to identify new malicious patterns by studying past contracts. 

As for features, the opcodes are essentially a series of predefined number of opcode instructions which looks very similar to a body of text or document with a fixed number of words in the dictionary. So treating the opcodes as a body of text, I borrowed some NLP techniques used for sentiment classification. This enabled extracting important opcodes that are commonly found in malicious and benign contracts. 

The technique I used is called TF-IDF (term frequency–inverse document frequency), and it converts text into a numerical representation. This is a commonly used technique in machine learning called “bag of words” to tokenize the text and count the occurrences of words in the document. In this specific case, I treated the opcode instructions as words. 

I used TF-IDF to split opcodes into n-grams or chunks of 1 to 4 opcodes at a time because this helps retain the opcodes’ relative position information in the contract. I excluded the opcode operands because contracts may have widely varying values and increase the noise.

Example opcode n-grams

N-gram typeOpcode ExampleunigramPUSH1bigramMLOAD PUSH20trigramADD PUSH1 SHA3four-gramPUSH4 EQ PUSH2 JUMPI

These chunks of opcodes are then fed into a logistic regression model that predicts malicious contracts. Logistic Regression is a classification algorithm that’s used to predict the binary outcome’s probability. For example, it can be used to predict whether a credit card transaction is fraudulent or if an email is spam. 

After training, I evaluated the model using stratified 5-fold cross validation and a decision threshold of 0.5. The model predicted malicious contracts with an average precision of 88.6% and recall of 59.4%. 

Here are a few examples of exploit contracts it was able to classify:

-  Wintermute 2 Exploit Contract: 0x0248F752802B2cfB4373cc0c3bC3964429385c26
-  Audius Exploit: 0xbdbB5945f252bc3466A319CDcC3EE8056bf2e569
-  Inverse Finance Exploit:  0xf508c58ce37ce40a40997C715075172691F92e2D

Deploying the ML model on the Forta Network

Next, I deployed the model as a detection bot on the Forta Network (to learn more, please check out this guide on deploying ML models). Deploying this model on Forta significantly reduced the infrastructure setup required for model input data retrieval, serving, monitoring, and alerting. 

The Forta Network has thousands of scan nodes that continuously scan new transactions on seven EVM-compatible chains, in real-time, and broadcast them to detection bots. For this ML model, the detection bot retrieves newly deployed contracts from the scan nodes, so the last steps were converting the contract data into model features, producing model predictions, and formatting them into actionable alerts. The detection bot was then configured to detect malicious contracts on all 7 chains so that anyone can subscribe and receive alerts via slack, telegram, or email. 

Can we understand why the ML bot triggered?

One of the challenges with ML predictions is understanding why the models yield certain results, and this was also the next step in my process. More specifically, I wanted to understand the opcode patterns the model attributes to maliciousness.

Fortunately, there is a tool we can use to help us explain ML models’ predictions. This is called ELI5, and a python package that helps debug machine learning models and explain their predictions. This package supports model explanation in most machine learning frameworks including the Logistic Regression model that was built with scikit-learn. 

With the ELI5 tool, I was able to extract the opcode patterns the model thinks are common and uncommon in a malicious contract.

Top 10 Common and uncommon opcode patterns in malicious contracts

Top 10 Common OpcodesTop 10 Uncommon OpcodesPUSH20 PUSH20 AND PUSH4PUSH32PUSH20 PUSH20 ANDMLOADPUSH20DUP2 MSTORE PUSH1PUSH20 PUSH20MSTORE PUSH1 ADDADD PUSH1 DUP2 MSTOREPUSH1 ADD SWAP1 DUP2PUSH1 ADD PUSH1 DUP2PUSH3 JUMPI PUSH1DUP1 ISZERO PUSH3 JUMPIADD PUSH1 SHA3DUP1 ISZERO PUSH3PUSH1 ADD PUSH1 SHA3ADD MLOAD PUSH1 ADDSHL SUB AND PUSH4JUMPDEST POPDUP5

Analyzing these patterns, and with the help from OpenZeppelin’s security researchers, I arrived at several hypotheses for why the model predicted certain contracts as malicious:

Patterns including PUSH20 are common in malicious contracts: PUSH20 can be used to push addresses on the stack, and if there are many addresses, it may suggest that a malicious contract is interacting with many protocols to prepare an attack, initiate an attack, and move stolen funds. In Team Finance’s case, the hacker contract interacted with the protocol contract, drained multiple Uniswap liquidity pools, and transferred stolen funds to multiple money laundering addresses.

PUSH32 is not common in malicious contracts: PUSH32 can be used to push event signatures (to learn more about events in Ethereum Contracts, check out A Guide to Events and Logs in Ethereum Smart Contracts) that are defined as 32 bytes of the keccak hash (sha3). Few PUSH32 opcodes can mean little or no events are emitted, likely aligning with a hacker’s goal to stay hidden. Moreover, a hacker typically will not want to advertise what the contract is doing. In contrast, a benign protocol may need to broadcast events to integrate with their user interfaces.

SHA3 is not common in malicious contracts: SHA3 computes the keccak-256 hash of a given data in memory. This can be used to authenticate and control users’ access to admin actions. However, malicious contracts may not have access control or manage user data in general, so it’s less likely for them to generate hashes.

With the ELI5, it’s also possible to analyze predictions individually and see certain opcode patterns that are highlighted. For example, the screenshot below shows some of the model’s prediction explanations on the Team Finance Exploit Contract. Opcodes highlighted in purple are considered malicious, and blue suggests benign.

In the above screenshot, we can see that PUSH20 opcodes are common in the Team Finance exploiter contract.

This tool is also useful for debugging false positives (i.e. contracts that were incorrectly labeled as malicious). In fact, the model isn’t perfect and has been labeling MEV contracts as malicious too. 

This may be due to MEV contracts displaying similar characteristics as malicious contracts. MEV contracts are incentivized to stay discreet and hide their unique strategies from their competition. What’s more, they most likely work with many addresses to execute liquidation or arbitrage. For example, MEV bots tend to use the following mask for an address type to parse out addresses for arbitrage: PUSH20 0xffffffffffffffffffffffffffffffffffffffff

Areas of improvement

There are several areas of improvement for this model. To mitigate misclassifying MEV contracts, it may make sense to include unverified MEV contracts in the benign contract dataset so that the model can differentiate them from malicious contracts. It may also help to include or further process certain operands such as PUSH20’s operands to differentiate pushing addresses vs pushing a mask value. 

Pushing an address: PUSH20 0x161cebB807Ac181d5303A4cCec2FC580CC5899Fd
Pushing an address type mask: PUSH20 0xffffffffffffffffffffffffffffffffffffffff

Next, this model was built specifically for Ethereum, but it is currently deployed on other EVM-compatible chains as well. Certain attacks may be more common on certain chains, so it may be worth building chain-specific models. For example, according to web3rekt, there were 2.2x more hacks reported on BNB Smart chain than Ethereum. 

There are also other approaches to explore like deep learning that can extract patterns more effectively. As you can see, feature engineering is a bit of trial and error that also requires domain knowledge. For the ML model in this blog post, this meant consulting with OpenZeppelin’s security researchers and opcode experts to understand the model predictions. Deep learning, on the other hand, may be able to streamline this process by creating features automatically without human intervention.

There has been extensive research using deep learning methods for Web2 malware detection. Here are a couple examples:

-  Op2Vec: An Opcode Embedding Technique and Dataset Design
-  Malware Classification with Word Embedding Features
-  Detection of Prevalent Malware Families with Deep Learning
-  Data Augmentation for Opcode Sequence Based Malware Detection

As you can see, the recent ML detection was just the beginning, and there are many more opportunities for improvement using ML.

In order to invite a much broader community of ML contributors, I made the training dataset public. So if this is the type of project that interests you, please check out this doc to get access, build a better model, and share it with me and the rest of the Forta community!

Final takeaways

The hacks mentioned earlier were unfortunate, but was one of the first times the Forta Network showed its potential to detect threats earlier than previously thought possible. Had these protocols been subscribed to Forta’s attack detector feed already and taken timely action, it could have saved over $23.5 million of assets. If you are a protocol dev, please consider subscribing to the attack detector feed to prevent attacks. 

If this kind of work interests you, such as analyzing blockchain data and using ML to detect threats, the Forta Foundation is also funding a full-time data scientist (as of Dec 2022), so consider applying if you’d love to do this full time. 

If you’re still excited to contribute in smaller ways, join the growing community of data scientists and ML engineers working on Forta. This group will work together to build useful solutions on Forta and collaborate to add more value to Web3 security. Please message me in Discord if you’re interested!

Acknowledgements

Thank you to the following individuals who shared valuable insights and feedback on this work:

-  Christian Seifert, Security Researcher in Residence, Forta
-  Raul Martinez, Forta Smart Contract Core Developer
-  OpenZeppelin Security Researchers Andrew D. Anderson, Dario Lo Buglio, Franco Valencia, Yuguang Ipsen, and Jose Carlos Montero