UPDATE: (Sep 15th, 2020) ALL FUNDS ARE RECOVERED!
All funds have been recovered from the attacker. We are restoring the system. Yesterday, we did an open live-stream with both the Co-founders Tom Bean and Kyle Kistner addressing the issue. You can watch the video here.
Highlights from the Call and The Double Dipping Problem:
The team stayed up for 36 hours and recovered the funds through decisive action. The funds are now in the team wallet and being used to restore the lending pools.
A bug bounty was paid out to Marc Thalen for the amount of $45,000. As per our bug bounty policy, we polled three security experts in the space to determine whether a bug bounty was applicable and how much it should pay. The experts gave wildly differing opinions ranging from $2,500 to $30,000, all of them noting a potential “double dip” problem allowing attackers to execute an exploit and subsequently claim the bounty. They cited this as an issue with issuing a top level payout, as Marc had only reported the issue when the attack had mostly concluded. Security expert Nick Johnson weighed in on Twitter, leading to the conclusion that pro-rating the bounty in proportion to the amount of funds saved would solve the Double Dipping problem.
Marc agreed to the $45,000 bounty which he found reasonable. We’re thankful to Marc for noticing what was happening and reaching out to us.
On the product side, Lend is now enabled for all the assets on Fulcrum.
PeckShield just issued a letter to the bZx community. We’d like to thank PeckShield for all the support and confidence they’ve shown in us.
Previous update
Timeline
- The team noticed a strange movement in the protocol TVL.
- Identified anomalous behavior with the _internalTransferFrom() function on the iToken contract.
- Minting and burning of iTokens was paused as the fix was identified.
- Borrowing and trading was not impacted.
- A new version of the affected iToken contracts were deployed with the balances corrected for duplications.
- The patched code was sent to Peckshield and Certik for review.
- Minting and burning of iTokens was unpaused.
Technical Details
Every ERC20 token has a transferFrom() function that is responsible for transferring tokens. It was possible to call this function to create and transfer an iToken to yourself, allowing you to artificially increase your balance.
Here is what happened:
- Transfer function was called with same _from and _to address
- Immediately _internalTransferFrom was called with the same arguments
- The following lines of code are at fault:
At this point having _from and _to the same address will result in _balancesFrom _balancesTo being equal.
Then
Above decreases the balance of _balancesFrom and increases balance of _balancesTo and lastly the most important part is storing _balancesFromNew and _balancesToNew. The user was effectively able to increase his balance artificially.
The Patch
The fix saw the move of balancesTo being set after the deduction from balances[_from]. This prevents a user from inflating their balance.
The patched code was sent to Peckshield and Certik for review. Both Certik and Peckshield have approved the changes.
Insurance Fund Debt
The following debts have been added to the insurance fund:
- 219,199.66 LINK
- 4,502.70 ETH
- 1,756,351.27 USDT
- 1,412,048.48 USDC
- 667,988.62 DAI
Precautions
The protocol was heavily audited by top security firms Peckshield and Certik. The Peckshield audit was 12 person weeks. This is the same amount of time that Peckshield audited the Multi-collateral DAI (MCD) contracts for MakerDAO. The Certik audit was 7 person weeks. Additionally, we performed extensive automated testing. Unfortunately, audits are not silver bullets. Our protocol is the most powerful, fully functioned lending protocol in the space, and this means that there is a lot of code to cover. It is partly its scope and ambitions that makes it more difficult to secure than many other projects. As disclosed on our security page, we kept control of the administrative key for the protocol initially to be able to address any incidents as they arose in the initial days of the relaunch.
Much like MakerDAO backstops system debt with the value of the token, so too does our system. We are grateful that our precautions and system design are capable of resolving incidents like this without issue.