The unluckiest DeFi protocol? A personal take on bZX’s tumultuous year

Decentralized finance platform bZX has frequently been in the spotlight this year, only not for the right reasons. Most DeFi platforms popular today, including bZX, began their journey around 2018, at the tail-end of the initial coin offering boom. In 2019, DeFi started gaining traction, though it was still a somewhat ignored sector of the industry.

As growth continued, suspicions began to rise that major hacks, typical of the digital asset sector, were overdue. Due to the complexity and novelty of these platforms, it was reasonable to assume that not all of them were impervious to bugs.

This year can be characterized as a testament to the saying, “When it rains, it pours.” Unfortunately for bZX, it became the first major DeFi platform to suffer a large hack, in February of 2020. It also became the second platform to be exploited, as two back-to-back attacks crippled the project and forced it to miss out on the majority of the DeFi boom.

While some other platforms followed suit, bZX’s woes were not truly over: shortly after its relaunch in September, it was hacked once again. While it may appear to have been the final blow for the project, co-founder Kyle Kistner remains optimistic that the platform will bounce back.

“Ever since we got the money back and the funds are safe, we’ve got a whole bunch more total value locked and a huge amount of trading volume,” Kistner said in an interview with Cointelegraph. “We haven’t quite made it back to where we were, but our trading volumes have been really exploding.”

Kistner reiterated many times throughout the interview that despite all these hacks, the platform never conclusively lost its users’ money. The early victims were refunded, while the September hacker was essentially caught red-handed through blockchain analytics and returned the money. Be that as it may, Kistner and the bZX team’s journey this year has been tumultuous, to say the least.

Caught with their drinks up

Cointelegraph: The first bZX hack occurred on Feb. 14 while the team was away at the ETHDenver conference. How did you learn of the attack?

Kyle Kistner: We were at this afterparty, it was the Keep and Compound happy hour. We’re sitting there, we’re talking with Ryan [Berkun, CEO of Tellor] and he was telling me about how he had just put in some money in Fulcrum, he was showing me the interest rates. I noticed that the interest rates for ETH were abnormally high. And I was like, “Oh, that’s really strange.”

I talked to Tom [bZX’s CEO] about it and I felt like something’s really weird about it. Later in the night we got a message from Lev Livnev from DappHub, who noticed a strange transaction, which was basically the one that created this very high interest on the iETH pool.

And you know, we had been drinking and so we needed to sober up. It was this crazy experience, it was 11:30 at night, we were partying with the rest of the industry people and suddenly you’re thrust into this very serious situation. As we were investigating, we realized that we need to pause the whole system.

There wasn’t really a pause button designed on this thing, but we did hack together a solution by disabling the oracle whitelist. This worked to prevent more money from being taken.

Then I called my wife, I’m saying “I don’t know how I’ll be able to face the people in the industry, go back down to ETHDenver, see everybody there.” I thought for a moment that maybe I’ll just pack my bags and go home, but my wife talked me out of it. Tom was just sitting there, catatonic for a little bit, the whole thing washing over him.

The second hack

Eventually Kistner and the team regrouped. They managed to catch a lucky break — the protocol did not automatically spread the loss of more than 1,100 ETH, worth about $300,000, among all platform users. This gave them a chance to fully return the money down the line and allowed the business to continue. “That gave us a lot of morale,” Kistner said.

When the team showed up at ETHDenver the next day, Kistner said that “people were actually congratulating us. There was a lot of support, people were saying, ‘We’re builders, you’re builders, we’re all in this together.’”

CT: And then the second attack happened. How did you find out about it?

KK: We had just arrived at this restaurant. We were up at the ski retreat in Colorado, we helped organize it and we were really excited about it. We ordered all of this food, and Tom is looking at his phone — he likes to just go through the different transactions that are on the system, especially if anything looks weird or strange. So he looked at this one transaction and it looked really weird because it had contracts being deleted and it had a flash loan and it had basically small amounts being called repeatedly over and over again.

So we looked at that transaction and it took us about two seconds to be like ‘Ok, somebody got hacked.’ This doesn't look right at all. We knew it involved our system.

So the food arrived, it was like a hundred dollars worth of food for three people. The moment it arrived on the table, I got up and I said, “Can I pay the bill?” and handed them the card. Tom was already sprinting home and we just all booked it, we just all started running through the snow and, you know, it was a seven-minute jog from the restaurant to our place.

We manned our battle stations, paused the system, started to triage and diagnose the issue. [...] By that point we were like ‘we know how to handle this, if there’s some money taken it’s not the end of the world.’ Unfortunately, since lightning did strike twice, a lot of the goodwill that people were extending us before had been substantially eroded.

Reflecting on what went wrong

The two hacks forced the team to shut down and rebuild the protocol. Since then, other projects saw vulnerabilities exploited as well, but none had multiple hacks occur within a short span.

CT: The number of breaches suffered by bZX raises questions about the project’s practices. Could it just be bad luck, or is there something deeper at play?

KK: It’s not a coincidence. So there’s two things: one is that we made a mistake, and we had a security auditor that kind of didn’t completely do [their job]. There’s one issue I’m trying to get at here — basically there’s a number of factors that went into why we had Kyber as an oracle [the primary vulnerability resulting in the second hack].

It was a conceptual vulnerability that really an auditor should have caught, but we shouldn’t have been using it. We had an understanding that Kyber wasn’t optimal, but we kind of stubbornly refused to centralize the oracle. We didn’t have Chainlink, which we could just plug in at the time, so the only other option was to centralize the oracle.

Now, the first hack was basically a typo-level bug. I think this was due to not having proper processes in place. [...] We were a small company. We were not backed by a whole bunch of venture money, like a lot of the other lending protocols. Now we are, we’re a much larger and much more mature company.

Auditors are not one and the same

Auditing smart contracts is considered a crucial step before the protocol’s launch. Unaudited protocols are considered less safe, so much so that Yearn Finance’s creator says he purposefully dampened excitement about his project by withholding the fact that the protocol was audited.

CT: So what exactly happened with the audit of your code by ZK Labs?

KK: I feel like somebody needs to know this story. So we were new and we were kind of green to the industry. We had just built this version one of our protocol, it was like the beginning of 2018. We just put our stuff on the testnet, but we didn’t really know the security auditors in the space.

So we asked around and first got referred to the Acacia Group. [...] They scoped it out and they basically said, “We’re out of our depth here.” So we needed to find a different auditor and eventually we found ZK Labs. We thought ZK Labs was super reputable. [...] Matthew DiFerrante [ZK Labs founder] was associated with the Ethereum Foundation, he had worked as a security engineer there.

Now, what I didn’t know is that behind the scenes, all the other security auditors in the space didn’t really like Matthew. They felt like he was very unprofessional and not doing a good job. [...] He seems like a smart guy, I guess, but it seemed that he had a lot of difficulty dealing with the workload.

We got our protocol audited by them, and it was pretty clear that there’s actually only Matthew DiFerrante doing the auditing. He charged us about $50,000, which for us — a completely bootstrapped company — was like a huge, huge sum of money.

But we tried our hardest to raise funds and do what we could — and we did. We raised fifty thousand for this audit, but it felt like we were somehow being jerked around. [...] We had our stuff ready for him around the beginning of March, but it was closer to September that it was actually done — and only after a lot of teeth pulling and yelling.

When we looked at the audit, we found these typos — there was a place where there was Chainlink’s name instead of ours. He didn’t replace the names. And we were like, “How long did you spend auditing this? Did you really audit this or did we get scammed by ZK Labs?”

That was kind of the question in our minds. He made some suggestions that were helpful, he noticed there was a critical bug. It’s not like he didn’t do anything at all, but we came away not being at all convinced by the audit.

Kistner further added that other security companies like OpenZeppelin or Trail of Bits would have cost the company about $200,000, “And we did not have that [money].”

Are code audits overrated?

BZX’s third hack came right after two major audits by Certik and PeckShield, which seem to have let a subtle bug pass through their nets. Platforms like Aave and Compound also suffered from at-launch vulnerabilities, he said, despite the fact that they were audited extensively.

CT: Do you still believe that audits add value?

KK: Audits are great. If you look at Compound, Aave or others, there are quite a few serious vulnerabilities that were found as a result of the audits. If they didn’t go through them, there’d just be that many more vulnerabilities.

You can’t expect two or three audits to find every single bug. People need to understand that. That’s what the bug bounties are for — when you have the code publicly audited, there are just so many more eyes.

The silver lining to these experiences

Following the initial incidents, bZX overhauled the company and its security practices. Its total value locked rebounded after September, reaching more than $20 million. While this is a far cry from some of the larger protocols, the figure is still notable given the project’s tumultuous year and lack of direct subsidies for putting assets in the protocol.

Kistner said that the team “probably parlayed the [negative] publicity into better recognition and more usage of the protocol overall.” The time has also allowed them to find “something that people really like,” he added. The team is focusing on a long-term perspective, and its twist on yield farming includes a vesting period, which is seen as a mechanism that discourages short-term capital from joining.

At the same time, Kistner believes that the experience allowed bZX to avoid becoming a venture-led project. “We see ourselves as more of a maverick, more of an outsider type of protocol.”

When asked about the investments that the company has received since, he said that “it was a very small round” and that they “didn’t give up any equity or control.”

In the end, the jury is still out on whether bZX can catch up on lost ground. The hacks dealt crippling blows that could have easily resulted in the death of the project, but the team persevered and is bouncing back. The bZX story, however it evolves, remains an important warning for other projects and DeFi users: There is a lot more that goes on in creating a safe product beyond just paying money to auditors.