For reasons of vanity as well as for those of industry health, it’s important to have a good method for ranking crypto exchanges and coins. But as the past few years have shown, no foolproof method has yet been developed. CoinMarketCap has enjoyed the de facto status of being the leading ranking platform, but according to some, a string of controversies has shaken faith in its ability to provide neutral results.

CMC’s first flirtation with scandal happened when Bitwise published a report suggesting 95% of the volume reported on the website was fake. The revelation rocked the sector, as many business leaders, investors and commentators alike saw the data as evidence of adoption and a growing number of use cases.

CMC has been in the spotlight almost continuously since April of this year when the data provider was purchased by Binance for an unspecified amount rumored to be in the hundreds of millions. Almost immediately after the takeover, CMC changed its ranking system, and its new owner shot to the top spot. The action drew fierce criticism from competing exchanges as well as from industry leaders.

CMC changes again

Binance and CMC both claim that they function as independent entities following the takeover. But in a now-infamous tweet, Binance CEO Changpeng Zhao appeared to imply that he had managerial control over CMC when weighing in on the Twitter debate about its ranking of exchanges by web traffic. With some questioning the extent to which the two companies are actually independent of each other, CMC has been continually fine-tuning its ranking methodology.

Perhaps alluding to shaken trust in its own services over the past few months, CMC’s interim CEO, Carylyne Chan, explained the platform’s new “confidence score.” Chan told Cointelegraph that the idea behind the new confidence indicator is to not rely on volume as the sole data source, rather using a range of factors to ascertain data accuracy from each exchange:

“With these new changes, CoinMarketCap currently reports the liquidity of all market pairs using our Liquidity Score, and estimates the number of traders on the exchange using our proprietary Web Traffic Factor. Taking these factors into account, together with time and sales, we constructed a machine learning model to estimate volumes of every single market pair that exchanges report.”

CMC then uses the estimated volumes to detect “outliers,” where its machine learning model can spot which exchanges are reporting exponential volumes relative to its predictions.

Under the new algorithm, the top exchange is awarded a score of 1,000 points. Other exchanges are then given scores against the performance of the top exchange.

While this may be a big change from the previous system that reportedly relied heavily on web traffic figures, there hasn’t been a major reshuffle at the top of the rankings: Binance is still in first place. Chan said that the company’s decision to implement the changes was not influenced by its new owner:

“No, this new update has been in the works all along. As we mention in our blog post on the updates to our ranking algorithm, we are adopting an iterative approach to changes, and the Web Traffic Factor ranking was just one of those steps."

While the confidence factor could be viewed as an improvement, it has not been met with universal acclaim. One Twitter user going by the name of Cosmonaut criticized CMC for placing BitMEX, a popular derivatives exchange, in 175th place. Deribit and Bybit, two other well-known derivatives platforms, were ranked closely at 179 and 177, respectively, at the time of reporting.

Puzzlingly, BitMEX was found to have a near-perfect web traffic score of 960 but a liquidity score of zero. In fact, 175th place appears to be a cliff-edge for exchanges that find themself at the precarious end of CMC’s rankings, as after this all exchanges have liquidity scores of zero. A CMC representative offered an explanation that the ranking only takes spot exchanges into account, with the derivatives platforms soon to be included.

But this does not account for another of Cosmonaut’s hawkish observations. The Twitter user pointed out that several exchanges in the top 50, such as CoinDXC and Huobi Russia, also have liquidity scores of zero.

Related: Crypto Exchanges Speak Out as Binance Takes CoinMarketCap’s Top Spot

Messari issues its own ranking

Binance’s sudden rise to the top of CMC’s exchange listing caused other exchanges to say that it was a clear conflict of interest, with few accepting the changes as coincidence. For some, CMC was fast gaining pariah status. But on May 22, new research from crypto data and research platform Messari added a further 10 exchanges to its “real volume” metric, initially defined by the landmark Bitwise report, as part of a methodology update.

While the premise of the overhaul was to establish the rank of exchanges without including any wash trading — a process in which a trader buys and sells a security with the intention of providing misleading information to the market — the change also confirmed Binance as the leading exchange. The new methodology ranks trading volume using 10 third-party statistics:

“Exchange rankings, ratings, and liquidity estimates from CoinGecko, CoinMarketCap, CryptoCompare, CryptoWatch (Kraken), Nomics, and FTX’s volume monitor, plus onchain transaction data from Chainalysis and other on-chain data providers, and anecdotal data from top traders.”

CMC’s Chan told Cointelegraph that while the data aggregator does not rule out using third-party statistics in the future, it is not planning to do so in the immediate term. While the CMC CEO stopped short of saying that Messari’s ranking of Binance as the top exchange was a vindication of its much-criticized methodology, Chan told Cointelegraph that it stands firm on its algorithms:

“We stand behind our methodology and machine learning models in providing accurate data, and will continue to evaluate our data on our own principled evaluations, using statistical data and analysis. At the same time, we are looking to constantly iterate on the algorithms, to adapt to changes in expectations and market conditions.”

Rival rankers weigh in

One of the most detailed criticisms of CMC’s previous web traffic ranking system came from Huobi Group’s Ciara Sun. Among Sun’s grievances with what she deemed a “highly flawed” methodology was the fact that search engine optimization, location and language need to be taken into account in order to create a balanced measurement.

CryptoCompare CEO Charles Hayter also echoed Sun’s call for a comprehensive analysis of exchanges. Hayter told Cointelegraph that there is still room for improvement with CMC’s methodology and pointed to CryptoCompare’s own twice-yearly analysis. As pointed out on the company’s website: “The ranking components include: geography; legal/regulatory; investment; team/company; data provision; trade surveillance; market quality and a penalty factor for negative events.”

Bobby Ong, the co-founder and chief operating officer of CoinGecko, told Cointelegraph that while CMC’s previous ranking of Binance as the top exchange based on web traffic alone was likely inaccurate, Binance also tops CoinGecko’s ranks:

“Binance is ranked as the top exchange by several parties using various ranking methodology and this includes CoinGecko and Messari. Messari started their exchange rank initially using Bitwise ‘Real 10’ volume and recently updated it by adding more exchanges.”

Ong outlined his view to Cointelegraph that web traffic is a metric that is “easily manipulated and makes a lot of simplification,” making it not suitable for ranking exchanges, as it is easy to provide misleading information and is based on what he sees as three fixed assumptions:

“That is a) the percentage of users trading on the website vs mobile app is the same for all exchanges, b) the percentage of users trading on the website vs API is the same for all exchanges, and c) the estimate for global web traffic including web traffic from China behind the great firewall is similarly accurate. Web traffic should be used as a guiding factor, not the sole measure in ranking exchanges.”

Ong commended CMC for making the latest improvements to its ranking algorithm but said that it is almost a carbon copy of CoinGecko’s “Trust Score” ranking: “For those who are not aware of our Trust Score, we first rolled it out in May 2019 by looking at multiple factors for exchanges.”

Transparency

For some, the anonymity of cryptocurrency may make it seem like a cloak-and-dagger industry. But for many investors, cryptocurrency’s decentralization and open-source nature place a premium on community trust and transparency. While this is often hard to see on crypto Twitter, it appears to be even murkier when looking at how highly influential exchanges collate and publish their data.

CryptoCompare’s Hayter said that exchanges can be more open in setting industry standards for market surveillance as well as Know Your Customer and Anti-Money Laundering protocals to bring transparency regading management and internal procedures. Ong took a more cynical view, arguing that the industry has its fair share of cheaters and that “any metric that can be manipulated, will be manipulated,” adding:

“We tell all exchanges that the best way to grow their exchange is not by cheating their way but by doing the real work of building a superior product that users want to use. Because of all the cheating, we have to stay one step ahead of the cheaters. This is a cat-and-mouse game and we have to constantly improve our Trust Score algorithm as exchanges become more aware of our ranking factors and try to manipulate them.”

Is it time to unite? 

Many people enjoy the narrative that cryptocurrency is more inherently based on cooperation than mainstream financial sectors. A cursory glance through the headlines on any given day does not do much to bolster confidence in this idea, but is an alliance between exchanges or ranking platforms to solve the issue of varying or inaccurate data out of the realms of possibility?

While experts see the benefits of the idea, most are skeptical about the possibility of such an arrangement ever being made reality. Ong told Cointelegraph that one difficulty would be the variety of ways in which exchanges and ranking platforms currently gather data: “An alliance will be tough to achieve because various ranking platforms and exchanges place different factors as important.”

For Hayter, such an alliance could lead to monopolistic trust issues as well as cause clashes with incompatible regulations in different territories around the world. Nonetheless, Hayter added that this has happened in mainstream finance, citing the World Federation of Exchanges and the Federation of European Securities Exchanges in the European Union. Hayter added that the prohibitive costs and regulation involved could prove problematic for smaller players:

“The problem is the barriers to entry that this entails for new entrants — in traditional markets the capital requirements to set up an exchange are huge due to the bureaucratic, regulatory and procedural drag. A balance has to be taken to ensure a rent seeking monopoly doesn’t form in the long run at the expense of consumers.”