Statoshi project tracks Bitcoin node statistics

In March, software engineer and bitcoin blogger Jameson Lopp posed an interesting question on CoinChomp: How many Bitcoin nodes is enough? To his surprise, Lopp found that while it was relatively easy to determine the number of nodes in the Bitcoin system, it was impossible to determine how well those nodes were performing. And even more worryingly, the overall number of full nodes appears to be dropping even as the overall hashing power of the network is exploding.

For an open-source, peer-to-peer system, the lack of network-performance data is a curious oversight. It’s also a recipe for disaster as bitcoin adoption grows an an unprecedented pace. As Lopp explains:

Given the transparency of many aspects of the Bitcoin network’s infrastructure, I’m disappointed that we can’t get more insight into the volume of requests being served by the nodes. I’m quite interested in finding a way to measure the load the network is experiencing in realtime. It seems to me that if we develop a programmatic way of determining this, we could then automatically spool up new nodes on virtualized servers as needed to meet increases in demand. However, as far as I can tell, because any given node is helping the network in a myriad of ways and it’s not possible to query the nodes for more granular statistics, it’s not possible to programmatically determine how busy the nodes are. What are we to do?

Thankfully, Lopp didn’t leave it there. He decided to solve the problem himself.

To address this issue, Lopp and his collaborators have created the Statoshi project, an open-source fork of Bitcoin Core that logs metrics to StatsD. Unlike Bitcoin Core, Statoshi provides real-time data for full node hosts to monitor node-to-node interaction and performance. The system can also aggregate data from multiple connected nodes, and exports visually using open-source data-graphing system Graphite.

Why is this an important development? Full nodes are the backbone of the entire bitcoin system, serving to both validate the blocks and transactions, as well as relaying them to other points across the network. Nodes perform a variety of other functions, such as hosting copies of the entire 19 GB version of the blockchain and providing lookup resources, but few miners or users have the resources, knowledge or motivation to host full versions.

If left unchecked, early warnings of network-threatening performance issues could wreak havoc, slowing transactions to a crawl without any clear indication of the problem. By providing access to new reporting tools, the Statoshi project can prevent many of these while the issues are small and the fixes easily implemented.