Historical Bitcoin Core Client Performance

Historical Bitcoin Core Client Performance

After reading about performance optimizations on the Bitcoin Core blog, I wanted to see a chart of that. I couldn’t find one, so I spun up a bunch of EC2 nodes and set out to measure how long it took each new version of the bitcoin core client to sync the full blockchain.

I described the exact steps I took to compile the nodes on Amazon EC2 in a Github gist.

Starting with the blue bars and from the right side of the chart, downloading the full blockchain and verifying all signatures took 28 hours using the upcoming v0.15 release, while v0.14.2, released in early 2017, took 32 hours. Moving left we see similar performance for versions 0.12 and 0.13. Although the older versions appear slightly faster, I suspect that’s either random variation or perhaps a side-effect of my own EC2 nodes connecting to each other. See Caveats below.

There’s a remarkable jump (5x) when going back to 0.11 (released in 2015). This is the last version that used OpenSSL, which was replaced in 0.12 by the much faster libsecp256k1, a library purpose built for Bitcoin.

Going even further back in time we see comparable performance for version 0.9 and 0.8, although it took me considerably more effort to get those up and running.

I managed to compile even older versions, down to 0.5 from late 2011. Unfortunately they either crashed or grinded to a halt when they reached blocks from around mid 2015. I’m not sure what caused the old clients to fail, but it happened with blocks that were created in 2015, around the time transaction count first crossed 150K per day.

In order to compare older clients, I lowered the ambition and made them sync only blocks until the end of 2013; the light blue bars. It took version 0.5 almost three days — with some handholding, version 0.8 a little over three hours and less than an hour for 0.15.

The light red bars demonstrate the Assume Valid Block feature that was introduced in v0.14. This avoids the need to validate signatures before the release date. It took as little as 2 and 20 minutes for v0.15 to download the blockchain up to July 2017. The result is comparable to what the Core team found in their own test using similar equipment.

If you download the client half a year after its release, it will need to verify more signatures. The dark red bars shows this effect, which is just a few hours.

Moore’s Law?

I wanted to adjust the chart for Moore’s Law, in order to get a sense of how long a device from 2013 would have taken to sync todays blockchain, or how much it would have cost on EC2 back then. This turned out to be non-trivial.

First of all I could not find a historical dataset of EC2 pricing. This actually surprised me, because EC2 has become a commodity and normal commodities like oil and gold do have historical price charts. Either nobody cares or Amazon and their competitors are really keen on keeping that information private.

Second, one of the major resource constraints for bitcoin is RAM. Again I couldn’t find an index of that commodity, but from more anecdotal data it seems RAM didn’t get much cheaper in the past four years:

Example RAM price chart showing little decline from 2012 to 2016

I’m sure over the long run prices will continue to decline, but it seems that we should never assume that over shorter time periods. It’s also important to consider where the bottleneck is. E.g. if harddisk space is getting cheap that’s great, but with pruning that perhaps isn’t the bottleneck at the moment. If the price of the current bottlneck (CPU, RAM, storage, or bandwidth, electricity etc) doesn’t decline, it doesn’t matter much if all other costs do decrease.

I think it’s fair to say — assuming RAM is actually the bottleneck — most of the performance gains over the past 4 years have been due to engineering effort, while Moore has been letting us down a bit.

The Challenge: run bitcoind 0.2

I got a little carried away with the experiment and managed to rack up $600 in EC2 bills. Part of that is due to false starts; many older nodes crashed after a week and I had to recompile them. I probably could have used lower-end machines, e.g. I doubt the old versions even used all four CPU cores. But I wanted to keep the comparison fair.

Version 0.2 is from 2010, which is before the 1 MB “soft fork”. Funny enough, that was introduced in a commit called ‘cleanup’ (see Bitcoin Wiki Scalability FAQ for context). In theory this ancient version should follow the 2 MB chain in November if SegWit2x goes to plan. That is, if it manages to sync up before then…

If someone can send me the exact series of commands to run on an EC2 instance (Ubuntu or otherwise), including how to keep the node from crashing, I can do it for less.

Special thanks to the kind folks at bitcoin.stackexchange.com for helping me out with the older clients, especially Pieter Wuille, Nick ODell and Andrew Chow.

Caveats & Roadblocks

• I suspect some nodes were connecting to each other. Because I deployed them in order from new to old, that might explain why 0.13 seems to be slightly faster than 0.14, and 0.12 slightly faster than 0.13. Or that could just be random variance.
• You might get more realistic results by excluding other EC2 nodes, e.g. by avoiding their IP range or refusing to connect to any client with a ping off less than ? ms.
• one could run the same experiment on a more resource constrained device like a Rasberry Pi 3 behind some sort of network throttler. Without pruning this would be difficult or expensive.
• Initial Blockchain Download (IBD) is not the only relevant metric when it comes to scaling issues, but it is a relevant benchmark, because it’s a very resource intense task.
• Pieter Wuille suggested using -connect=IP to prevent older clients from getting confused and even stopping sync. I haven’t tried this, and I’m not sure if it’s “cheating”; I suppose that depends on much trust this option implies in that other node.
• T2 may not be ideal for this measurement, because performance is not constant “Burstable CPU, governed by CPU Credits”
• bitcoind 0.7 froze at block 364,670 (mid 2015) after about nine days; before that it was syncing about 4,000 blocks per day, which means it would have taken a month to catch up. I shared debug.log (558 MB) on Dropbox.
• bitcoind 0.6 ran at about the same pace as 0.7, but crashed at 329,998, I didn’t try a restart. I shared debug.log (400 kb) on Dropbox.
• in July bitcoind 0.5 ran at about the same pace as 0.7, but crashed at 332,150. In September I had to restart it a few times while it went through the December 2013 blocks. It might have run a bit faster if I built an automated script to kill and restart it every time it got stuck.
• I had to patch earlier versions to play nicely with the new post-heartbleed OpenSSL.
• I plan to wipe the nodes one week from now, so if you want me to publish any log files, let me know before then.
• I did not take Laanwj’s performance tips and other potential tweaks into account; taking full advantage of those might reveal a difference between versions that look similar in my current chart.
• v0.15 was tested two months after v0.14 so it’s possible the EC2 t2.xlarge configuration changed.

Update 2017–07–30: here’s the Amazon billing info. Green is for storage (11x 200 GB), which you pay for even if the machines are turned off. At the peak I was running about six nodes in parralel. Pruning was added in v0.11, so modern clients allow you to rent less storage. Something similar should hold true for pruning signature data for SegWit transactions.