Debugging Bitcoin Core Functional Tests

Debugging Bitcoin Core Functional Tests

I was trying to improve the functional tests for bumpfee, a Bitcoin Core wallet feature that lets you increase the fee of a transaction that’s unconfirmed and stuck. Unfortunately I introduced a bug in the test, which I’m still in the process of tracking down. Every disadvantage has its advantage, so I took the opportunity to better understand the functional test framework and its powerful debugging tools.

Thanks to everyone who pointed me in the right direction on IRC (as well as for tips on how to use IRC without going insane).

You can view the changes I made, including the bug, in this pull request. Caveat: this is a PR to my own fork: don’t make pull requests like this to It’s generally a bad idea to change so many things at the same time, if only because it’s too much burden for code reviewers.

To reproduce the error yourself (assuming OSX or Linux, see here for Windows build instructions at the time this post was written):

The test will start running and you’ll see a log message “Running tests”.

Log statements

There’s not much information between the “Running tests” and “Assertion failed” log messages. To see more, switch the log level from the default INFO to DEBUG:

./ --loglevel=DEBUG

As I was investigating, I added several self.log.debug statements to the test file, to get a better sense of what was going on before the error. Now my log looked something like this:

I used self.log.debug(rbftx)to print information about the RBF transaction that the test generated, self.log.debug(rbf_node.getrawmempool()) to show that the transaction made it into the mempool of the node that created it.

self.log.debug(peer_node.getrawmempool()) shows that it didn’t propagate to the other node. At least not immedidately, which makes sense: synchronisation of both mempools is not expected to happen until the next statement sync_mempools((rbf_node, peer_node)). This test helper function waits for both mempols to be identical and fails otherwise.

A good way to learn more about what’s going on is to intentionally break things. For example if I modify the spend_one_input() helper function to pay a 1 BTC miner fee, the test predicatably fails in a different way:

Print RPC commands

The functional tests work by sending commands to the test nodes via RPC. You can log these commands and their responses using ./ --tracerpc. Now you can clearly see sync_mempools in action:

sync_mempools works by sending both nodes the getrawmempool command and comparing the result. After a while it gives up and throws an error.

We still don’t know why the mempools aren’t synchronizing. So let’s dig deeper.

View node log files

So far we’ve been looking at the test logs. But there’s more: each test node has its own directory which includes a log file that you can inspect.

Note the path given after “Initializing test directory”, in this example /var/…/test4ivt2s9y. The logs for the node that created the test transaction are in /var/…/test4ivt2s9y/node1/regtest/debug.log.

You can change the level of detail in these blogs by adding "-debug=all" to self.extra_args in set_test_params(). More fine-grained log options can be found via ../../src/bitcoind --help:

Output debugging information (default: 0, supplying <category> is
optional). If <category> is not supplied or if <category> = 1,
output all debugging information. <category> can be: net, tor,
mempool, http, bench, zmq, db, rpc, estimatefee, addrman,
selectcoins, reindex, cmpctblock, rand, prune, proxy, mempoolrej,
libevent, coindb, qt, leveldb.

You can even combine the log files of all test nodes, in order to get a chronological picture of what happened:

test/functional/ /var/.../test4ivt2s9y > combined.log

I published my combined logs here:

It’s clear that node1 sent the transaction to node0 and there’s no obvious error message. One interesting observation is that node1 broadcast the transaction twice, even though the test was only run once.

View other node artifacts

Both node directories contain a file mempool.dat. Although you need other tools to really inspect their contents, in my case it was trivial to see that this file was empty for node0 and not empty for node1, consistent with the--tracerpc output above.

Use Python debugger

So we still don’t know what went wrong. Perhaps through manually interacting with the test nodes we can find out more. One way to do that is to use --pdbonfailure.

This gives you a Python console, where you ask things like:self.nodes[0].getrawmempool() and notice it’s still empty.

Let’s try a manual broadcast:

Aha, that’s interesting! After this manual broadcast, we find that our transaction finally made it to node0. This still doesn’t solve our mystery, but at least provides another clue.

Inspect running nodes

If things get really desperate, you can leave the nodes running after the test, using--noshutdown. That way you can poke at it using some other tool.

Use serendipity

No, that’s not a tool, it’s a proccess. Just go do other stuff. Eventually you might run into a solution. It turned out the test nodes thought they were still in IBD (Initial Blockchain Download), during which process they don’t synchronize their mempools. To tell the test nodes IBD is over, you need to mine an additional block using peer_node.generate(1). So I broke the tests by removing peer_node.generate(110).

More tests welcome

There’s still plenty of tests to write and improve in Bitcoin Core. Some integration tests, like the one in this article, are written in Python. Those could be a good place to start, until you’re a bit more familiar with the codebase. Please follow recommended practices. There are also C++ integration tests, as well as unit tests.


A Short History of Replay Protection

A Short History of Replay Protection

This article is based on the slides I used for a presentation at the Hong Kong Bitcoin Developer meetup on November 1st, plus some feedback I received on the Slack. This was before SegWit2x was called off, but in the interest of (my) time, I haven’t adjusted this article to reflect that. I’m sure something similar will happen again anyway and it’s a good mental exercise to think through what could have happened.

Just click YES

Investor TL&DR

For non-technical readers a useful perspective — even if technically not accurate — is to distinguish between airdrops and contentious hard forks. This assumes you are in possession of your private keys, as you should.


  • “free” coins based on BTC balance at date X
  • safe to ignore, risky to use

Free money?! Bitcoin Cash, Bitcoin Gold, etc.

  • 1 BTC on Aug 1 means 1 BCH
  • same private key controls both
  • distrust “official” wallets; assume malware. Better safe than sorry. Sooner or later one of these airdrops coins will contain malware. Even without malware, simple incompetence of developers can lead to loss of your bitcoin. Most Bitcoin developers have better things to do than inspect this code. They will write gloating articles explaining what went wrong after you lost your Bitcoin. Wait for well established wallets to support; but they can make mistakes too. Remember Cryptsy.
  • move BTC to fresh wallet first (just in case)
  • privacy (traces on two blockchains)

It’s safe to ignore due to replay protection, risky to use due the above concerns.

Contentious Hard Fork:

  • disagreement on what Bitcoin is
  • not safe to ignore, unless you HODL

SegWit2x might have gotten messy:

  • 1 BTC on Nov ~15 -> 1 BT1 + 1 BT2
  • some companies claim BT1 is Bitcoin
  • other companies claim BT2 is Bitcoin
  • several companies will go back and forth
  • no or little replay protection
  • never assume companies know what they’re doing

It’s not safe to ignore due to lack of replay protection, unless you don’t use it (HODL). It’s risky to use due the above concerns, though unlike airdrops at least the official wallets are unlikely to contain malware.

Just click YES

Remember The DAO?

  • Code is Law!
  • $60M ETH stolen from smart contract
  • Most developers, holders and miners agreed on need to fork
  • Soft-fork wasn’t possible (halting problem)
  • Deadline for hard fork was not self imposed
Just click YES

Ethereum Classic is born

Not everyone agreed with this hard-fork. Initially many people didn’t think ETC had a chance to survive, as the theory up to then was that majority hash power would simply crush a minority chain into obvlivion.

Just click YES

The First Replay Attacks

Don’t assume companies in this space know what they’re doing under all circumstances.

Just click YES

Manual replay protection — Split contract

Just click YES

Various split contracts were proposed and their drawbacks were discussed in great detail, probably after people lost money.

Just click YES
Just click YES

Manual replay protection — 6 Easy Steps


  • ETH wallet + ETC wallet
  • 1 teaspoon pure ETH
  • two block explorers

 1. send ETH balance, including the teaspoon to one address (can’t be replayed, because the ETC balance is insufficient)
 2. send ETC balance to another address

Just click YES

This can still go wrong if an attacker sends you a teaspoon on the other chain quickly enough.

Automatic — EIP 155

A few months of this later…

Just click YES
  • another hard fork (Spurious Dragon, Nov 2016)
  • opt-in, but wallets use by default (ecosystem is more kumbaya than Bitcoin, so these upgrades can be rolled out quicker)
  • same address format, so people can still accidentally send ETH to an ETC address, etc.
  • Each hard fork needs to decide if they want to add replay protection, so this requires guessing if it’s going to be contentious.

Bitcoin Cash

  • initially opt-in
  • last minute change to mandatory
  • same address format

BIP143: new signature algorithm

  • BIP143 is part of SegWit
  • covers value of the input being spent
  • solves quadratic hashing
  • must be combined with a SegWit transaction
  • BCH uses BIP143 without SegWit, causing BCH tx to invalid on BTC chain (they also get some SegWit benefits, while still being able to denounce this technology in their investment pitch)

Note: this only works in one direction, which is where the next section comes in.


Mastering Bitcoin has a chapter on what SIGHASH_TYPE is about.

  • mandatory for BCH
  • valid but non-standard for BTC
  • makes BTC transactions invalid on BCH chain

Combined with BIP143: protection both ways

Just click YES

Note that the SigHash field is four bytes when you sign it, but it gets truncated to the last byte when you serialise the signature (same in SegWit).

Ledger hardware signing

Sometimes it’s really hard to figure out what’s going on based on just (lack of) specs and blog posts. So just read the source code! The changes made on the Ledger hardware side seem pretty simple:

Just click YES

The Chrome extension which generates the unsigned transaction is a bit more complicated, but the magic seems to happen here. I think that when it uses BIP143, if there’s no SegWit it assumes it must be Bitcoin Cash.

I find the following diagram somewhat helpful to visualize what’s going on:

Just click YES

Bitcoin Gold

  • Replay protection mechanism TDB… (YOLO)
  • They did commit to making addresses start with a G (A for SegWit), which is nice.

SegWit2x Constraints

SegWit2x imposed a number of constrainst on any potential replay mechanism. I don’t think these were terribly well though out, but they make some sense.

  1. minimal changes to software of participants (most participants are adding non-protocol level replay protection)
  2. capture light weight wallets (light weight clients can just inspect block 494784 to prevent this)
  3. nice to have; mostly a gesture to Core
  4. limited development and review resources
  5. (?) avoid hard-fork with BU
Just click YES

1x-only using magic address

  • any funds sent to 3Bit1xA4ap… would make the transaction invalid on the 2x chain
  • manual, no wallet change needed (though not all wallet support sending to multiple addresses)
  • UTXO “spam”
  • phishing “tutorials” (Google is generally not in a hurry to remove phishing ads
  • BU willing to support it, using IsStandard()
Just click YES

1x-only using OP_RETURN

  • OP_RETURN RP=!>1x (PR 134)
  • no UTXO spam
  • does require wallets to implement it
Just click YES

2x-only — SIGHASH magic

  • PR 131, not that this approach is quite different from BCH
  • Sets bit 8 in pre-image (BCH used bit 6)
  • Bit 8 isn’t appended to signature
  • Core node consider signature invalid
  • hard-fork relative to BU
Just click YES
Just click YES
Just click YES
Just click YES

The Future — Spoonnet and other proposals

Spoonnet is a series of proposals of things that can be improved in a hard-fork. It also contains a proposal for replay protection, which is somewhat similar to the 2x-only SIGHASH magic above.

  • uses nVersion (“A tx is invalid if the highest nVersion byte is not zero, and the network version bit is not set”)
  • hardfork network version bit is 0x02000000
  • 0x02000000 is added to the nHashType
  • leaves serialized `SIGHASH_TYPE` alone

Another proposal is being discussed on the bitcoin developer mailinglist, which also includes an address change.

SegWit2x — Unprotected?

  • HODL! Easiest thing to do during fork is to not use Bitcoin for a while, but not everyone has that luxery.
  • UTXO mixing
  • nLockTime
  • >1 MB transaction (actually slightly less than 1 MB)
  • Or just use a custodial service 🙁

Custodial wallets and exchanges can take care of the splitting. They can split customer funds in batches, saving money. Unless something goes wrong and they become insolvent.

UTXO Fairy Dust

Update: has thought about these proposal more than I have.

  • Ask miner: coinbase tx unique for each side (natural, organic replay protection, but can’t be done until 100 blocks after the fork)
  • Service can split using other method
  • paid) API with anyone-can-spend UTXO’s?
  • Wallet coin selection must include these inputs (they would need some sort of proof-of-replay-protection…)

nLockTime — 4 easy steps

nLockTime: not mined (consensus rule) or relayed (IsStandard() rule) before block N.

H1: block height of 1x chain, H2: block height of 2x chain

  1. generate two addresses (A1, A2)
  2. check which chain moves faster (e.g. H2 > H1)
  3. sign tx to A2 with H1 < nLockTime < H2
  4. send to A1 w/o nLockTime (wait until confirmed, try again if needed)


This is hard to do manually, but also hard to automate for non-custodial wallets. User needs to come back several times, lots of edge cases to handle in UI.

  • wallet must monitor both chains
  • need to wait for gap in block height; only works while one side of fork has a big enough lead. Can’t be used immediately after fork
  • sweep is bad for privacy
  • must wait for step 4, risks:
     * reorg (e.g. intentional wipeouts)
     * fees in BTC terms > balance
  • receiving new unsplit funds. When receiving new funds, wallet must reason if those funds are already replay protected, or its coin selection must always include coins that are known to be protected.

We’ll learn all sorts of new problems as people start losing their money.

> 1MB block

Actually the block needs to be smaller than 1 MB (ex. witness), such that it wouldn’t fit into a 1 MB block due to the space needed for the block header and coinbase transaction. It would just fit under the 1,000,000 byte transaction limit on the 2x chain.

It’s non-standard, so requires coordinatation with miner. Expensive, so easier for a service.

Maybe use CoinJoin (if there’s a way to guarantee the tx will be big enough)?


Just dump your 1x / 2x coins here: 3G8ad4bq7omdk7YT8fPQfWHtHcBrZUDRBL