Author: sjors

Decoding a BIP-70 Payment Request

I was trying to understand BIP-70 Payment Requests a bit better, mainly because I am confused by BitPay’s claim that they can somehow block “mistaken” transactions:

We can also analyze transactions to make sure an adequate bitcoin miner fee is included. If the fee isn’t sufficient to allow the transaction to confirm on the bitcoin network on time, BitPay can return a helpful message back to the wallet to let the user know. Mistaken payments will never reach the Bitcoin network.

The wording suggests that a wallet sends the transaction to BitPay for approval and they forward it, but afaik that’s not what the BIP-70 does. According to the specification wallets broadcast the transaction via the P2P network like any other transaction. Of course BitPay can always choose to not honor a transaction they receive, but I don’t see how BIP-70 changes that.

In fact, it would be quite unsafe if it did work that way. Other than reputation, what’s to prevent BitPay from rejecting a transaction, telling the customer to submit a new one and then broadcasting both to the network? The wallet would have to be very careful to prevent this; it would have to reuse at least one input in each attempt. Or it could propose unsigned transactions. Or, combined with Replace-By-Fee, perhaps users could send BitPay a series of overlapping transactions with escalating fees, and they would then broadcast the higher fee ones if the confirmation deadline is approached. However that’s totally beyond what BIP-70 is about.

Perhaps they meant that BIP-70 makes it less likely for a user to pay the wrong fee? However the specification doesn’t have a field for a (suggested) fee. Neither does the simpler and more commonly used BIP-21.

Even if there was an ad-hoc method to suggest a fee, at least Bitcoin Core doesn’t honor that. Maybe other wallets do?

Time to look under the hood.

Whenever I get confused, I prefer to just look at what actual software does, rather than speculate based on what people write on blogs or even what a spec says.

BitPay has a demo page where you can generate a payment request. You can use software like QR Journal on MacOS to see what’s in the QR code:

The URI starts with bitcoin: which is defined in BIP-21. Both browsers and mobile apps use URIs like this to determine which application to open. In the case of bitcoin: that’s usually whichever Bitcoin wallet you installed last. This is similar to how mailto:someone@example.com?subject=Hello opens your mail app (even if it’s web based) and creates a draft email with the right address and subject.

A typical BIP-21 URI would contain the destination and amount, e.g. bitcoin:3AcqBykYEos8EHREr7oEzSxg7DxxjH6DCf?amount=0.0001

In this case, the first and only argument is r= which is defined in BIP-72 as an extention of BIP-21 to indicate a URL where further details can be fetched. This saves space compared to putting all the details directly in the QR code, although I wonder if the more QR friendly bech32 could mitigate that.

If you open that URL in a browser, it will show you the invoice page. But a wallet will pass a special HTTP header to tell the server it wants the actual payment request:

wget --header="Accept: application/bitcoin-paymentrequest" https://bitpay.com/i/8JcqpMf87GJWpma6mRCPGD

The result is a protocol buffer; you can tell Mike Hearn, one of the BIP-70 authors, worked at Google before. 🙂 There’s various ways to decode those, though none are properly documented in the BIP.

The first step is to download the Payment Request protocol buffer description file. The paymentrequest.proto file linked to in the BIP doesn’t specify the protocol buffer version, so I’m using paymentrequest.proto from Bitcoin Core.

You can recognise the above high level structure with a simple command (I replaced long binary stuff with…):

protoc --decode payments.PaymentRequest paymentrequest.proto < 8JcqpMf87GJWpma6mRCPGD

payment_details_version: 1
pki_type: "x509+sha256"
pki_data: "\n\263\0160\202\007/...\007Georgia1\0200\016\006\003U\004\007\023\007Atlanta1\0250\023\006\003U\004\n\023\014BitPay, Inc.1\0230\....\026+http://certificates.godaddy.com/repository/0\007\006\206!http://crl.godaddy.com/gdroot.crl0F\006\003U\035 \004?0=0;\006\004U\..."
serialized_payment_details: "\n\004main\022\037\010\...*NPayment request for BitPay invoice 8JcqpMf87GJWpma6mRCPGD for merchant BitGive2+https://bitpay.com/i/8JcqpMf87GJWpma6mRCPGD:L{\"invoiceId\":\"8JcqpMf87GJWpma6mRCPGD\",\"merchantId\":\"TxZ5RyChmZw2isKjJWGhBc\"}"
signature: "\\\231\225\216..."

The interesting bit is serialized_payment_details, which isn’t decoded here. It’s defined as:

So let’s use a better tool. First we convert paymentrequest.proto to something Python understands:

pip install protobuf
protoc --python_out=. paymentrequest.proto

We can then load the invoice and inspect it:

Notice that indeed no fee is specified. However they do increase the amount to offset fees they need to pay to sweep it. This raises some other questions such as why they don’t just give you the the address that they utlimately want to forward it to (and would they have the same policy, and so forth, meaning fees should be infinite?). Also, what time horizon do they have in mind for that sweep? If they’re not in a hurry, or wait for signature aggregation techniques to become available, those fees might be much lower. Anyway, this has no bearing on BIP-70.

I think the confusion arises around the Payment message:

This is sent by the wallet to the merchant before the wallet broadcast the payment via the P2P protocol, but there’s nothing in the spec that says the wallet needs to wait for approval (and see above for why this would be risky with signed transactions).

Note that in the description these steps are reversed, but it doesn’t really matter:

BitPay can inspect the Payment message and refuse to send a PaymentACK, but that’s too late. That said, perhaps it’s not actually implemented that way in some wallets; I haven’t checked. For that I’d need to figure out how to intercept the message Bitcoin Core sends (or study the code). Maybe I’ll update this post later.

Sidechains — assuming they work — peg the exchange rate. Similar to Lightning, it locks up X Bitcoin on the main chain and allows redeeming that X bitcoin. Who gets to redeem what part of X depends on the rules of the sidechain. As long as those rules don’t have bugs, sidechain coins should retain the same value as mainchain coins (maybe slightly less because it takes time to redeem and you never really, really know there’s no bug, or maybe more because it’s more convenient, or because there’s overhead cost as you suggest).

That said, there’s probably plenty of use cases where altoins are fine, especially if you don’t hold them over long periods of time.

On Fees

When sending less than €1,000 of Bitcoin it’s worth paying attention to fees, but keep mind that your payment is competing with transactions that move €100,000 on equal terms. Transactions are charged per byte, not as a percentage of the amount. But willingness to pay is obviously a percentage of the amount.

It’s interesting to note that although the price has increased more than 10x over the past year, transaction amounts in BTC terms haven’t changed much:

Neither have fees as a percentage, they still hover around 0.75%:

Why is this? Perhaps it’s because many hodlers became rich and got comfortable moving 10x the amounts around that they were used to. Or perhaps higher value activity entered the ecosystem, pushing lower value out, or users became more efficient.

The use of SegWit addresses allows for 50% cost savings, yet its adoption is stalling around 12%. This is likely because major services haven’t upgraded yet, and that may be partially due to them being distracted by massive user growth due to the price rise. But then, why aren’t people switching over to competitors that already offer SegWit support?

Is the market for wallets so inefficient that users don’t churn in order to save 50%? It’s understandable that free wallets don’t have advertisement budget to get this message accross, but not all wallets are free. If we were to take Erik Voorhees’ $40 fees claim at face value (which you shouldn’t), then Ledger could advertise their wallet by saying you can earn it back after using it thrice.

In that case, the following affiliate link should make me rich:

Why aren’t they sold out? Why aren’t there $25 e-courses on saving transaction fees? Why isn’t Youtube full of advice to save on fees, targeted at non-technical folks? Most, but not all, of the content around this topic serves to promote investment offers, an obviously very lucrative market. But at present 500 BTC per day levels, fees are becoming a $2.5 billion market. You’d expect that to attract some entrepeneurs as well. Where are they?

Why are people not flocking to coins with low fees, despite good wallet support and descent liquidity? BCash has about 29,000 transactions per day, Litecoin 77,000 transactions vs Bitcoin’s 220,000. With no mechnism to prevent spamming, because those coins are not at full capacity, those numbers are probably optimistic. Pareto’s principe suggests that there should be far more low value transactions than high value transactions, so if there really was huge demand for low-fee currencies, one would expect these altcoins to have far more, not fewer, transactions.

There’s much room for improved efficiency to take better advantage of the existing block space, but a more pessistimic interpretation of what’s going on is that the majority of existing Bitcoin users don’t care about fees. And if that’s true those fees won’t go down no matter what we try in terms of better coin selection, SegWit, Replace-By-Fee, transaction batching, etc. The people moving these large amounts of money won’t bother using these techniques, thus freeing up space for others, unless perhaps we make it trivial, default and beg or guilt-trip them into using it.

Which brings me to Drivechain, which offers an interesting advantage in this situation that Lightning doesn’t have (yet?). In order to use Lightning you need to open a channel first and you need to close it later. In addition to that, you first need to actually receive those coins from somewhere else. Drivechain on the other hand, is more similar to altcoins in the sense that any user can just generate an address and buy the sidechain-coin directly, very much like how they buy bitcoin. It decouples the usage of layer 2 from the task of moving between layer 1 and 2. That allows economies of scale for moving between layers, i.e. far fewer transactions. That is, if I understand Drivechain correctly; it’s hard to wrap my head around. This interview is a good place to start:

Of course these technologies don’t contradict each other and Lighthning is much closer to actually being deployed. It also benefits high value transactors due to it’s orders of magniture improvement on speed of settlement, giving them a stronger incentive to move off-chain than SegWit did. It’s also possible to unilatterally fund a channel, so exchanges can do for every new customer, allowing them toreceive their first coins directly on a Lightning channel (someone still needs to pay fees though, even if the channel is never closed in practice).

Debugging Bitcoin Core Functional Tests

I was trying to improve the functional tests for bumpfee, a Bitcoin Core wallet feature that lets you increase the fee of a transaction that’s unconfirmed and stuck. Unfortunately I introduced a bug in the test, which I’m still in the process of tracking down. Every disadvantage has its advantage, so I took the opportunity to better understand the functional test framework and its powerful debugging tools.

Thanks to everyone who pointed me in the right direction on IRC (as well as for tips on how to use IRC without going insane).

You can view the changes I made, including the bug, in this pull request. Caveat: this is a PR to my own fork: don’t make pull requests like this to github.com/bitcoin/bitcoin. It’s generally a bad idea to change so many things at the same time, if only because it’s too much burden for code reviewers.

To reproduce the error yourself (assuming OSX or Linux, see here for Windows build instructions at the time this post was written):

The test will start running and you’ll see a log message “Running tests”.

Log statements

There’s not much information between the “Running tests” and “Assertion failed” log messages. To see more, switch the log level from the default INFO to DEBUG:

./bumpfee.py --loglevel=DEBUG

As I was investigating, I added several self.log.debug statements to the test file, to get a better sense of what was going on before the error. Now my log looked something like this:

I used self.log.debug(rbftx)to print information about the RBF transaction that the test generated, self.log.debug(rbf_node.getrawmempool()) to show that the transaction made it into the mempool of the node that created it.

self.log.debug(peer_node.getrawmempool()) shows that it didn’t propagate to the other node. At least not immedidately, which makes sense: synchronisation of both mempools is not expected to happen until the next statement sync_mempools((rbf_node, peer_node)). This test helper function waits for both mempols to be identical and fails otherwise.

A good way to learn more about what’s going on is to intentionally break things. For example if I modify the spend_one_input() helper function to pay a 1 BTC miner fee, the test predicatably fails in a different way:

Print RPC commands

The functional tests work by sending commands to the test nodes via RPC. You can log these commands and their responses using ./bumpfee.py --tracerpc. Now you can clearly see sync_mempools in action:

sync_mempools works by sending both nodes the getrawmempool command and comparing the result. After a while it gives up and throws an error.

We still don’t know why the mempools aren’t synchronizing. So let’s dig deeper.

View node log files

So far we’ve been looking at the test logs. But there’s more: each test node has its own directory which includes a log file that you can inspect.

Note the path given after “Initializing test directory”, in this example /var/…/test4ivt2s9y. The logs for the node that created the test transaction are in /var/…/test4ivt2s9y/node1/regtest/debug.log.

You can change the level of detail in these blogs by adding "-debug=all" to self.extra_args in set_test_params(). More fine-grained log options can be found via ../../src/bitcoind --help:

-debug=<category>
  Output debugging information (default: 0, supplying <category> is
  optional). If <category> is not supplied or if <category> = 1,
  output all debugging information. <category> can be: net, tor,
  mempool, http, bench, zmq, db, rpc, estimatefee, addrman,
  selectcoins, reindex, cmpctblock, rand, prune, proxy, mempoolrej,
  libevent, coindb, qt, leveldb.

You can even combine the log files of all test nodes, in order to get a chronological picture of what happened:

test/functional/combine_logs.py /var/.../test4ivt2s9y > combined.log

I published my combined logs here:

It’s clear that node1 sent the transaction to node0 and there’s no obvious error message. One interesting observation is that node1 broadcast the transaction twice, even though the test was only run once.

View other node artifacts

Both node directories contain a file mempool.dat. Although you need other tools to really inspect their contents, in my case it was trivial to see that this file was empty for node0 and not empty for node1, consistent with the--tracerpc output above.

Use Python debugger

So we still don’t know what went wrong. Perhaps through manually interacting with the test nodes we can find out more. One way to do that is to use bumpfee.py --pdbonfailure.

This gives you a Python console, where you ask things like:self.nodes[0].getrawmempool() and notice it’s still empty.

Let’s try a manual broadcast:

Aha, that’s interesting! After this manual broadcast, we find that our transaction finally made it to node0. This still doesn’t solve our mystery, but at least provides another clue.

Inspect running nodes

If things get really desperate, you can leave the nodes running after the test, using--noshutdown. That way you can poke at it using some other tool.

Use serendipity

No, that’s not a tool, it’s a proccess. Just go do other stuff. Eventually you might run into a solution. It turned out the test nodes thought they were still in IBD (Initial Blockchain Download), during which process they don’t synchronize their mempools. To tell the test nodes IBD is over, you need to mine an additional block using peer_node.generate(1). So I broke the tests by removing peer_node.generate(110).

More tests welcome

There’s still plenty of tests to write and improve in Bitcoin Core. Some integration tests, like the one in this article, are written in Python. Those could be a good place to start, until you’re a bit more familiar with the codebase. Please follow recommended practices. There are also C++ integration tests, as well as unit tests.

A Short History of Replay Protection

This article is based on the slides I used for a presentation at the Hong Kong Bitcoin Developer meetup on November 1st, plus some feedback I received on the chainspl.it Slack. This was before SegWit2x was called off, but in the interest of (my) time, I haven’t adjusted this article to reflect that. I’m sure something similar will happen again anyway and it’s a good mental exercise to think through what could have happened.

Investor TL&DR

For non-technical readers a useful perspective — even if technically not accurate — is to distinguish between airdrops and contentious hard forks. This assumes you are in possession of your private keys, as you should.

Airdrop:

• “free” coins based on BTC balance at date X
• safe to ignore, risky to use

Free money?! Bitcoin Cash, Bitcoin Gold, etc.

• 1 BTC on Aug 1 means 1 BCH
• same private key controls both
• distrust “official” wallets; assume malware. Better safe than sorry. Sooner or later one of these airdrops coins will contain malware. Even without malware, simple incompetence of developers can lead to loss of your bitcoin. Most Bitcoin developers have better things to do than inspect this code. They will write gloating articles explaining what went wrong after you lost your Bitcoin. Wait for well established wallets to support; but they can make mistakes too. Remember Cryptsy.
• move BTC to fresh wallet first (just in case)
• privacy (traces on two blockchains)

It’s safe to ignore due to replay protection, risky to use due the above concerns.

Contentious Hard Fork:

• disagreement on what Bitcoin is
• not safe to ignore, unless you HODL

SegWit2x might have gotten messy:

• 1 BTC on Nov ~15 -> 1 BT1 + 1 BT2
• some companies claim BT1 is Bitcoin
• other companies claim BT2 is Bitcoin
• several companies will go back and forth
• no or little replay protection
• never assume companies know what they’re doing

It’s not safe to ignore due to lack of replay protection, unless you don’t use it (HODL). It’s risky to use due the above concerns, though unlike airdrops at least the official wallets are unlikely to contain malware.

Remember The DAO?

• Code is Law!
• $60M ETH stolen from smart contract
• Most developers, holders and miners agreed on need to fork
• Soft-fork wasn’t possible (halting problem)
• Deadline for hard fork was not self imposed

Ethereum Classic is born

Not everyone agreed with this hard-fork. Initially many people didn’t think ETC had a chance to survive, as the theory up to then was that majority hash power would simply crush a minority chain into obvlivion.

The First Replay Attacks

Don’t assume companies in this space know what they’re doing under all circumstances.

Manual replay protection — Split contract

Various split contracts were proposed and their drawbacks were discussed in great detail, probably after people lost money.

Manual replay protection — 6 Easy Steps

Ingredients:

• ETH wallet + ETC wallet
• 1 teaspoon pure ETH
• two block explorers

Procedure:
 1. send ETH balance, including the teaspoon to one address (can’t be replayed, because the ETC balance is insufficient)
 2. send ETC balance to another address

This can still go wrong if an attacker sends you a teaspoon on the other chain quickly enough.

Automatic — EIP 155

A few months of this later…

• another hard fork (Spurious Dragon, Nov 2016)
• opt-in, but wallets use by default (ecosystem is more kumbaya than Bitcoin, so these upgrades can be rolled out quicker)
• same address format, so people can still accidentally send ETH to an ETC address, etc.
• Each hard fork needs to decide if they want to add replay protection, so this requires guessing if it’s going to be contentious.

Bitcoin Cash

• initially opt-in
• last minute change to mandatory
• same address format
• SIGHASH_FORKID and BIP43

BIP143: new signature algorithm

• BIP143 is part of SegWit
• covers value of the input being spent
• solves quadratic hashing
• must be combined with a SegWit transaction
• BCH uses BIP143 without SegWit, causing BCH tx to invalid on BTC chain (they also get some SegWit benefits, while still being able to denounce this technology in their investment pitch)

Note: this only works in one direction, which is where the next section comes in.

SIGHASH_FORKID

Mastering Bitcoin has a chapter on what SIGHASH_TYPE is about.

• mandatory for BCH
• valid but non-standard for BTC
• makes BTC transactions invalid on BCH chain

Combined with BIP143: protection both ways

Note that the SigHash field is four bytes when you sign it, but it gets truncated to the last byte when you serialise the signature (same in SegWit).

Ledger hardware signing

Sometimes it’s really hard to figure out what’s going on based on just (lack of) specs and blog posts. So just read the source code! The changes made on the Ledger hardware side seem pretty simple:

The Chrome extension which generates the unsigned transaction is a bit more complicated, but the magic seems to happen here. I think that when it uses BIP143, if there’s no SegWit it assumes it must be Bitcoin Cash.

I find the following diagram somewhat helpful to visualize what’s going on:

Bitcoin Gold

• Replay protection mechanism TDB… (YOLO)
• They did commit to making addresses start with a G (A for SegWit), which is nice.

SegWit2x Constraints

SegWit2x imposed a number of constrainst on any potential replay mechanism. I don’t think these were terribly well though out, but they make some sense.

1. minimal changes to software of participants (most participants are adding non-protocol level replay protection)
2. capture light weight wallets (light weight clients can just inspect block 494784 to prevent this)
3. nice to have; mostly a gesture to Core
4. limited development and review resources
5. (?) avoid hard-fork with BU

1x-only using magic address

• any funds sent to 3Bit1xA4ap… would make the transaction invalid on the 2x chain
• manual, no wallet change needed (though not all wallet support sending to multiple addresses)
• UTXO “spam”
• phishing “tutorials” (Google is generally not in a hurry to remove phishing ads
• BU willing to support it, using IsStandard()

1x-only using OP_RETURN

• OP_RETURN RP=!>1x (PR 134)
• no UTXO spam
• does require wallets to implement it

2x-only — SIGHASH magic

• PR 131, not that this approach is quite different from BCH
• New: `SIGHASH_2X_REPLAY_PROTECT`
• Sets bit 8 in pre-image (BCH used bit 6)
• Bit 8 isn’t appended to signature
• Core node consider signature invalid
• hard-fork relative to BU

The Future — Spoonnet and other proposals

Spoonnet is a series of proposals of things that can be improved in a hard-fork. It also contains a proposal for replay protection, which is somewhat similar to the 2x-only SIGHASH magic above.

• uses nVersion (“A tx is invalid if the highest nVersion byte is not zero, and the network version bit is not set”)
• hardfork network version bit is 0x02000000
• 0x02000000 is added to the nHashType
• leaves serialized `SIGHASH_TYPE` alone

Another proposal is being discussed on the bitcoin developer mailinglist, which also includes an address change.

SegWit2x — Unprotected?

• HODL! Easiest thing to do during fork is to not use Bitcoin for a while, but not everyone has that luxery.
• UTXO mixing
• nLockTime
• >1 MB transaction (actually slightly less than 1 MB)
• Or just use a custodial service 🙁

Custodial wallets and exchanges can take care of the splitting. They can split customer funds in batches, saving money. Unless something goes wrong and they become insolvent.

UTXO Fairy Dust

Update: chainspl.it has thought about these proposal more than I have.

• Ask miner: coinbase tx unique for each side (natural, organic replay protection, but can’t be done until 100 blocks after the fork)
• Service can split using other method
• paid) API with anyone-can-spend UTXO’s?
• Wallet coin selection must include these inputs (they would need some sort of proof-of-replay-protection…)

nLockTime — 4 easy steps

nLockTime: not mined (consensus rule) or relayed (IsStandard() rule) before block N.

H1: block height of 1x chain, H2: block height of 2x chain

1. generate two addresses (A1, A2)
2. check which chain moves faster (e.g. H2 > H1)
3. sign tx to A2 with H1 < nLockTime < H2
4. send to A1 w/o nLockTime (wait until confirmed, try again if needed)

Problems:

This is hard to do manually, but also hard to automate for non-custodial wallets. User needs to come back several times, lots of edge cases to handle in UI.

• wallet must monitor both chains
• need to wait for gap in block height; only works while one side of fork has a big enough lead. Can’t be used immediately after fork
• sweep is bad for privacy
• must wait for step 4, risks:
   * reorg (e.g. intentional wipeouts)
   * fees in BTC terms > balance
• receiving new unsplit funds. When receiving new funds, wallet must reason if those funds are already replay protected, or its coin selection must always include coins that are known to be protected.

We’ll learn all sorts of new problems as people start losing their money.

> 1MB block

Actually the block needs to be smaller than 1 MB (ex. witness), such that it wouldn’t fit into a 1 MB block due to the space needed for the block header and coinbase transaction. It would just fit under the 1,000,000 byte transaction limit on the 2x chain.

It’s non-standard, so requires coordinatation with miner. Expensive, so easier for a service.

Maybe use CoinJoin (if there’s a way to guarantee the tx will be big enough)?

Conclusion

Just dump your 1x / 2x coins here: 3G8ad4bq7omdk7YT8fPQfWHtHcBrZUDRBL

I see, that committee certainly sounds dubious. It’s not clear to me if it’s related to the travel budget, or only to who get to present. Both would be problematic without more disclosure.

Opt-in hard-fork without alternate transaction history?

IETF’s RFC 7282 is an eloquent document which describes important aspects on consensus, and worthwhile if you want a more nuanced interpretation than “widespread agreement and disagreements addressed (even if not acommodated)”.

Measuring Consensus

Once we have a concrete technical proposal, and it seems to have some traction, we need to figure out if we really have consensus before it gets deployed.

Moving from RFC 7282 style technical rough consensus to economical and political (rough) consensus is quite problematic. If you want to stay in the spirit of RFC 7282 then you should only use polls to see if there is any opposition. You then need to actively go out and figure out what people’s concerns are and make sure those are reasonably addressed. You have to go through all that before you accept anything below 100% support.

This seems impossible in many cases, as non-technical objections can go all over the place; you may end up having to refute all of Nitsche to adequately address some convoluted philosophical objection and finally reach rough consensus between all users. It gets even worse if you need to consider potential future users.

The most pragmatic way out of this problem seems to be to make changes opt-in, hence a preference for soft-forks (though not all kinds).

This doesn’t work when it comes to hard forks; you can’t guarantee they won’t be controversial. Once a hard fork is controversial, exchanges start trading it and users will get confused. Replay protection doesn’t solve this problem, because users still need to choose which chain they believe in which is an enormous burden. They might not have agreed to the code changes had they known this outcome.

You can certainly hold off on any hard fork while it’s controversial, but you can’t predict if it suddenly becomes controversial after the point of no return.

The easiest solution would be to never risk a hard fork. One problem with that solution is that you can’t stop others from doing a hard fork and persuading a large economic and hash power majority to join. There’s always someone willing to take more risk. When the scope of this fork is far outside technical rough consensus, perhaps ignoring it and informing users about the risks is the best approach. However when it’s close to rough consensus, pre-empting may be better than ignoring. Thus it may be prudent to have one or more well tested hard-fork candidates ready to go at any moment, even if the preference is to never deploy them.

A second solution, something I think is worth (re)considering, is to kill off the original chain during a hard fork. Perhaps through some sort of merged mining, where the old chain only gets empty blocks or through a soft fork which makes the entire UTXO set unspendable on the original chain. This requires being even more certain about non-technical consensus, which I’ve argued above is near impossible.

We may need to look for a third solution. Something that is opt-in but doesn’t create two alternate transaction histories.

This could use a link to an explanation of what BIP143 hashing is, how Bcash uses it and why it’s bad.

Replay Protection in The Blockchain Wars

First, I’ll let Jimmy Song explain transaction replay attacks:

However what this short video doesn’t cover, is that there are multiple levels of replay protection. I believe a lot of anger and confusion stems from not properly understanding this. So I’ll do my best to explain.

Levels of Replay Protection

I believe there are four levels of protection that can be offered in the event of a hard-fork.

1. Forced replay protection. After the fork, users have coins on both sides. Whenever they make a transaction, they have to choose which side to send it on. When your goal is to create an alt coin, this makes perfect sense. It’s used by Bitcoin Cash.
2. Opt-out replay protection: when a user makes a transaction, by default it is only accepted on one side of the fork. The user can choose to send it on the other side of the fork. This makes sense if you want there to be one chain eventually, but you only want users to switch with explicit consent. This seems to be what the Spoonnet proposals are aiming for (as well as a longer timeline and several useful upgrades). It’s like how the European Union adds new members.
3. Opt-in replay protection: by default a transaction is accepted on both sides of the fork. It is a fork of conquest. It makes sense if you have control over a large portion of liquidity and wallet users, and you want those users on your side of the fork from the get go.
4. No replay protection. This is every man for themselves, e.g. through mixing coinbase transactions, as Jimmy mentions in the video. Like (3) this gives some advantage to the side with the most initial momentum.

There are various technical ways to implement any of these four levels. The debate gets really confused when arguments about technical feasibility are confused with arguments about which level of protection is desirable.

Solutions can be implemented by just the forking side, or just the remaining side, or both. People often have strong opinions on who should do what. This adds even more confusion.

Finally, developers on one side can write code that’s used on the other side. Or the organizers on one side can offer to include code written by the other side, but refuse to write it themselves. This adds even more confusion.

Mechanims for opt-in Replay Protection

I’d like to zoom in on opt-in replay protection here, because this seems the more likely route for the SegWit2x team to take, and preferable to scenario (4) which is what their currently released software will do. I also find it very interesting.

Bitcoin transactions need to be valid in order to be included in a block. There is a subset of valid transactions which is called standard. By default wallets do not produce non-standard transactions and by default nodes and miners don’t include them in blocks. But if they do end up in a block, it’s not a problem, because they are valid. If however you include an invalid transaction in a block, that block will be rejected. It leads to a hard fork if enough miners mine on top of it regardless.

The SegWit soft fork took advantage of standard-ness to make the upgrade process safe. This ingredient is combined with valid-ness to offer replay protection.

When you write software for a hardfork, you can choose to make previously invalid transactions valid. This appears to be an absolutely necessary ingredient. It’s why Jimmy points out Core can’t add proper replay protection without a (rushed) hard fork. So why is it that?

The key to opt-in replay protection is this:

1. by default, use transactions that are standard on the original chain
2. define a transaction type that is valid but non-standard on the original chain. Make this transaction type invalid on the new chain.
3. define a transaction type that is invalid on the original chain, and make this valid on the new chain.

By default wallets craft transactions of type (1). These are broadcast on both chains. It crafts a transaction of type (2) in order to broadcast on the original chain only. It crafts a transaction of type (3) in order to broadcast on the new chain only.

To take advantage of opt-in replay protection does require a wallet upgrade. It also needs to connect to a special node that relays this non-standard transaction and convince miners to accept it (it’s in their interest). Core could make this easier with a new release, following a new SegWit2x release, but it’s not strictly necessary.

Nodes and miners supporting the new chain have to upgrade anyway, so relay won’t be an issue. Wallets supporting the new chain don’t have to upgrade, but they do need to upgrade to support this replay protection.

Wallet providers with a strong opinion on which chain to follow can choose to make this replay protection the default. This makes the behavior of upgraded wallets more predicatable and prevents spam on the other chain as a bonus.

To be clear, the default behavior offers no guarantee to get included on both chains. Depending on fee levels, it might be included on the new chain first and a long time later on the original chain. It could disappear from the mempool for years and suddenly reappear and get confirmed. Coinbase transaction mixing and double spending could mean it never confirms on one side.

Caveats to replay protection:

1. it is critical to rely on invalid-ness, not just on non-standard-ness. An attacker can easily get a non-standard transaction mined, so this would create a false sense of security.
2. currently the only reason SegWit2x is a hardfork is the block size increase. Adding replay protection means breaking another consensus rule, which adds risk. The biggest risk in my view is long term maintainability. Once Core starts working on Spoonnet, it may be become non-trivial to merge these changes if it clashes with their replay mechanism.
3. when dealing with partially validating wallets, there may be scenario’s where the replay protection itself can be used in an attack!
4. it’s possible that the SegWit2x team can’t find a safe and sufficiently easy way to achieve opt-in replay protection. At that point they have to choose between no replay protection, a delay to keep looking, or giving up entirely. Opt-out or forced replay protection seems completely out of their scope.

A Geneva Convention?

If blockchains are war, should we agree on a minimum standard of conduct?

My impression from following the SegWit2x mailinglist and Github discussions, is that they are open to opt-in replay protection (3). They even recognise this as important, but not at all cost. Several Core developers have insisted on either (1) or (2), which the SegWit2x team sees as unacceptable. It’s like a conquering army being asked to leave their tanks at the border.

It’s interesting to remember UASF. They took the “blockchains are war” analogy quite seriously and portrayed themselves as a militia. This parody video illustrates it quite well:

The UASF node software relied on lack of replay protection (even lack of wipeout protection, but that’s a different story) to persuade others to join their effort. I pointed this out in another article at the time.

Greg Maxwell called them out on this:

“First do no harm.” We should use the least disruptive mechanisms available, and the BIP148 proposal does not meet that test. To hear some people — non-developers on reddit and such — a few even see the forced orphaning of 148 as a virtue, that it’s punitive for misbehaving miners. I could not not disagree with that perspective anymore strongly.

Thankfully this game of chicken ended safely.

“do no harm” is probably not realistic in the blockchain wars. But perhaps opt-in replay protection can be agreed on as a miminum standard.

This would come with responsiblity on both sides. The aggresor should use the least disruptive mechanism possible to achieve their goal. They should build in such replay protection if at all possible. However, the defending side should also make a reasonable effort to facilitate this, or at least not actively sabotage it. I don’t see signs of such sabotage thankfully. I’ve seen some accusations, but I believe those are misinterpreted.

Would that be through merge mining? Otherwise, what’s the incentive?