At the beginning of this series we discussed Neutrino at a high level and looked at benefits for wallets. In the last post, we started our deep dive into the inner workings of Neutrino clients by detailing how probabilistic filters are constructed as specified by BIP 158. In this post, we will continue our detailed tour by discussing how these filters are used by clients in their operations.
What is Neutrino And Why We Need It
Before diving into the P2P messages that were introduced in BIP 157, we must first talk about another “new” construct, block filter headers. Bitcoin blocks have block headers that are bare-bones, 80-byte piece of meta-data, sufficient to allow clients to ensure that they are tracking a single longest chain. Likewise, block filters have 32-byte headers aid clients in following a single longest block filter chain. Block filter headers are simply a hash of their block filter’s hash concatenated with the previous block filter header:
Block Filter Header n := Hash(Hash(Block Filter n) || Block Filter Header n-1).
Now that we have block filters and block filter headers at our disposal, let’s look at Neutrino’s P2P messages. BIP 157 defines three new request messages: getcfilters, getcfheaders, and getcfcheckpt as well as their corresponding response messages: cfilter, cfheaders, cfcheckpt (note: cf stands for compact filter). Unsurprisingly, getcfilters and getcfheaders are requests for a range of filters and filter headers respectively. And getcfcheckpt is a request for every 1000th block filter header, i.e. “Please send me block filter headers numbered 1000, 2000, 3000, …” up to some specified end. Note that these new messages are available to all nodes on the Bitcoin P2P network, not just Neutrino nodes.
We now have all the tools needed to run a Neutrino node: block filters, block filter headers, and P2P messages to send and receive these over the network.
So how exactly does a Neutrino client operate?
Let’s say that I’m turning on a fresh Neutrino client. It will first download the block header chain and validate Proof of Work. This is the first step taken by all Bitcoin nodes (full, BIP 37, or Neutrino). Neutrino will also download the block filter header chain from its peers. Specifically, the client will ask all peers it wishes to download from for their checkpoint headers (every 1000th header) and ensure that all peers agree. The node can then download different 1000 header intervals from different peers concurrently. Once these two steps are done, the node can begin downloading block filters (from multiple peers) and verify that they match the block filter headers. And of course, with these filters a client is able to see whether a given block mentions outputs that this node is interested in tracking. When a match occurs, the node will download the full block using the Bitcoin P2P network – from a random peer that it is not using for filters.
If at any point during this process, or after, two peers send conflicting block filter headers or block filters, the Neutrino client must interrogate those peers and discover the truth. This is accomplished by finding the first block filter that the two nodes disagree on by doing a binary search on the relevant 1000 block filter header interval. Once this first disagreement is found, the Neutrino client downloads that full block (from a random peer that it is not using for filters) and computes the block filter and block filter header from scratch. It is then able to determine which peer is lying and ban that peer.
In this manner, Neutrino nodes are assured that so long as they have one truthful peer, all fraudulent peers will be discovered and banned!
BIP 157 does not specify exactly how many block filters and block filter headers should be persisted by a node. Rather, every node can decide how much or how little it wants to store. For many nodes (for example, on mobile phones), it may make a lot of sense to only persist the last 100 or so filters. This allows Neutrino wallets to stay extremely light on storage.
On the other hand, if a node has the capability to keep around all or many of its block filters – which is still much less than keeping full blocks around – then it has the ability to do really fast wallet rescans. This is a trade-off that each wallet or wallet user must consider: storage space vs. fast rescans. For most light clients that won’t ever contain any old addresses but rather only generate new ones, it will make a lot of sense to simply throw away old block filters.
As mentioned previously, the new P2P messages defined in BIP 157 are available to all Bitcoin P2P nodes. I expect that in the future most full nodes will support Neutrino server functionality. This is important because it is vital to Neutrino’s privacy model that the nodes serving filters not also be the nodes serving full blocks to client nodes. This is because if a node receives a request for a block filter, then soon after it receives a request for the corresponding full block from the same peer, it can safely infer that the client matched an output mentioned in that block. If this happens a couple different times, or during a rescan, then this can very quickly lead to a privacy breach. The best practice to avoid this is to have a large enough set of block filter servicing peers, a disjoint large enough set of full block servicing peers and to pick from the latter set at random when a full block is required. In the future, when feasible, some nodes may support downloading full blocks using Private Information Retrieval which should essentially “solve” this problem/concern.
And that is all there is to a Neutrino node! They validate block filters by keeping track of block filter headers. They then use those block filters to see when their outputs are referenced in blocks and privately download those full blocks. Lastly, they are able to interrogate and ban any malicious peers so long as any one of their peers is truthful.
In our next and last post in this series, we will tell our own story about the experience of implementing Neutrino in Bitcoin-S and reflecting on what we learned in the process.
All of our API services, for both Cryptocurrency APIs as well as Sports APIs, are built using Lightning technology and the Lightning Network. All API services are live on Bitcoin’s mainnet. Our fully customizable data service allows customers to stream as much or as little data as they wish and pay using bitcoin.
You can connect to our Lightning node at the url:
038bdb5538a4e415c42f8fb09750729752c1a1800d321f4bb056a9f582569fbf8e@ln.suredbits.com
To learn more about how our Lightning APIs work please visit our API documentation or checkout our Websocket Playground to start exploring fully customized data feeds.
If you are a company or cryptocurrency exchange interested in learning more about how Lightning can help grow your business, contact us at [email protected].