Thus far in our Neutrino series we’ve taken a deep dive into the details of what Neutrino is, why it’s great for users, how filters work, and how Neutrino uses them. At Suredbits, we have been implementing a Neutrino node in our own open-source library bitcoin-s. In this post we will discuss how that process has gone and what new things we’ve learned about Neutrino.
*Many thanks to Suredbits Software Engineer Roman Taranchenko who was the lead person on our Neutrino implementation.*
What Is Neutrino And Why We Need It
We began by implementing a simple version of BIP 158 block filters where matching was done by simply deserializing the entire filter and then doing a binary search on its contents. After implementing BIP 157 block filter headers, our implementation was good enough to pass the BIP 158 test vectors.
At the time that we were implementing our Neutrino node, bitcoin-core had not yet published a branch or binary with support for BIP 157 P2P messages enabled. They were done with implementation however, and so we modified and built our own modified binary for bitcoin-core that allowed Neutrino peering. We began by doing the simplest thing and downloading block filter headers and then block filters sequentially from this Bitcoin node and writing to a database. This worked, but took around 4 hours to complete.
The major bottleneck for syncing filter headers turned out to be disk IO (meaning writing to the database too often). So we implemented batching database writes for every 2000 block filter headers and this brought our header sync time from 1-2 hours down to 10-20 minutes on mainnet! It also brought our 15+ minute testnet header sync down to 1 or 2 minutes. Unfortunately, batching hardly made a dent in the block filter download time. It only improved things by 5% and at a huge cost of overhead and complexity. So we left block filter download to being sequential. In the future, when we have implemented support for multiple peers, downloading block filters should get much faster.
At this point we added many tests including property based tests for our data structures as well as integration tests that peered against our modified bitcoin-core node on a local regtest network.
With most of the P2P work completed, we moved on to integrating a simple Neutrino node with a wallet (backend) so that it could be useful and interface with users.
The first step was to add filter matching to the block filter download process, as well as downloading full blocks over the P2P network when a match was found. With this done, our very simple wallet could now track addresses on the blockchain.
We then implemented wallet rescans to support importing keys from elsewhere into the wallet. Neutrino is meant to support really fast blockchain rescans after all. But our initial implementation took an hour and a half to rescan mainnet (on a 2.6 GHz Intel Core i7 6 core CPU). The major bottleneck this time was CPU usage, so we made rescans a multi-threaded process and this gave us a 6x improvement (as expected on a 6 core machine) bringing rescans – for 50+ new keys – down to 15 minutes on mainnet (and even faster when doing partial rescans).
To further improve rescans speeds – we returned to the BIP 158 matching code and remade it more in line with the BIP. Instead of deserializing the whole filter, we now matched all addresses as we went and stopped when we were done, rather than decoding the entire filter and searching within it. This doubled mainnet rescans of a single new address down to 7 minutes but had little to no tangible effect when rescanning for 10 or more new addresses. This makes sense because when searching for only a couple new keys within a filter, it is very likely that we can stop early since max_hash(key1, key2) is likely to be much less than max_hash(all keys in a block) very often. However when looking to match a large set of keys, the difference is much less and you essentially end up decoding every filter almost entirely anyway.
Last, we finished wallet integration so that on updates, balances get updated and filters get stored. That leads to the minimal viable Neutrino node we have at the time of this post. The Bitcoin-S Neutrino node currently supports fast initial block header and block filter syncs, fast wallet rescans for importing new keys as well as sending and receiving funds. We also support Bech32 addresses.
Along with optimizing and generally improving our Neutrino node, the main piece of technical work still ahead of us is support for peering with multiple bitcoin nodes as specified in BIP 157. We plan on adding this support to improve privacy and security, as well as download speeds, in the near future. We will also work on improving the user interface to the node which is currently very minimal and focused on technical users.
You can try our node on Testnet now:
Sample Configuration:
To sync our Neutrino node:
Example wallet rescan:
This completes our series on Neutrino. We hope that this has been informative and helped people get excited about Bitcoin’s new light client solution, as well as given insight on how Neutrino works, why we need it and what it takes to develop a node. With bitcoin-core recently adding full Neutrino support in the most recent release (v0.19.0.1), we look forward to more Neutrino implementations entering the community.
Get in touch with the team on Twitter:
All of our API services, for both Cryptocurrency APIs as well as Sports APIs, are built using Lightning technology and the Lightning Network. All API services are live on Bitcoin’s mainnet. Our fully customizable data service allows customers to stream as much or as little data as they wish and pay using bitcoin.
You can connect to our Lightning node at the url:
038bdb5538a4e415c42f8fb09750729752c1a1800d321f4bb056a9f582569fbf8e@ln.suredbits.com
To learn more about how our Lightning APIs work please visit our API documentation or checkout our Websocket Playground to start exploring fully customized data feeds.
If you are a company or cryptocurrency exchange interested in learning more about how Lightning can help grow your business, contact us at [email protected].