Failover Strategies for Lightning Nodes
Previously in our Lightning 201 series we discussed failover strategies for a Lightning node. The first thing necessary to safely failover a Lightning node is move the database to a remote machine. This means that if the machine that is running the Lightning node fails, the database won’t fail as well. Now another Lightning node can connect to your remote database and resume operations.
Node backups and failovers leave us with another problem though: What happens if two “shadow” lightning nodes are trying to access the database at the same time? The consequences could be dire. This could create an internally inconsistent state in the Lightning database that could cause you to publish a revoked state on the Lightning network leading to a loss of funds.
This can be solved by having an application level database “LOCK” that only allows one of our shadow nodes to access the database at a single time. The idea is that every time a new “shadow” lightning node is started up. We read from a table in the shared Lightning database to see if any other node currently owns the “lease” to the database. If another node does own the lease to the database, then the shadow node stops attempting to connect to the database. If there is no active lease for the database, the “shadow” can create a row in the table that gives it ownership of the lease.
We have implemented this locking scheme in eclair here.
An important design consideration when implementing this locking scheme is liveness of the system. Currently we ensure liveness by giving each of our leases a timeout. If the current lease owner does not renew their lease after 5 minutes another node can create a new lease.
However, our locking scheme does have a tradeoff. If the current lease owner were to terminate 1 minute into their lease ownership, it will be another 4 minutes until another node can take ownership of the lease for the Lightning database. This means there is potentially 4 minutes of downtime for the Lightning system.
In the worst case there can be 5 minutes of unavailability for our default configuration of leases in Eclair. The lease duration is a configurable parameter that you can adjust for what you think is an appropriate amount of downtime.
*Special thanks to Christian Decker for the insights and conversation around locking schemes.
Get in touch with the team on Twitter:
All of our API services, for both Cryptocurrency APIs as well as Sports APIs, are built using Lightning technology and the Lightning Network. All API services are live on Bitcoin’s mainnet. Our fully customizable data service allows customers to stream as much or as little data as they wish and pay using bitcoin.
You can connect to our Lightning node at the url:
038bdb5538a4e415c42f8fb09750729752c1a1800d321f4bb056a9f582569fbf8e@ln.suredbits.com
To learn more about how our Lightning APIs work please visit our API documentation or checkout our Websocket Playground to start exploring fully customized data feeds.
If you are a company or cryptocurrency exchange interested in learning more about how Lightning can help grow your business, contact us at [email protected].