Every time I think of bitcoin and the Lightning Network, I keep security and privacy in mind. Nobody wants to lose money, right? When building infrastructure, I keep the same mindset. What happens when you build infrastructure for bitcoin? Well…

Let’s talk from a security and reliability standpoint: how can we build a strong and secure Linux box to run LND? This is critical for a Lightning node because it is software that manages a hot wallet. Also, this is relatively new software, so it is less battle tested than others.

In this post, we analyze possible edge cases and security breaches on services. We also share some specific insights on LND secret management.

Starting with a solid and secure base

Nowadays, instance hardening is a must for whatever you are building. Your software could be great at security, but your entire ecosystem is only as secure as its weakest link.

Let’s be super clear on this: it's mandatory to have all the security updates enabled and to keep the Linux kernel and Lightning software updated.

Let's consider a Linux instance. Why is this important? Mainly because you run the software that boots up and loads the kernel, with all the tools you need to ship your delightful application. All this software must be protected against malicious bugs such as 0-day attacks, privilege escalation, and so on.

In the case of LND, a dangerous exploit was discovered a few months ago, and the ability to smoothly upgrade our nodes was critical:

A good practice to keep the software updated is to watch the LND repository for releases, so for each release, we can dig into the changelog and apply it on our nodes.
All in all, I always recommend the K.I.S.S. (“keep it simple, stupid”) principle: install just the software you need, configure it properly, and upgrade it periodically.

You also need to consider which parts you will expose to the Internet: is the service safe enough to be exposed to all the Internet? Is this an attack vector? Do you need a firewall to protect it?

Some software packages use network control endpoints, such as RPC endpoints, to control the software and do operations over it. Basically, every control point you need to use may go over that network control endpoint.

At least once in the life of any developer, you will be tempted to grant all permissions to an app or just forward all the ports on the router to the app. Nothing is wrong with it; you’re just fully exposing a control endpoint to the whole Internet, where the bad guys are more than ready to start attacking using a wide variety of known tools and strategies!

So, is this service robust enough to be exposed to all the Internet? Maybe not at all. Is this an attack vector? Sure it is! Do you need a firewall? Yes!

Let’s look at LND’s case in particular. Under the hood, LND exposes three major ports:

  • 9735: the port from which nodes are reachable. There’s no way around this one; it must be exposed to all the Internet to allow other nodes to connect to you and open channels.
  • 10009: the management port (the RPC port). This port SHOULDN’T be exposed to the Internet, because it is the way you control your node, create invoices, open channels, connect to peers, and so on. Ideally, this port should listen only on localhost, where you run lncli (the command line interface software to talk with LND). You could, though, expose it to a private network subnet where your management server is. Just be sure to keep an eye on the network neighbors.
  • 8080: If enabled, this is the REST API listen port. Like the previous one, DON’T expose this to the Internet. It is fair enough to expose it to your internal network or subnet.

A note on listening on your private network: your private network now turns into your weak link! If you share your network connection with someone else, take care of access management. We may explore this in future posts.

What happens when you leave the default SSH connection port exposed to the Internet? Yup. Thousands of bots hitting your SSH server bruteforcing usernames and passwords every second. The same happens with almost every service you expose on its default port, given that automated attacks are incredibly fast these days. If you are handling money, better safe than sorry!

When your Linux box is strong enough to serve a shiny Lightning node, your next concern is securing the secret values. LND strongly relies on secret files, passwords, and certificates, and we must take care of them.

Storing your secrets

LND handles several secrets on each instance. These are a wallet password, keys, certificates, macaroon files for gRPC authentication, and a recovery seed.
What do you do with them? Shall we write it on plain paper and keep it in a secret box under your bed?

Well, working with LND has some challenges worth mentioning:

A software update requires restart, and restarting a node requires the wallet password. Losing your password makes things difficult: you will not be able to start your node again.

Recovering your funds from a broken LND instance requires your 24-word seed. If you cannot remember your seed, you're in big trouble. Of course, you won’t be in as much trouble as you would be if it's stolen: an attacker will drain your wallet in a matter of minutes if this happens.

Last, the node interaction (opening a channel, listing peers, making a payment, or generating an invoice) requires macaroon files.

So, is plain paper storage a good idea? Certainly not, at least for management files. If someone steals your macaroon files, he will get control over your running node with full capabilities to drain it.

Things could get even more complex. What if you want to run several nodes at a time? In this case, secret organization becomes crucial, and considering a custom piece of software to handle them might be a good idea.

If you want to cut down on human error, a simple piece of software that interacts with the node and a secret storage system (a key: value system) are enough. However, you must take into account the secret rotation feature: a must when doing updates or node migrations, because certificates will change if the node's IP or hostname changes.

AWS Secrets Manager is a good example: with a good trade-off between cost, implementation difficulty, scalability, and security, it became the source of truth for LND Secrets. Another solution, Hashicorp Vault, is recommended as well.

This choice requires serious consideration because it’s not easy to fulfill the requirements for a secret management strategy. Here are some key aspects you need to consider:

  • Scalability: consider that you might end up running multiple environments, networks, and nodes. For example, two environments (development, production) could have two networks (testnet, mainnet) each. Even worse, if you happen to be running more than one node, you will need to consider all combinations for each one of them.
  • Security: choose secret and secure storage that only you and your nodes can access. Make sure there is enough granularity, and pay attention to how you distribute the ability to grant access by the right node/network/environment combination. This is very important: we don't want a development/testnet node to have access to a production/mainnet node credentials. If something fails in development, we may expose production to a big security breach!
  • Flexibility: take into consideration the fact that secrets such as tls.cert and tls.key need to be replaced every time your node’s domain or IP address changes (The key-cert pair is attached to a defined IP/hostname). Therefore, make sure you have the ability to update secrets regularly. A well-tested key management software is a game changer when working with LND.

Conclusion

Bringing up a Lightning node could be a great Lightning experience and an opportunity to learn about security, software scalability, and secret management. It will also help you think about keeping a clean and simple way to deploy and upgrade your nodes. Because LND is in constant development, it’s important to be up to date with the software and its requirements to build a fast, reliable infrastructure ready to join the Lightning network.

Visit Muun Website: https://muun.com/

Follow Muun on Twitter: https://twitter.com/MuunWallet