We&; pleased to announce go-libp2p 6.0.0. This release includes a massive refactor of go-libp2p that paves the way for new such as QUIC. Unfortunately, as it is broad sweeping, there are some breaking changes, especially for maintainers of custom transports.
Below, we cover the changes you’ll likely care about. For convenience, we’ve broken this into a section for users and transport authors/maintainers. However, transport maintainers should really read both sections.
💁 For Users
Libp2p users should be aware of a few major changes.
- Guarantees and performance concerning connect/disconnect notification processing have improved.
- Handling of half-closed streams has changed (READ THIS SECTION).
- Some constructors and method signatures have changed slightly.
Dialing And Source Addresses
We’ve improved the logic that selects the source address when dialing. In the past, you may have run into an issue where you couldn’t dial non-local nodes when listening on 127.0.0.1 as opposed to 0.0.0.0. This happened because go-libp2p would randomly pick the source address from the set of addresses on which the node was listening. We did this to ensure that the other end could dial us back at the same address. Unfortunately, one can’t use 127.0.0.1 as a source address when dialing any non-local address so outbound dials failed.
go-libp2p now tries to be smarter about this and avoids picking source addresses that have no route to the destination address.
To start out on an unhappy note, bandwidth metrics are now less accurate. In the past, transports returned “minimal” connections (e.g., a TCP connection) so we could wrap these transport connections in “metrics” connections that counted every byte sent and received.
Unfortunately, now that we’ve moved encryption and multiplexing down into the transport layer, the connection we’re wrapping has significantly more under-the-covers overhead.
However, we do hope to improve this and get even better bandwidth metrics than we did before. See libp2p/go-libp2p-transport#31 for details.
This release brings performance improvements and simple ordering guarantees for connect/disconnect events:
- For any given connection/stream, libp2p will wait for all connect/open event handlers to finish exit before triggering a disconnect/close event for the connection/stream.
- When a user calls the
Reset) method on a connection or stream, go-libp2p will process the close event asynchronously (i.e., not block the call to
Close). Otherwise, a call to
Closefrom within a connect event handler would deadlock.
- Unless otherwise noted, events will be handled in parallel.
What does this mean for end users? Well:
- Reference counting connections to a peer using connect/disconnect events should “just work” and should never go negative.
- Under heavy connect/disconnect loads, connecting to new peers should be faster (usually).
For those interested in the history of this issue, …
In the past, (dis)connect and stream open/close notifications have been a bit of a pain point. For a long time, they were fired off in parallel and one could, for example, process a disconnect notification before a connect notification (we had to support negative ref-counts in several places to account for this).
After no end of trouble, we finally “fixed” this by synchronizing notification delivery. We still delivered notifications to all notifiees in parallel, we just processed the events in series.
Unfortunately, under heavy connect/disconnect load, new connections could easily get stuck on open behind a queue of connect events all being handled in series. In theory, these events should have been handled quickly but in practice, it’s very hard to avoid locks entirely (bitswap’s event handlers were especially problematic).
Worse, this serial delivery guarantee didn’t actually provide us with an in-order delivery guarantee as it was still possible for a disconnect to happen before we even started to fire the connect event. The situation was slightly better than before because the events couldn’t overlap but still far from optimal.
However, this has all been resolved now. From now on, you’ll never receive a disconnect event before a connect event.
Conn.GetStreams Signature Change
The signature of the
GetStreams method on
go-libp2p-net.Conn has changed from:
GetStreams() (Stream, error)
Listing the streams on an open connection should never involve IO or do anything that can fail so we removed this error to improve usability.
If you’re not already doing so, you should be using the
libp2p.New constructor to make your libp2p nodes. This release brings quite a few new options to the libp2p constructor so if it hasn’t been flexible enough for you in the past, I recommend that you try again. A simple example can be found in the echo example.
Given this work and in an attempt to consolidate all of our configuration logic in one place, we’ve removed all default transports from go-libp2p-swarm.
TL;DR: Please use the libp2p constructor.
From this release on, when you’re done with a stream, you must either call
Reset on it (in case of an error) or close it and read an EOF (or some other error). Otherwise, libp2p can’t determine if the stream is actually closed and will hang onto it indefinitely.
To make properly closing streams a bit easier, we’ve added two methods to go-libp2p-net:
AwaitEOF(stream)tries to read a single byte from the stream. If
Readreturns an EOF,
AwaitEOFreturns success. Otherwise, if
Readeither reads some data or returns some other error,
AwaitEOFresets the stream and returns an error. To avoid waiting indefinitely,
AwaitEOFresets the stream unconditionally after 1 minute.
FullClose(stream)is a convenience function that closes the stream and then calls
Like with libp2p notifications, this issue has a bit of history…
In the beginning, libp2p assumed that calling
Close on a stream would close the stream for both reading and writing. Unfortunately, none of our stream multiplexers actually behaved this way. In practice,
Close always closed the stream for writing.
After realizing this, we made two changes:
- We accepted the fact that
Closeonly closed the stream for writing.
- We added a
Resetmethod for killing the stream (closing it in both directions, throwing away any buffered data).
However, we ran into a bit of a snag because we try to track open streams and need some way to tell when a stream has been closed. In the past this was easy: when the user calls
Close on the stream, stop tracking it. However, now that
Close only closes the stream for writing, we still technically needed to track it until the other end closed the stream as well. Unfortunately, without actually reading from the stream, we have no way of knowing about this. Therefore, if the user calls
Close on a stream and then walks away, we’d have to hang onto the stream indefinitely.
Our solution was to simply stop tracking streams once they were closed for writing. This wasn’t the correct behavior but it avoided leaking memory in the common case:
- The user calls
Closeand drops all references to the stream.
- The other end calls
Closewithout writing any additional data.
- The stream multiplexer observes both closes and drops its reference to the stream.
- The garbage collector garbage collects the stream.
However, this meant that:
- The list of “open” streams was technically incomplete.
- If the other side either failed to call
Closeor tried to send data before closing, the stream would remain “open” (until the connection was closed).
In this release, we’ve changed this behavior. Now, when you
Close a stream for writing, libp2p continues to track it. We only stop tracking it when either:
- You call
Reset(throwing away the stream).
- You finish reading any data off of it and observe either an EOF or an error.
This way, we never “forget” about open streams or leave them in a half-forgotten state.
In the future, I’d like to add a
CloseAndForget method to streams that:
- Closes the stream (sends an EOF).
- Tells the swarm to stop tracking the stream.
- Tells the stream muxer to stop tracking the stream and throw away any data the other side may send (possibly resetting the stream on unexpected data).
- This would likely require modifying our stream muxers which may not be feasible.
- Explicitly waiting for an EOF is still the correct thing to do unless you really don’t care if the operation succeeded.
📞 For Transport Maintainers
For transport maintainers, quite a bit has changed. Before this change, transports created simple, unencrypted, stream connections and it was the job of the libp2p Network (go-libp2p-swarm) to negotiate security, multiplexing, etc.
However, when attempting to add support for the QUIC protocol, we realized that this was going to be a problem: QUIC already handles authentication and encryption (using TLS1.3) and multiplexing. After much debate, we inverted our current architecture and made transports responsible for encrypting/multiplexing their connections (before returning them).
To make this palatable, we’ve also introduced a new “upgrader” library for upgrading go-multiaddr-net connections/listeners to full libp2p transport connections/listeners. Transports that don’t support encryption/multiplexing out of the box can expect to have an upgrader passed into the constructor.
To get a feel for how this new transport system works, take a look at the TCP and WebSocket transports and the transport interface documentation:
This release sees the deprecation of a few packages:
- go-peerstream has been deprecated and all functionality has been merged into go-libp2p-swarm. go-peerstream was written as a general-purpose (not libp2p specific) listener, connection, and stream manager. However, this package caused more problems than it solved and was incompatible with the new transport interface.
- go-libp2p-interface-conn has been deprecated. These interfaces to bridge the gap between transport-level connections and go-libp2p-net connections however, now that transport connections are fully multiplexed/encrypted, this is no longer needed.
- go-libp2p-conn has also been deprecated and most of the functionality has been moved to go-libp2p-transport-upgrader. This package used to provide connection “upgrade” logic for upgrading transport-level connections to go-libp2p-interface-conn connections however, transport-level connections now provide the required functionality out of the box.
- The swarm duplicated this exact function for its own tests.
- The swarm couldn’t depend on go-libp2p-netutil because go-libp2p-netutil depends on go-libp2p-swarm.
We’ve also added a new transport test suit go-libp2p-transport/test. If you implement a new transport, please consider testing against these suite. If you find a bug in an existing transport, please consider adding a test to this suite.
In go-addr-util, we’ve removed the
SupportedTransportProtocols transport registries and the associated
AddTransport function. These registries were updated by
init functions in packages providing transports and were used to keep track of known transports.
However, importing a transport doesn’t mean any libp2p nodes have been configured to actually use that transport. Therefore, in the new go-libp2p, it’s go-libp2p-swarm’s job to keep track of which transports are supported (i.e., which transports have been registered with the swarm).
We’ve also removed the associated
Pluggable Security Transports
This release brings a new pluggable security transport framework. Implementing a new security framework is now as simple as:
- Implement the interfaces defined in go-conn-security.
- Pass it into the libp2p constructor using the