ANNOUNCE: libnbd 1.8 and nbdkit 1.26 - high performance network block device client and server

[Richard W.M. Jones]

I'm pleased to announce the release of nbdkit 1.26, a high performance
plugin-based Network Block Device (NBD) server, and libnbd 1.8, a high
performance client library.

https://en.wikipedia.org/wiki/Network_block_device

You can get these from:

tarballs: https://download.libguestfs.org/nbdkit/
          https://download.libguestfs.org/libnbd/
git:      https://gitlab.com/nbdkit

Key features of nbdkit:

 * Multithreaded NBD server written in C with good performance.
 * Minimal dependencies for the basic server.
 * Liberal license (BSD) allows nbdkit to be linked to proprietary
   libraries or included in proprietary code.
 * Well-documented, simple plugin API with a stable ABI guarantee.
   Lets you export “unconventional” block devices easily.
 * You can write plugins in C, Go, Lua, Perl, Python, OCaml, Ruby,
   Rust, shell script or Tcl.
 * Filters can be stacked in front of plugins to transform the output.

Key features of libnbd:

 * Synchronous API for ease of use.
 * Asynchronous API for writing non-blocking, multithreaded clients.
   You can mix both APIs freely.
 * High performance.
 * Minimal dependencies for the basic library.
 * Well-documented, stable API.
 * Bindings in several programming languages.
 * Shell (nbdsh) for command line and scripting.
 * Copying tool (nbdcopy) for high performance copying and streaming.
 * Query tool (nbdinfo) to query NBD servers.
 * FUSE support (nbdfuse) to mount NBD in the local filesystem.

Release notes for nbdkit 1.26

       nbdkit is now hosted on gitlab:
       https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html

   Security
       There were no security issues found.  All past security issues and
       information about how to report new ones can be found in
       nbdkit-security(1).

   Plugins
       nbdkit-gzip-plugin and nbdkit-tar-plugin have been removed.  They were
       deprecated in nbdkit 1.22.  The functionality has been replaced by
       nbdkit-gzip-filter(1) and nbdkit-tar-filter(1) respectively.

       nbdkit-streaming-plugin(1) has been deprecated and will be removed in
       nbdkit 1.30.  You can use nbdcopy(1) instead.

       nbdkit-curl-plugin(1) adds these features: "followlocation=false" to
       disable following redirects.  "cookiefile" and "cookiejar" to control
       saving cookies.

       nbdkit-ssh-plugin(1) now advertises multi-conn if the server is
       OpenSSH, improving performance (Eric Blake).

       nbdkit-ondemand-plugin(1) has a new "wait=true" flag allowing you to
       choose between clients waiting for an export to become free or being
       immediately rejected.

       nbdkit-sparse-random-plugin(1) now does not generate fully random data
       within each block.  Using perf we observed that the plugin was spending
       34% of the total time during benchmarks just doing this.  You can
       enable the old behaviour using "random-content=true".

       nbdkit-file-plugin(1) now checks if the file is read-only and exports a
       read-only NBD connection in this case (Eric Blake).

       nbdkit-vddk-plugin(1) has been tested with VDDK 7.0.2.

   Filters
       There is a new API for use by filters to manage "contexts".  Instead of
       the previous 1-1 relationship between filter connections and plugin
       connections, filters can now open more than one context into the
       backend from a single connection, or share contexts between connections
       (Eric Blake).

       New nbdkit-multi-conn-filter(1) which allows you to add multi-conn
       support to plugins which don't support it (although the emulation is
       expensive), or change whether multi-conn is advertised to clients (Eric
       Blake).

       nbdkit-cow-filter(1) now ignores flush/FUA requests (which are not
       useful for a temporary overlay).  The filter now implements extents and
       efficient trimming.  Fine-grained locking is used around shared
       structures, greatly improving parallel performance.  Added support for
       multi-conn.

       nbdkit-cache-filter(1) advertises multi-conn when possible (Eric
       Blake).

       nbdkit-cache-filter and nbdkit-cow-filter can now handle arbitrary
       sized disks, instead of rounding the disk size down.

       nbdkit-blocksize-filter(1) supports parallel requests (Eric Blake).

       nbdkit-checkwrite-filter(1) supports multi-conn (Eric Blake).

       nbdkit-ext2-filter(1) no longer supports multi-conn.  The filter is
       single-threaded so it does not matter, but it was not safe.  Fix trim,
       zero and flush support (Eric Blake).

       nbdkit-readahead-filter(1) fixes multi-conn support (Eric Blake).

       nbdkit-gzip-filter(1) and nbdkit-xz-filter(1) now both support multi-
       conn (Eric Blake).

   Language bindings
       nbdkit-python-plugin(3) supports the following new features:
       "after_fork" and "preconnect" methods.  "nbdkit.shutdown()" binding.

       nbdkit-rust-plugin(3) updated for newer Rust compiler versions (Alan
       Somers).

       nbdkit-golang-plugin(3) now uses and requires golang modules.  Golang ≥
       1.13 must be used.

   API
       New ".cleanup" callback for plugins and filters.  This is called
       shortly before ".unload", but unlike ".unload" it is guaranteed that no
       plugin or filter code has been dlclosed (Eric Blake).

       New "nbdkit_extents_full" is a helper function for filters which can be
       used to read all extents from a plugin covering a region.

   Tests
       Old plugins that we use for testing were rebuilt so they no longer
       depend on libselinux (thanks Thomas Weißschuh).

       nbdkit-ssh-plugin(1) can still be built even if ssh-keygen is
       unavailable for testing it (thanks Thomas Weißschuh).

       Improve tests/test-captive.sh to avoid crashes during testing.

       Test coverage on FreeBSD has been improved, fewer tests should skip
       unnecessarily.

   Build
       nbdkit and plugins can now be built with "-fvisibility=hidden", which
       allows the compiler to make some optimizations.

       Add a dependency so nbdkit is relinked if the linker script
       (nbdkit.syms) changes (Eric Blake).

       xorriso is now preferred over genisoimage or mkisofs (see
       https://wiki.debian.org/genisoimage).

       It is now possible to compile nbdkit with clang with warnings enabled.
       Several warnings have been suppressed or fixed.

       "MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the
       tests, not when running nbdkit from the build directory.  This makes it
       easier to do benchmarks.

       "nbdkit_peer_*" implemented on OpenBSD.

   Internals
       When fuzzing nbdkit we now recommend using the "-t 1" flag (which
       disables threads).  This improves the fuzz-stability of nbdkit.

AUTHORS
       Authors of nbdkit 1.26:

       •   Alan Somers

       •   Eric Blake

       •   Michael Ablassmeier

       •   Richard W.M. Jones

       •   Timm Bäder


Release notes for libnbd 1.8

       libnbd is now hosted on gitlab:
       https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html

   Security
       If you find a security issue, please read SECURITY in the source
       (online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY).
       To find out about previous security issues in libnbd, see
       libnbd-security(3).

       There was one security bug found in libnbd during this release cycle.

       CVE-2021-20286 denial of service when using nbd_set_opt_mode(3)

       See the full announcement here:
       https://listman.redhat.com/archives/libguestfs/2021-March/msg00092.html
       (Found and fixed by Eric Blake).

   New APIs
       nbd_get_private_data(3)
       nbd_set_private_data(3)
           These calls allow you to store either an unsigned integer or a
           pointer in the handle for the application to use for its own
           purposes.

       nbd_get_uri(3)
           This call constructs an NBD URI which can be used to connect back
           to the same server (using nbd_connect_uri(3) or from other tools
           that support the NBD URI standard).

   Enhancements to existing APIs
       Numeric IPv6 URIs (like "nbd://[::1]/") are now supported.

       libnbd gives a better error message if the server backlog overflows
       (thanks Xin Long, Lukas Doktor, Eric Blake, Martin Kletzander).

   New features
       Continuous integration (CI) tests now run on every commit and merge
       request to the upstream repository (Martin Kletzander).

   Tools
       A great deal of work has been done to improve the performance of
       nbdcopy(1) (Nir Soffer, Eric Blake).

       On Linux, nbdcopy now uses readahead when reading from files, and takes
       steps to preserve the host page cache.

       nbdcopy new --request-size option to allow controlling the size of
       requests made to NBD servers.  Adjusting this can affect performance
       (Nir Soffer).

       nbdcopy now supports "null:" pseudo-target, where it discards the
       output.  This is useful for benchmarking.

       nbdcopy new -v flag to enable libnbd and other debugging.

       nbdinfo(1) --list (which lists all exports of a server) no longer exits
       early if one of the exports is inaccessible or there is a recoverable
       error (Eric Blake).

       nbdinfo now prints the URI of each export in the output making it
       easier to connect back to each export.

       nbdfuse(1) is now multithreaded and supports issuing parallel NBD
       commands on a single connection as well as multi-conn, and other
       enhancements have been made.  This resulted in around 50% performance
       improvement when tested with fio.

       nbdfuse now supports nbdcopy-style "[ CMD ]", eg:
        nbdfuse dir/ramdisk [ nbdkit memory 1G ]

       nbdfuse now supports trimming.  Also it supports fast zeroing, but this
       requires a forthcoming kernel patch before it will work.

       nbdfuse sets the virtual file permissions appropriately if the NBD
       export is read-only.

       nbdfuse new -v flag to enable libnbd and other debugging.

   Tests
       Added a test of interoperability with qemu-storage-daemon(1).  Note
       that qemu-storage-daemon was broken in qemu 6.0.0 so this test can fail
       unless you update to a fixed version (thanks Stefan Hajnoczi, Daniel
       Berrangé).

       Added a test of nbdcopy with TLS which was previously untested.

       Python code style tests are now opt-in using
        ./configure --enable-python-code-style

       More tests should now run on FreeBSD and not be skipped.

       nbdfuse tests now ensure that errors from NBD are transmitted through
       FUSE back to the local client.

       "MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the
       tests, not when running libnbd programs from the build directory.  This
       makes it easier to do benchmarks.

   Other improvements and bug fixes
       The minimum C compiler is now ISO C99 (previously ANSI C90).

       FUSE 3 (instead of FUSE 2) is now required to build nbdfuse.

       The library tries harder not to leak thread-local storage memory on
       dlclose or exit.  (Because of the design of Linux it is not always
       possible to avoid this, especially in multithreaded programs.)

       Fix the progress bar in nbdcopy(1).

       UTF-8 export names are now processed and displayed properly by
       nbdinfo(1).

       The --help output of nbdcopy, nbdfuse and nbdinfo now displays a brief
       summary of all options and has some examples.

       The generator now only requires ocamlc, not ocamlfind (Abhay Raj
       Singh).

       Golang bindings now use and require golang modules.

   Documentation
       Document using libnbd with the Meson build system.

       Added Python examples.

       Add example of how to integrate libnbd with libev (Nir Soffer).

AUTHORS
       Eric Blake

       Richard W.M. Jones

       Martin Kletzander

       Nir Soffer



-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW