[virt-tools-list] virt-install and cloud-init, feedback wanted

Mon Nov 25 11:48:15 UTC 2019

On Mon, Nov 25, 2019 at 11:29:38AM +0000, Richard W.M. Jones wrote:
> On Mon, Nov 25, 2019 at 11:23:14AM +0000, Daniel P. Berrangé wrote:
> > On Mon, Nov 25, 2019 at 11:13:00AM +0000, Richard W.M. Jones wrote:
> > > On Fri, Nov 22, 2019 at 09:56:17AM +0000, Daniel P. Berrangé wrote:
> > > > On Thu, Nov 21, 2019 at 09:39:46PM -0500, Dusty Mabe wrote:
> > > > > 
> > > > > 
> > > > > On 11/21/19 6:20 AM, Daniel P. Berrangé wrote:
> > > > > > On Thu, Nov 21, 2019 at 11:07:24AM +0000, Richard W.M. Jones wrote:
> > > > > >> On Thu, Nov 21, 2019 at 10:34:14AM +0000, Daniel P. Berrangé wrote:
> > > > > >>> On Wed, Nov 20, 2019 at 08:18:01PM -0500, Dusty Mabe wrote:
> > > > > >>>> Basically in Fedora CoreOS we need a generic user data mechanism that works across
> > > > > >>>> platforms (x86_64, aarch64, ppc64le, s390x) and doesn't have possible race conditions.
> > > > > >>>> Right now we're using `-fw_cfg` but it's not cross platform. We don't have an answer
> > > > > >>>> yet: https://github.com/coreos/ignition/issues/666#issuecomment-452835654
> > > > > >>>
> > > > > >>> For platform portability you need to find some device that is common
> > > > > >>> across all platforms, and either disk or network are the only two
> > > > > >>> good options that exist today or for the forseeable future.  If those
> > > > > >>> aren't acceptable then all we have left are platform specific options.
> > > > > >>
> > > > > >> While it's not a "good option that exists today", AF_VSOCK would be a
> > > > > >> good choice to settle on in the future.  It's completely cross
> > > > > >> platform, available for Windows, and doesn't interfere with existing
> > > > > >> network or disk devices.
> > > > > >>
> > > > > >> Would needing virtio be a barrier?  Our impl of AF_VSOCK runs over
> > > > > >> virtio, but there are other transports.
> > > > > > 
> > > > > > From a cloud-init POV, I don't see virtio as a barrier. Defining an
> > > > > > AF_VSOCK data source for it should be quite straightforward really
> > > > > > and they already have so many data sources, it seems reasonable
> > > > > > that they'd accept one more.
> > > > > > 
> > > > > > On the host side there's still the task of providing a metdata
> > > > > > service to expose the data, which is outside scope of virt-install.
> > > > > > 
> > > > > 
> > > > > Thanks for the fruitful conversation here regarding a cross platform data source
> > > > > that we could use. Is this worth us writing up into a request for an issue tracker
> > > > > somewhere where it could be discussed further?
> > > > 
> > > > Possibly - depends if AF_VSOCK is viable or not for Ignition.
> > > > 
> > > > Previously when we suggested use of a plain disk for Ignition, the issue
> > > > raised was that the /dev/sdXXX device nodes take a non-deterministic
> > > > amount of time to appear, and ignition doesn't know how long to wait
> > > > for them, because the disks might not exist at all in the first place.
> > > > 
> > > > Using AF_VSOCK is going to have exactly this same problem, so personally
> > > > I can't see it can satisfy Ignition, where virtio-blk failed.
> > > 
> > > Be interesting what exactly their concerns were.  Obviously both PCI
> > > initialization and (especially) kernel disk enumeration can take a
> > > long time with guests that have a lot of PCI slots or disks.  But
> > > that's what udev is for, right :-)  Couldn't they just hook up their
> > > initialization to a udev rule?
> > 
> > I mention it above - they don't know whether the user has even
> > provided the metadata in the first place. They don't want to wait
> > for a device that doesn't exist or they'll be waiting forever. If
> > you wait with a timeout, then it delays boot, and creates failure
> > conditions under high load.
> 
> Maybe kernel command line option indicating "yes there really is a
> metadata socket/disk, please wait for it"?  Are these Ignition images
> built specially for this purpose?

The problem here is that the kernel args are hidden inside the bootloader
config inside the disk image, so you can't augment them from the QEMU
CLI config. This lead to the use of -fw_cfg instead, and my suggestion
to use SMBIOS.

> > > > The main benefit I see of AF_VSOCK over virtio-blk disk / cdrom, is that
> > > > it is a live data channel so you are not restricted to providing data that
> > > > is fixed at boot time, it can be live updated on the fly.
> > > > 
> > > > The only mechanism that can avoid the problem of waiting for some
> > > > device driver to initialize asynchonrously is to use a directly
> > > > mapped memory region that is immediately exposed by the core kenrel
> > > > functionality. This is what fw_cfg, SMBIOS, and/or kernel command
> > > > line args all achieve, and the other options not based on mapped
> > > > memory will all fail to achieve.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|