[virt-tools-list] virt-install and cloud-init, feedback wanted

[Richard W.M. Jones]

On Mon, Nov 25, 2019 at 11:23:14AM +0000, Daniel P. Berrangé wrote:
> On Mon, Nov 25, 2019 at 11:13:00AM +0000, Richard W.M. Jones wrote:
> > On Fri, Nov 22, 2019 at 09:56:17AM +0000, Daniel P. Berrangé wrote:
> > > On Thu, Nov 21, 2019 at 09:39:46PM -0500, Dusty Mabe wrote:
> > > > 
> > > > 
> > > > On 11/21/19 6:20 AM, Daniel P. Berrangé wrote:
> > > > > On Thu, Nov 21, 2019 at 11:07:24AM +0000, Richard W.M. Jones wrote:
> > > > >> On Thu, Nov 21, 2019 at 10:34:14AM +0000, Daniel P. Berrangé wrote:
> > > > >>> On Wed, Nov 20, 2019 at 08:18:01PM -0500, Dusty Mabe wrote:
> > > > >>>> Basically in Fedora CoreOS we need a generic user data mechanism that works across
> > > > >>>> platforms (x86_64, aarch64, ppc64le, s390x) and doesn't have possible race conditions.
> > > > >>>> Right now we're using `-fw_cfg` but it's not cross platform. We don't have an answer
> > > > >>>> yet: https://github.com/coreos/ignition/issues/666#issuecomment-452835654
> > > > >>>
> > > > >>> For platform portability you need to find some device that is common
> > > > >>> across all platforms, and either disk or network are the only two
> > > > >>> good options that exist today or for the forseeable future.  If those
> > > > >>> aren't acceptable then all we have left are platform specific options.
> > > > >>
> > > > >> While it's not a "good option that exists today", AF_VSOCK would be a
> > > > >> good choice to settle on in the future.  It's completely cross
> > > > >> platform, available for Windows, and doesn't interfere with existing
> > > > >> network or disk devices.
> > > > >>
> > > > >> Would needing virtio be a barrier?  Our impl of AF_VSOCK runs over
> > > > >> virtio, but there are other transports.
> > > > > 
> > > > > From a cloud-init POV, I don't see virtio as a barrier. Defining an
> > > > > AF_VSOCK data source for it should be quite straightforward really
> > > > > and they already have so many data sources, it seems reasonable
> > > > > that they'd accept one more.
> > > > > 
> > > > > On the host side there's still the task of providing a metdata
> > > > > service to expose the data, which is outside scope of virt-install.
> > > > > 
> > > > 
> > > > Thanks for the fruitful conversation here regarding a cross platform data source
> > > > that we could use. Is this worth us writing up into a request for an issue tracker
> > > > somewhere where it could be discussed further?
> > > 
> > > Possibly - depends if AF_VSOCK is viable or not for Ignition.
> > > 
> > > Previously when we suggested use of a plain disk for Ignition, the issue
> > > raised was that the /dev/sdXXX device nodes take a non-deterministic
> > > amount of time to appear, and ignition doesn't know how long to wait
> > > for them, because the disks might not exist at all in the first place.
> > > 
> > > Using AF_VSOCK is going to have exactly this same problem, so personally
> > > I can't see it can satisfy Ignition, where virtio-blk failed.
> > 
> > Be interesting what exactly their concerns were.  Obviously both PCI
> > initialization and (especially) kernel disk enumeration can take a
> > long time with guests that have a lot of PCI slots or disks.  But
> > that's what udev is for, right :-)  Couldn't they just hook up their
> > initialization to a udev rule?
> 
> I mention it above - they don't know whether the user has even
> provided the metadata in the first place. They don't want to wait
> for a device that doesn't exist or they'll be waiting forever. If
> you wait with a timeout, then it delays boot, and creates failure
> conditions under high load.

Maybe kernel command line option indicating "yes there really is a
metadata socket/disk, please wait for it"?  Are these Ignition images
built specially for this purpose?

Rich.

> > > The main benefit I see of AF_VSOCK over virtio-blk disk / cdrom, is that
> > > it is a live data channel so you are not restricted to providing data that
> > > is fixed at boot time, it can be live updated on the fly.
> > > 
> > > The only mechanism that can avoid the problem of waiting for some
> > > device driver to initialize asynchonrously is to use a directly
> > > mapped memory region that is immediately exposed by the core kenrel
> > > functionality. This is what fw_cfg, SMBIOS, and/or kernel command
> > > line args all achieve, and the other options not based on mapped
> > > memory will all fail to achieve.
> > > 
> > > Regards,
> > > Daniel
> > 
> > I added Stefan to CC because he might be interested in this use case too.
> > 
> > Rich.
> > 
> > -- 
> > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > Fedora Windows cross-compiler. Compile Windows programs, test, and
> > build Windows installers. Over 100 libraries supported.
> > http://fedoraproject.org/wiki/MinGW
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v