[virt-tools-list] virt-install and cloud-init, feedback wanted

Mon Nov 25 11:23:14 UTC 2019

On Mon, Nov 25, 2019 at 11:13:00AM +0000, Richard W.M. Jones wrote:
> On Fri, Nov 22, 2019 at 09:56:17AM +0000, Daniel P. Berrangé wrote:
> > On Thu, Nov 21, 2019 at 09:39:46PM -0500, Dusty Mabe wrote:
> > > 
> > > 
> > > On 11/21/19 6:20 AM, Daniel P. Berrangé wrote:
> > > > On Thu, Nov 21, 2019 at 11:07:24AM +0000, Richard W.M. Jones wrote:
> > > >> On Thu, Nov 21, 2019 at 10:34:14AM +0000, Daniel P. Berrangé wrote:
> > > >>> On Wed, Nov 20, 2019 at 08:18:01PM -0500, Dusty Mabe wrote:
> > > >>>> Basically in Fedora CoreOS we need a generic user data mechanism that works across
> > > >>>> platforms (x86_64, aarch64, ppc64le, s390x) and doesn't have possible race conditions.
> > > >>>> Right now we're using `-fw_cfg` but it's not cross platform. We don't have an answer
> > > >>>> yet: https://github.com/coreos/ignition/issues/666#issuecomment-452835654
> > > >>>
> > > >>> For platform portability you need to find some device that is common
> > > >>> across all platforms, and either disk or network are the only two
> > > >>> good options that exist today or for the forseeable future.  If those
> > > >>> aren't acceptable then all we have left are platform specific options.
> > > >>
> > > >> While it's not a "good option that exists today", AF_VSOCK would be a
> > > >> good choice to settle on in the future.  It's completely cross
> > > >> platform, available for Windows, and doesn't interfere with existing
> > > >> network or disk devices.
> > > >>
> > > >> Would needing virtio be a barrier?  Our impl of AF_VSOCK runs over
> > > >> virtio, but there are other transports.
> > > > 
> > > > From a cloud-init POV, I don't see virtio as a barrier. Defining an
> > > > AF_VSOCK data source for it should be quite straightforward really
> > > > and they already have so many data sources, it seems reasonable
> > > > that they'd accept one more.
> > > > 
> > > > On the host side there's still the task of providing a metdata
> > > > service to expose the data, which is outside scope of virt-install.
> > > > 
> > > 
> > > Thanks for the fruitful conversation here regarding a cross platform data source
> > > that we could use. Is this worth us writing up into a request for an issue tracker
> > > somewhere where it could be discussed further?
> > 
> > Possibly - depends if AF_VSOCK is viable or not for Ignition.
> > 
> > Previously when we suggested use of a plain disk for Ignition, the issue
> > raised was that the /dev/sdXXX device nodes take a non-deterministic
> > amount of time to appear, and ignition doesn't know how long to wait
> > for them, because the disks might not exist at all in the first place.
> > 
> > Using AF_VSOCK is going to have exactly this same problem, so personally
> > I can't see it can satisfy Ignition, where virtio-blk failed.
> 
> Be interesting what exactly their concerns were.  Obviously both PCI
> initialization and (especially) kernel disk enumeration can take a
> long time with guests that have a lot of PCI slots or disks.  But
> that's what udev is for, right :-)  Couldn't they just hook up their
> initialization to a udev rule?

I mention it above - they don't know whether the user has even
provided the metadata in the first place. They don't want to wait
for a device that doesn't exist or they'll be waiting forever. If
you wait with a timeout, then it delays boot, and creates failure
conditions under high load.

> > The main benefit I see of AF_VSOCK over virtio-blk disk / cdrom, is that
> > it is a live data channel so you are not restricted to providing data that
> > is fixed at boot time, it can be live updated on the fly.
> > 
> > The only mechanism that can avoid the problem of waiting for some
> > device driver to initialize asynchonrously is to use a directly
> > mapped memory region that is immediately exposed by the core kenrel
> > functionality. This is what fw_cfg, SMBIOS, and/or kernel command
> > line args all achieve, and the other options not based on mapped
> > memory will all fail to achieve.
> > 
> > Regards,
> > Daniel
> 
> I added Stefan to CC because he might be interested in this use case too.
> 
> Rich.
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|