[virt-tools-list] virt-install and cloud-init, feedback wanted

Thu Nov 21 10:34:14 UTC 2019

On Wed, Nov 20, 2019 at 08:18:01PM -0500, Dusty Mabe wrote:
> 
> 
> On 11/20/19 5:49 PM, Cole Robinson wrote:
> > Hi all. The purpose of this mail is to get some feedback on pending
> > cloud-init support in virt-install. If you're on the CC list here, I
> > either pulled your email from a cloud-init discussion on the the
> > virt-tools-list mailing list, or from the CC list of this virt-install bug:
> > 
> > RFE: Provide cloud-init integration for VMs
> > https://bugzilla.redhat.com/show_bug.cgi?id=981693
> > 
> > For GSOC 2019 Athina Plaskasoviti completed some cloud-init integration
> > work in virt-install. You can read her wrap up here:
> > https://athinapl.home.blog/2019/08/25/gsoc-2019-cloud-init-configuration-for-virt-manager-virt-install/
> 
> First off, thanks Cole and Athina for working on this. This is something I have
> wanted for a long time! In my dayjob I've moved away from cloud-init a bit because
> our new platforms are running Ignition: https://github.com/coreos/ignition
> 
> From the Ignition side I've added Andrew Jeddeloh and Benjamin Gilbert to CC.
> 
> > 
> > Right now the code is sitting in a virt-manager.git branch, not yet
> > pushed to master:
> > https://github.com/virt-manager/virt-manager/tree/cloudinit
> > 
> > I'll summarize the current behavior, and then ask some questions.
> > 
> > 
> > The branch adds a new 'virt-install --cloud-init' argument with several
> > sub options. When specified, virt-install generates an empty meta-data
> > file, a user-data file with the requested changes, stuffs them both into
> > a cidata iso, which is used for the first VM boot and then deleted.
> 
> Is there any chance we could go a different route than an ISO image? The developers
> of ignition have experienced problems with relying on a device here (i.e. is
> there no device coming or has the device just not shown up yet?). Some discussion over
> in: https://github.com/coreos/ignition/issues/656
> 
> Basically in Fedora CoreOS we need a generic user data mechanism that works across
> platforms (x86_64, aarch64, ppc64le, s390x) and doesn't have possible race conditions.
> Right now we're using `-fw_cfg` but it's not cross platform. We don't have an answer
> yet: https://github.com/coreos/ignition/issues/666#issuecomment-452835654

For platform portability you need to find some device that is common
across all platforms, and either disk or network are the only two
good options that exist today or for the forseeable future.  If those
aren't acceptable then all we have left are platform specific options.

I think relying on platform specific options *is* acceptable, provided
we can identify a viable option for each platform.  We have SMBIOS
OEM strings as recommended solution for x86 and aarch64. We are yet
to identify any good options for ppc or s390 though. Perhaps we can
define a mapping for SMBIOS into these arches, as SMBIOS is ultimately
just a spec for a set of tables in a region of memory. It possible,
probably even likely, that we will have to choose something entirely
different though for these arches.

> > This behavior is only triggered when --cloud-init is specified in some
> > form, there's no automagic invocation of this support.
> > 
> > The command sub options are:
> > 
> > $ ./virt-install --cloud-init=help
> > --cloud-init options:
> >   disable
> >   root-password-file
> >   root-password-generate
> >   ssh-key
> >   user-data
> > 
> > Their behavior:
> > 
> > * disable=yes: boolean option to disable cloud-init in the VM for
> > subsequent boots. Adds this block to user-data:
> >   runcmd:
> >   - [ sudo, touch, /etc/cloud/cloud-init.disabled ]
> > 
> > * root-password-file=/MY/PATH: set the desired root password from the
> > content of /MY/PATH on the host
> > 
> > * root-password-generate=yes: boolean option, generate a random root
> > password, set it in user-data, print it to the host text console and
> > pause for a bit for the user to see and copy it. sorta inspired by
> > virt-builder
> > 
> > * ssh-key=/MY/KEY.pub: inject the ssh key from /MY/KEY.pub on the host
> > into the cloud-init user-data
> > 
> > * user-data=/PATH/TO/user-data: ignore all other options and copy this
> > file to the .iso as user-data
> > 
> > When bare '--cloud-init' is specified, we default to generating a random
> > root password and disabling cloud-init for subsequent boots:
> > --cloud-init root-password-generate=yes,disable=yes
> > 
> > We've explicitly rejected something like root-password=MY-PASSWORD
> > because of the security implications of encouraging a password to end up
> > in command line history. We've already had a CVE for something similar
> > in virt-install.
> > 
> > Also, I don't want virt-install to be in the business of specifying
> > every cloud-init option under the sun, there's gotta be a better tool
> > for that already. So I'd like to keep --cloud-init suboptions
> > specifically targeted to expected virt use cases, and anything else can
> > be served with custom user-data=
> 
> I think we should keep it as slim as possible. You've given users enough
> to get in to the machine. If they need to do anything else they can either
> provision via SSH after boot or they can pass a custom user data to do more
> provisioning.
> 
> > 
> > One more point: my main interaction with cloud-init has historically
> > been by grabbing a Fedora/RHEL cloud image, passing it to
> > virt-install/virt-manager, and watching the boot hang, because there's
> > no data provider and cloud-init times out talking to the network, and
> > then I can't log in. I expect many people have hit this issue before.
> > I've always worked around this by using 'virt-customize' to disable
> > cloud-init and reset the root password. That's about the extent of my
> > usage here, which is broadly why the bare `--cloud-init` is the way it was.
> 
> If you need any help understanding things or testing them I can try to help!
> 
> > 
> > I'm also thinking to the future, if one day virt-install can detect that
> > it was passed a distro cloud-init image, perhaps we can invoke some
> > default behavior that gives the user a better chance of this config
> > being usable out of the box. I figure that will match whatever we choose
> > for the bare '--cloud-init' behavior
> > 
> > 
> > So, my questions are:
> > 
> > * What are the usecases you see for virt-install cloud-init support?
> > * Does the above meet your expectations?
> > * Are we missing anything vital?
> 
> The only thing I can think of is *maybe* being able to specify which user you want
> the ssh-key to apply to.
> 
> > * Do you have an opinion of what behavior bare '--cloud-init' should give?
> > * Any other ideas, thoughts, feedback?
> 
> I would love to have something like this support Ignition as well. Do you think
> that would be welcome? 
> 
> If so, does it change your thoughts on the approach? Would we stick with `--cloud-init`
> and just add `--ignition` or would we want to group it under another heading
> `--userdata type=cloud-init,ssh-key=.....` `--userdata type=ignition,ssh-key=...`?

If ignition is not a drop-in replacement for cloud-init, then I think
using the --cloud-init arg would be a bad idea. Instead we would need
a separate --ignition arg, or we could need a more general --userdata
as you sugggest.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|