[virt-tools-list] virt-install and cloud-init, feedback wanted

Fri Nov 22 16:14:48 UTC 2019

On Thu, Nov 21, 2019 at 6:43 PM Cole Robinson <crobinso at redhat.com> wrote:

> On 11/21/19 5:28 AM, Daniel P. Berrangé wrote:
> > On Wed, Nov 20, 2019 at 05:49:45PM -0500, Cole Robinson wrote:
> >> Hi all. The purpose of this mail is to get some feedback on pending
> >> cloud-init support in virt-install. If you're on the CC list here, I
> >> either pulled your email from a cloud-init discussion on the the
> >> virt-tools-list mailing list, or from the CC list of this virt-install
> bug:
> >>
> >> RFE: Provide cloud-init integration for VMs
> >> https://bugzilla.redhat.com/show_bug.cgi?id=981693
> >>
> >> For GSOC 2019 Athina Plaskasoviti completed some cloud-init integration
> >> work in virt-install. You can read her wrap up here:
> >>
> https://athinapl.home.blog/2019/08/25/gsoc-2019-cloud-init-configuration-for-virt-manager-virt-install/
> >>
> >> Right now the code is sitting in a virt-manager.git branch, not yet
> >> pushed to master:
> >> https://github.com/virt-manager/virt-manager/tree/cloudinit
> >>
> >> I'll summarize the current behavior, and then ask some questions.
> >>
> >>
> >> The branch adds a new 'virt-install --cloud-init' argument with several
> >> sub options. When specified, virt-install generates an empty meta-data
> >> file, a user-data file with the requested changes, stuffs them both into
> >> a cidata iso, which is used for the first VM boot and then deleted.
> >>
> >> This behavior is only triggered when --cloud-init is specified in some
> >> form, there's no automagic invocation of this support.
> >>
> >> The command sub options are:
> >>
> >> $ ./virt-install --cloud-init=help
> >> --cloud-init options:
> >>   disable
> >>   root-password-file
> >>   root-password-generate
> >>   ssh-key
> >>   user-data
> >>
> >> Their behavior:
> >>
> >> * disable=yes: boolean option to disable cloud-init in the VM for
> >> subsequent boots. Adds this block to user-data:
> >>   runcmd:
> >>   - [ sudo, touch, /etc/cloud/cloud-init.disabled ]
> >>
> >> * root-password-file=/MY/PATH: set the desired root password from the
> >> content of /MY/PATH on the host
> >>
> >> * root-password-generate=yes: boolean option, generate a random root
> >> password, set it in user-data, print it to the host text console and
> >> pause for a bit for the user to see and copy it. sorta inspired by
> >> virt-builder
> >>
> >> * ssh-key=/MY/KEY.pub: inject the ssh key from /MY/KEY.pub on the host
> >> into the cloud-init user-data
> >>
> >> * user-data=/PATH/TO/user-data: ignore all other options and copy this
> >> file to the .iso as user-data
> >>
> >> When bare '--cloud-init' is specified, we default to generating a random
> >> root password and disabling cloud-init for subsequent boots:
> >> --cloud-init root-password-generate=yes,disable=yes
> >>
> >> We've explicitly rejected something like root-password=MY-PASSWORD
> >> because of the security implications of encouraging a password to end up
> >> in command line history. We've already had a CVE for something similar
> >> in virt-install.
> >>
> >> Also, I don't want virt-install to be in the business of specifying
> >> every cloud-init option under the sun, there's gotta be a better tool
> >> for that already. So I'd like to keep --cloud-init suboptions
> >> specifically targeted to expected virt use cases, and anything else can
> >> be served with custom user-data=
> >>
> >> One more point: my main interaction with cloud-init has historically
> >> been by grabbing a Fedora/RHEL cloud image, passing it to
> >> virt-install/virt-manager, and watching the boot hang, because there's
> >> no data provider and cloud-init times out talking to the network, and
> >> then I can't log in. I expect many people have hit this issue before.
> >> I've always worked around this by using 'virt-customize' to disable
> >> cloud-init and reset the root password. That's about the extent of my
> >> usage here, which is broadly why the bare `--cloud-init` is the way it
> was.
> >>
> >> I'm also thinking to the future, if one day virt-install can detect that
> >> it was passed a distro cloud-init image, perhaps we can invoke some
> >> default behavior that gives the user a better chance of this config
> >> being usable out of the box. I figure that will match whatever we choose
> >> for the bare '--cloud-init' behavior
> >
> > One option is to take the lazy approach and always enable cloud init
> > if we are given a pre-built disk image.
> >
> >  * If the disk image doesn't support cloud init, then its fairly
> >    harmless to provide cloud-init data.
> >
> >  * If the disk image does support cloud init but has already been
> >    booted, and the user has marked cloud init as disable, then it
> >    is again harmless to provide cloud-init data
> >
> >  * If the disk image does support cloud init but has already been
> >    booted, and the user left cloud init enabled so that it runs on
> >    every boot, then passing cloud init data is probably good thing.
> >
> >  * If the disk image does support cloud init and has never been
> >    booted, then obviously providing cloud init data is good
> >
>
> If we had a SMBIOS or -fw_cfg option, then maybe this is an option. But
> I don't think I could stomach doing this by default with the current
> 'cdrom' approach, surely there would be complications, and at minimum
> complaints about a needless attached CDROM device.
>
> >> * What are the usecases you see for virt-install cloud-init support?
> >
> > To me the critical use case is simply providing a way for the user to
> > login to the guest. This is covered by password and/or SSH key inject.
> > Once they have that working, anything else is just nice to have.
> >
> >> * Does the above meet your expectations?
> >
> > Yep
> >
> >> * Are we missing anything vital?
> >
> > Probably the main question will be around the transport for providing
> > the metadata.
> >
> > The impl does config-drive:
> >
> >
> https://cloudinit.readthedocs.io/en/latest/topics/datasources/configdrive.html
> >
> > As you see from the above list, there are many other approaches
> > possible, though the majority of them end up being based on a
> > magic IP address.
> >
> > The main downside of using a config-drive is that it is visible
> > to the guest as as new disk. In simple cases this is no big deal,
> > but if the user is setting up a VM with multiple disks, it can be
> > annoying to have an extra disk added for cloud-init, as it can
> > cause the drive-letters to be reshuffled.  OpenStack tried to
> > mitigate this by making the cloud-init config drive the very
> > last one to be added.
> >
> > This isn't perfect though. eg consider you boot the VM with
> > config drive, then hotplug a disk, now reboot the VM. The config
> > drive disk is now gone, and the hotplugged disk may well have
> > changed its drive letter.
> >
> > This is one reason for the use of the IP based metadata service.
> >
> > The challenge in any of the IP based solutions is that something on
> > the host needs to provide a server to listen for these requests.
> >
> > I think that providing such a server is out of scope for virt-install.
> >
> > I could see value in someone creating a new project that provides a
> > simple self-contained metadata server, as all the currentl metadata
> > server impls are part of some larger project codebase and can't be
> > used independantly (AFAIK - someone correct me if a simple standalone
> > impl already exists somewhere).
> >
> > In any case if someone wants to use a network metadata server, then
> > virt-install doesn't need todo anything at all. Any config is entirely
> > respponsibility of whomever is running that service.
> >
> >
> > If neither config drive, or IP based solutions are viable, then you
> > are left with platform specific options. Ignition uses fw_cfg however
> > this is really frowned upon by QEMU maintainers, as fw_cfg is not
> > intended for general purpose application usage.  The suggested
> replacement
> > for fw_cfg for app usage is the SMBIOS OEM strings table.
> >
> > Neither of these are platform portable, just x86 + aarch64 right now.
> >
>
> Thanks for the breakdown. There's obviously a lot to consider here and a
> large test matrix to consider if we go with the non-cdrom case.
>
> I think the 'cdrom' case is good enough for a first crack at this and
> anything else can build on that initial support. I think the command
> line is flexible enough that in the future we could have a --cloud-init
> method=SMBIOS/... for users to explicitly choose what injection method
> they want, but otherwise virt-install will choose a default, which may
> be different depending on the arch.
>
> >
> >> * Do you have an opinion of what behavior bare '--cloud-init' should
> give?
> >
> > My feeling is that a bare --cloud-init should inject the current user's
> > SSH identity and/or authorized keys list.
> >
> > ie either we grab $HOME/.ssh/id_rsa.pub, or we grab all the keys from
> > the SSH agent connection (ie ssh-add -l)
> >
> > Given that virt-intsall is run as root, there probably isn't a
> > /root/.ssh/id_rsa.pub file for most people. Grabbing the ssh-agent
> > authorized keys is probably best default approach.
>
> How to handle ssh keys kinda makes my head spin. I've shrunk the CC list
> on this mail to just the folks that mentioned ssh in some way in their
> replies
>
> If I'm thinking of the ideal interactive virt-install UI, it would be:
>
> $ virt-install --name MYVM --disk Fedora-Cloud-...qcow2 --cloud-init
> INFO: SSH keys added to MYVM
> INFO: Connecting to MYVM with: ssh fedora at 192.168.122.XXX
> ...
> [fedora at localhost:~] $
>
>
> If all the pieces come together, that's pretty nice. But there's a lot
> of missing pieces needed for that, and some unclear bits:
>
> * What is the cloud-init user name? Passing in an ssh key gives access
> to the default cloud-init configured username, which is different
> depending on the distro (might be fedora, ubuntu, etc). To properly
> handle this, we either need libosinfo distro detection, or use
> libguestfs to fish the default user out of /etc/cloud/cloud.cfg in the
> VM. And root account is typically locked, and ssh access disabled, and
> even if we could hack things to give root ssh access it seems like going
> against the grain WRT cloud images.
>

We've had a similar discussion for multipass here, I've pointed out
a few options, one using the ssh_redirect_user

 https://github.com/CanonicalLtd/multipass/issues/929

Currently cloud-init does not interact with the host in a way that this
information is easily discoverable; ideally, cloud-init could emit some
of this information to the platform; similar to how cloud-init publishes
ssh public host keys to console, and on GCE to the metadata service,
cloud-init could publish information like what's the default user.

That still leaves the existing and older images which need something.

Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/virt-tools-list/attachments/20191122/09f6f2d5/attachment.htm>