[virt-tools-list] virt-install and cloud-init, feedback wanted

[Dusty Mabe]

On 11/20/19 5:49 PM, Cole Robinson wrote:
> Hi all. The purpose of this mail is to get some feedback on pending
> cloud-init support in virt-install. If you're on the CC list here, I
> either pulled your email from a cloud-init discussion on the the
> virt-tools-list mailing list, or from the CC list of this virt-install bug:
> 
> RFE: Provide cloud-init integration for VMs
> https://bugzilla.redhat.com/show_bug.cgi?id=981693
> 
> For GSOC 2019 Athina Plaskasoviti completed some cloud-init integration
> work in virt-install. You can read her wrap up here:
> https://athinapl.home.blog/2019/08/25/gsoc-2019-cloud-init-configuration-for-virt-manager-virt-install/

First off, thanks Cole and Athina for working on this. This is something I have
wanted for a long time! In my dayjob I've moved away from cloud-init a bit because
our new platforms are running Ignition: https://github.com/coreos/ignition

>From the Ignition side I've added Andrew Jeddeloh and Benjamin Gilbert to CC.

> 
> Right now the code is sitting in a virt-manager.git branch, not yet
> pushed to master:
> https://github.com/virt-manager/virt-manager/tree/cloudinit
> 
> I'll summarize the current behavior, and then ask some questions.
> 
> 
> The branch adds a new 'virt-install --cloud-init' argument with several
> sub options. When specified, virt-install generates an empty meta-data
> file, a user-data file with the requested changes, stuffs them both into
> a cidata iso, which is used for the first VM boot and then deleted.

Is there any chance we could go a different route than an ISO image? The developers
of ignition have experienced problems with relying on a device here (i.e. is
there no device coming or has the device just not shown up yet?). Some discussion over
in: https://github.com/coreos/ignition/issues/656

Basically in Fedora CoreOS we need a generic user data mechanism that works across
platforms (x86_64, aarch64, ppc64le, s390x) and doesn't have possible race conditions.
Right now we're using `-fw_cfg` but it's not cross platform. We don't have an answer
yet: https://github.com/coreos/ignition/issues/666#issuecomment-452835654

> 
> This behavior is only triggered when --cloud-init is specified in some
> form, there's no automagic invocation of this support.
> 
> The command sub options are:
> 
> $ ./virt-install --cloud-init=help
> --cloud-init options:
>   disable
>   root-password-file
>   root-password-generate
>   ssh-key
>   user-data
> 
> Their behavior:
> 
> * disable=yes: boolean option to disable cloud-init in the VM for
> subsequent boots. Adds this block to user-data:
>   runcmd:
>   - [ sudo, touch, /etc/cloud/cloud-init.disabled ]
> 
> * root-password-file=/MY/PATH: set the desired root password from the
> content of /MY/PATH on the host
> 
> * root-password-generate=yes: boolean option, generate a random root
> password, set it in user-data, print it to the host text console and
> pause for a bit for the user to see and copy it. sorta inspired by
> virt-builder
> 
> * ssh-key=/MY/KEY.pub: inject the ssh key from /MY/KEY.pub on the host
> into the cloud-init user-data
> 
> * user-data=/PATH/TO/user-data: ignore all other options and copy this
> file to the .iso as user-data
> 
> When bare '--cloud-init' is specified, we default to generating a random
> root password and disabling cloud-init for subsequent boots:
> --cloud-init root-password-generate=yes,disable=yes
> 
> We've explicitly rejected something like root-password=MY-PASSWORD
> because of the security implications of encouraging a password to end up
> in command line history. We've already had a CVE for something similar
> in virt-install.
> 
> Also, I don't want virt-install to be in the business of specifying
> every cloud-init option under the sun, there's gotta be a better tool
> for that already. So I'd like to keep --cloud-init suboptions
> specifically targeted to expected virt use cases, and anything else can
> be served with custom user-data=

I think we should keep it as slim as possible. You've given users enough
to get in to the machine. If they need to do anything else they can either
provision via SSH after boot or they can pass a custom user data to do more
provisioning.

> 
> One more point: my main interaction with cloud-init has historically
> been by grabbing a Fedora/RHEL cloud image, passing it to
> virt-install/virt-manager, and watching the boot hang, because there's
> no data provider and cloud-init times out talking to the network, and
> then I can't log in. I expect many people have hit this issue before.
> I've always worked around this by using 'virt-customize' to disable
> cloud-init and reset the root password. That's about the extent of my
> usage here, which is broadly why the bare `--cloud-init` is the way it was.

If you need any help understanding things or testing them I can try to help!

> 
> I'm also thinking to the future, if one day virt-install can detect that
> it was passed a distro cloud-init image, perhaps we can invoke some
> default behavior that gives the user a better chance of this config
> being usable out of the box. I figure that will match whatever we choose
> for the bare '--cloud-init' behavior
> 
> 
> So, my questions are:
> 
> * What are the usecases you see for virt-install cloud-init support?
> * Does the above meet your expectations?
> * Are we missing anything vital?

The only thing I can think of is *maybe* being able to specify which user you want
the ssh-key to apply to.

> * Do you have an opinion of what behavior bare '--cloud-init' should give?
> * Any other ideas, thoughts, feedback?

I would love to have something like this support Ignition as well. Do you think
that would be welcome? 

If so, does it change your thoughts on the approach? Would we stick with `--cloud-init`
and just add `--ignition` or would we want to group it under another heading
`--userdata type=cloud-init,ssh-key=.....` `--userdata type=ignition,ssh-key=...`?

Either way I'm glad we're having this conversation!

Dusty