[virt-tools-list] virt-install and cloud-init, feedback wanted

[George]

On Saturday, 23-11-2019 at 15:07 Cole Robinson wrote:
> On 11/22/19 5:12 AM, Daniel P. Berrangé wrote:
> > On Thu, Nov 21, 2019 at 07:44:38PM -0500, Cole Robinson wrote:
> >> On 11/21/19 5:28 AM, Daniel P. Berrangé wrote:
> >>> One option is to take the lazy approach and always enable cloud init
> >>> if we are given a pre-built disk image.
> >>>
> >>>  * If the disk image doesn't support cloud init, then its fairly
> >>>    harmless to provide cloud-init data.
> >>>
> >>>  * If the disk image does support cloud init but has already been
> >>>    booted, and the user has marked cloud init as disable, then it
> >>>    is again harmless to provide cloud-init data
> >>>
> >>>  * If the disk image does support cloud init but has already been
> >>>    booted, and the user left cloud init enabled so that it runs on
> >>>    every boot, then passing cloud init data is probably good thing.
> >>>
> >>>  * If the disk image does support cloud init and has never been
> >>>    booted, then obviously providing cloud init data is good
> >>>    
> >>
> >> If we had a SMBIOS or -fw_cfg option, then maybe this is an option. But
> >> I don't think I could stomach doing this by default with the current
> >> 'cdrom' approach, surely there would be complications, and at minimum
> >> complaints about a needless attached CDROM device.
> > 
> > Yeah, I can understand this concern.
> > 
> >> I think the 'cdrom' case is good enough for a first crack at this and
> >> anything else can build on that initial support. I think the command
> >> line is flexible enough that in the future we could have a --cloud-init
> >> method=SMBIOS/... for users to explicitly choose what injection method
> >> they want, but otherwise virt-install will choose a default, which may
> >> be different depending on the arch.
> > 
> > That sounds reasonable.
> > 
> >>>> * Do you have an opinion of what behavior bare '--cloud-init' should give?
> >>>
> >>> My feeling is that a bare --cloud-init should inject the current user's
> >>> SSH identity and/or authorized keys list.
> >>>
> >>> ie either we grab $HOME/.ssh/id_rsa.pub, or we grab all the keys from
> >>> the SSH agent connection (ie ssh-add -l)
> >>>
> >>> Given that virt-intsall is run as root, there probably isn't a
> >>> /root/.ssh/id_rsa.pub file for most people. Grabbing the ssh-agent
> >>> authorized keys is probably best default approach.
> >>
> >> How to handle ssh keys kinda makes my head spin. I've shrunk the CC list
> >> on this mail to just the folks that mentioned ssh in some way in their
> >> replies
> >>
> >> If I'm thinking of the ideal interactive virt-install UI, it would be:
> >>
> >> $ virt-install --name MYVM --disk Fedora-Cloud-...qcow2 --cloud-init
> >> INFO: SSH keys added to MYVM
> >> INFO: Connecting to MYVM with: ssh fedora at 192.168.122.XXX
> >> ...
> >> [fedora at localhost:~] $
> >>
> >>
> >> If all the pieces come together, that's pretty nice. But there's a lot
> >> of missing pieces needed for that, and some unclear bits:
> >>
> >> * What is the cloud-init user name? Passing in an ssh key gives access
> >> to the default cloud-init configured username, which is different
> >> depending on the distro (might be fedora, ubuntu, etc). To properly
> >> handle this, we either need libosinfo distro detection, or use
> >> libguestfs to fish the default user out of /etc/cloud/cloud.cfg in the
> >> VM. And root account is typically locked, and ssh access disabled, and
> >> even if we could hack things to give root ssh access it seems like going
> >> against the grain WRT cloud images.
> > 
> > IIRC, you can pass an SSH key for root separately, but not sure how
> > widely supported that is. Sticking to just providing 1 SSH key for
> > the default unprivileged user likely suffices.
> > 
> >> So best I can think of for an immediate solution for SSH is to print
> >> something explanatory:
> >>
> >> INFO: you can ssh to the VM using the distro's default username. This is
> >> usually one of: fedora, ubuntu, ...
> >>
> >> Or something along those lines. Maybe we can tie that into --os-variant
> >> as well so the user can inform us what distro it is. Still it will take
> >> some work.
> > 
> > This is a bit of a tangent, but this makes me wonder if qcow2 format
> > could benefit from having support for arbitrary user metadata fields
> > 
> > People building cloud images could then tag their images with useful
> > things including vendor, product name, product version, whether it
> > supports cloud-init or ignition, what the default user account name
> > is, suggested min ram, and so on.
> > 
> > Of course people might say this is all stuff that's within scope
> > of external metadata like OVF, but I've rarely, if ever, seen anyone
> > say anything nice about OVF. Also there's a certain convenience in
> > being able to directly pass around a bootable disk image, rather than
> > a disk image + separate metadata file, or worse a ZIP file containing
> > both which you have to unpack.
> > 
> > Having some simple metadata in the qcow2 file would simplify life I
> > think. If someone really wanted to use OVF, they could stuff the OVF
> > XML descriptor into a qcow2 metdata field.
> > 
> >> * Where to get ssh keys from? You mention ssh-add -l which is nice, I
> >> didn't know about that. There's the issue with running virt-install
> >> under sudo, but not sure there's anything we can do about that to make
> >> it 'just work'. Sounds like `ssh-add -l` is the way though for some sort
> >> of ssh-key=auto mode
> > 
> > Yeah, sudo breaks it, as does "su -".   The ssh agent only gets
> > forwarded if you're accessing the virt host over SSH. This is
> > the case for me usually, as my virt servers are separate from
> > my laptop.
> > 
> > I don't see much of an alternative to ssh-agent though, besides
> > just defining that virt-install will grab a key from a well known
> > place like $HOME/.config/virt-install/ssh-injection-keys
> > 
> >> * What to do if we can't find any ssh-keys to add? I guess explicitly
> >> fail, and print a descriptive error: We didn't find any SSH keys to add.
> >> Maybe you are running under sudo. You can manually specify a key with
> >> XXX or generate root password with YYY. (something better than that of
> >> course)
> > 
> > Depends whether we want to also auto-generate root and/or user passwords
> > by default, or only work with SSH keys by default.
> > 
> > My personal preference is to only deal with SSH keys by defaults, as
> > passwords are just terrible in general. This would mean if we can't
> > find a SSH key, we should abort the install & require an explicit
> > setting.
> > 
> >> * virt-install support for launching an ssh session as the UI. This is
> >> probably worth doing regardless. I'd like to add an explicit
> >> --autoconsole none|graphical|text anyways, adding ssh to that seems
> >> reasonable. Requires waiting for VM IP address to pop up, but that
> >> shouldn't be too bad to implement. The main problem for the --cloud-init
> >> case is that I don't know if it makes sense to use this if we are just
> >> guessing about the what the default username is?
> > 
> > We would need to solve the default username problem reliably
> > for this to be usable, otherwise it'll be a neverending source of
> > bug reports.
> > 
> >> So, to get that ideal UI, or at least point the user in the right
> >> direction, we need to:
> >>
> >> * determine the default user name for the disk image, which we don't
> >> have a solution for presently
> >> * successfully inject a relevant ssh key, which may not be available
> >> like under sudo
> >>
> >> So there's missing non-trivial work, and there's ways it can fail.
> >>
> >> The current UI, with root-password-generate=on (plus some work to
> >> default to '--console text' when --cloud-init is passed), is more
> >> reliable as it should work in all situations, but even when it works
> >> it's not as friendly as the working ssh case (need to copy the printed
> >> password and paste it into a terminal). And it doesn't sound like it's
> >> what people expect the default to be. And giving root access seems to be
> >> against the spirit of the cloud images, but maybe that's just for ssh access
> > 
> > Personally I think from a security POV we should do our best to
> > discourage & avoid use of passwords, at least for server based
> > installs.
> > 
> > IOW, if using the JEOS profile then we should prefer SSH keys.
> > Essentially my view is that passwords are just there as a fallback
> > for disaster recovery when you have to boot single user mode to
> > fix a broken system. Even then we don't really need that for
> > VMs, because virt-rescue provides a backdoor into any broken
> > VM.
> > 
> > The key place where passwords are needed are when we are doing
> > graphical installs. The user connects to the graphical console
> > with VNC/SPICE, and password is the only option there really.
> > 
> 
> Okay after mulling it over I think I've come full circle :) My plan:
> bare --cloud-init stays as is: generate a root password and set it to
> immediately expire. Anything ssh needs to be explicitly opt in. Never
> plan to trigger any --cloud-init behavior automatically, require it to
> be opt in.
> 
> Here's my thinking: the ssh case is too complicated to be the default.
> We've covered it in previous mails; there's many ways it may not 'just
> work'. It might be what some or most users want out of using cloud
> images with virt-install/virt-manager, but those users likely 1) already
> have some cloud-init or cloud image specific knowledge, and 2) have a
> good understanding of ssh keys. So, I'm comfortable letting those users
> figure out themselves that they need --cloud-init ssh... or similar.
> 
> The root-password-generate=on case, combined with using `--autoconsole
> text` to give terminal login access to the VM is the best mix I can come
> up with of simple and reliable. It should always work, and doesn't take
> much special explanation besides 'this is your root password, copy it
> into the login prompt', and then set a new password on first login. I
> understand that we shouldn't really be encouraging password auth but I
> think it's unavoidable here as a 'please just let me log in to this disk
> image' hammer
> 
> Another benefit of doing this as the sorta recommended --cloud-init
> option for virt-install is that it will be closer to what I think we
> should provide in virt-manager UI: some checkbox like 'This is a cloud
> image' that gives an option to set a custom root password. The user can
> log in to the imported VM easy enough through the graphical console.
> 
> I was making a big deal out of this bare '--cloud-init' option because I
> was presuming that some day we may want to use it by default if we
> detect we are passed a cloud image. But I've changed my mind a bit. Even
> if today we could perfectly detect if the user provided disk image is a
> distro cloud image, I think all we should do for virt-install is print a
> warning like: WARNING: distro cloud image detected. you may not be able
> to log in without specifying --cloud-init options. try FOO or BAR...`.
> This way the user always has to explicitly request some kind of
> --cloud-init interaction. I think this is the only way avoid a lot of
> ambiguity,  and possibly avoid undesirable behavior in some subtle
> scenarios.
> 
> The one case I could see us using some --cloud-init values by default,
> is if we use libosinfo to point us to a distros's cloud image which we
> download on demand. Say something like 'virt-install --install
> fedora31,variant=cloud`. In that case we know we are dealing with a
> cloud image, which distro, we know the image is pristine, so it would
> make sense to default to --cloud-init as well.
> 
> But in the end, bare `--cloud-init` will still be the recommendation for
> someone who justs wants to log in to that distro disk image they
> downloaded. If the user is cloud-init savy and knows they want to get
> ssh keys into the VM, they likely know enough to figure out how to make
> it work with the explicit options we will provide.
> 
> 
> Here's a finer grained task list. But for those who have read this far,
> I'm not planning on working on this in the near term, probably not
> before the end of the year at least
> 
> * Add --autoconsole none|default|text|graphical
> * Any --cloud-init usage implies --autoconsole text.
> * Add --cloud-init ssh-key=agent or similar: the ssh-add -l behavior. If
> ssh-add -l fails, print some hints and fail
> * Add `--cloud-init none` as future proof incase we ever choose to make
> --cloud-init implied for some usecases
> * make sure root-password-generate=on doesn't pause output unless run
> interactively
> * test a bunch of different distro cloud images, make sure our settings work
> * man page document it all


> * possibly virt-manager UI for it

+1 

> * stretch: possibly virt-install --install fedora30,variant=cloud install
> * stretch: libosinfo track default cloud image usernames. may come in
> handy at some point
> 
> Thanks,
> Cole
> 
> _______________________________________________
> virt-tools-list mailing list
> virt-tools-list at redhat.com
> https://www.redhat.com/mailman/listinfo/virt-tools-list