[virt-tools-list] virt-install and cloud-init, feedback wanted

[Cole Robinson]

On 11/22/19 5:12 AM, Daniel P. Berrangé wrote:
> On Thu, Nov 21, 2019 at 07:44:38PM -0500, Cole Robinson wrote:
>> On 11/21/19 5:28 AM, Daniel P. Berrangé wrote:
>>> One option is to take the lazy approach and always enable cloud init
>>> if we are given a pre-built disk image.
>>>
>>>  * If the disk image doesn't support cloud init, then its fairly
>>>    harmless to provide cloud-init data.
>>>
>>>  * If the disk image does support cloud init but has already been
>>>    booted, and the user has marked cloud init as disable, then it
>>>    is again harmless to provide cloud-init data
>>>
>>>  * If the disk image does support cloud init but has already been
>>>    booted, and the user left cloud init enabled so that it runs on
>>>    every boot, then passing cloud init data is probably good thing.
>>>
>>>  * If the disk image does support cloud init and has never been
>>>    booted, then obviously providing cloud init data is good
>>>    
>>
>> If we had a SMBIOS or -fw_cfg option, then maybe this is an option. But
>> I don't think I could stomach doing this by default with the current
>> 'cdrom' approach, surely there would be complications, and at minimum
>> complaints about a needless attached CDROM device.
> 
> Yeah, I can understand this concern.
> 
>> I think the 'cdrom' case is good enough for a first crack at this and
>> anything else can build on that initial support. I think the command
>> line is flexible enough that in the future we could have a --cloud-init
>> method=SMBIOS/... for users to explicitly choose what injection method
>> they want, but otherwise virt-install will choose a default, which may
>> be different depending on the arch.
> 
> That sounds reasonable.
> 
>>>> * Do you have an opinion of what behavior bare '--cloud-init' should give?
>>>
>>> My feeling is that a bare --cloud-init should inject the current user's
>>> SSH identity and/or authorized keys list.
>>>
>>> ie either we grab $HOME/.ssh/id_rsa.pub, or we grab all the keys from
>>> the SSH agent connection (ie ssh-add -l)
>>>
>>> Given that virt-intsall is run as root, there probably isn't a
>>> /root/.ssh/id_rsa.pub file for most people. Grabbing the ssh-agent
>>> authorized keys is probably best default approach.
>>
>> How to handle ssh keys kinda makes my head spin. I've shrunk the CC list
>> on this mail to just the folks that mentioned ssh in some way in their
>> replies
>>
>> If I'm thinking of the ideal interactive virt-install UI, it would be:
>>
>> $ virt-install --name MYVM --disk Fedora-Cloud-...qcow2 --cloud-init
>> INFO: SSH keys added to MYVM
>> INFO: Connecting to MYVM with: ssh fedora at 192.168.122.XXX
>> ...
>> [fedora at localhost:~] $
>>
>>
>> If all the pieces come together, that's pretty nice. But there's a lot
>> of missing pieces needed for that, and some unclear bits:
>>
>> * What is the cloud-init user name? Passing in an ssh key gives access
>> to the default cloud-init configured username, which is different
>> depending on the distro (might be fedora, ubuntu, etc). To properly
>> handle this, we either need libosinfo distro detection, or use
>> libguestfs to fish the default user out of /etc/cloud/cloud.cfg in the
>> VM. And root account is typically locked, and ssh access disabled, and
>> even if we could hack things to give root ssh access it seems like going
>> against the grain WRT cloud images.
> 
> IIRC, you can pass an SSH key for root separately, but not sure how
> widely supported that is. Sticking to just providing 1 SSH key for
> the default unprivileged user likely suffices.
> 
>> So best I can think of for an immediate solution for SSH is to print
>> something explanatory:
>>
>> INFO: you can ssh to the VM using the distro's default username. This is
>> usually one of: fedora, ubuntu, ...
>>
>> Or something along those lines. Maybe we can tie that into --os-variant
>> as well so the user can inform us what distro it is. Still it will take
>> some work.
> 
> This is a bit of a tangent, but this makes me wonder if qcow2 format
> could benefit from having support for arbitrary user metadata fields
> 
> People building cloud images could then tag their images with useful
> things including vendor, product name, product version, whether it
> supports cloud-init or ignition, what the default user account name
> is, suggested min ram, and so on.
> 
> Of course people might say this is all stuff that's within scope
> of external metadata like OVF, but I've rarely, if ever, seen anyone
> say anything nice about OVF. Also there's a certain convenience in
> being able to directly pass around a bootable disk image, rather than
> a disk image + separate metadata file, or worse a ZIP file containing
> both which you have to unpack.
> 
> Having some simple metadata in the qcow2 file would simplify life I
> think. If someone really wanted to use OVF, they could stuff the OVF
> XML descriptor into a qcow2 metdata field.
> 
>> * Where to get ssh keys from? You mention ssh-add -l which is nice, I
>> didn't know about that. There's the issue with running virt-install
>> under sudo, but not sure there's anything we can do about that to make
>> it 'just work'. Sounds like `ssh-add -l` is the way though for some sort
>> of ssh-key=auto mode
> 
> Yeah, sudo breaks it, as does "su -".   The ssh agent only gets
> forwarded if you're accessing the virt host over SSH. This is
> the case for me usually, as my virt servers are separate from
> my laptop.
> 
> I don't see much of an alternative to ssh-agent though, besides
> just defining that virt-install will grab a key from a well known
> place like $HOME/.config/virt-install/ssh-injection-keys
> 
>> * What to do if we can't find any ssh-keys to add? I guess explicitly
>> fail, and print a descriptive error: We didn't find any SSH keys to add.
>> Maybe you are running under sudo. You can manually specify a key with
>> XXX or generate root password with YYY. (something better than that of
>> course)
> 
> Depends whether we want to also auto-generate root and/or user passwords
> by default, or only work with SSH keys by default.
> 
> My personal preference is to only deal with SSH keys by defaults, as
> passwords are just terrible in general. This would mean if we can't
> find a SSH key, we should abort the install & require an explicit
> setting.
> 
>> * virt-install support for launching an ssh session as the UI. This is
>> probably worth doing regardless. I'd like to add an explicit
>> --autoconsole none|graphical|text anyways, adding ssh to that seems
>> reasonable. Requires waiting for VM IP address to pop up, but that
>> shouldn't be too bad to implement. The main problem for the --cloud-init
>> case is that I don't know if it makes sense to use this if we are just
>> guessing about the what the default username is?
> 
> We would need to solve the default username problem reliably
> for this to be usable, otherwise it'll be a neverending source of
> bug reports.
> 
>> So, to get that ideal UI, or at least point the user in the right
>> direction, we need to:
>>
>> * determine the default user name for the disk image, which we don't
>> have a solution for presently
>> * successfully inject a relevant ssh key, which may not be available
>> like under sudo
>>
>> So there's missing non-trivial work, and there's ways it can fail.
>>
>> The current UI, with root-password-generate=on (plus some work to
>> default to '--console text' when --cloud-init is passed), is more
>> reliable as it should work in all situations, but even when it works
>> it's not as friendly as the working ssh case (need to copy the printed
>> password and paste it into a terminal). And it doesn't sound like it's
>> what people expect the default to be. And giving root access seems to be
>> against the spirit of the cloud images, but maybe that's just for ssh access
> 
> Personally I think from a security POV we should do our best to
> discourage & avoid use of passwords, at least for server based
> installs.
> 
> IOW, if using the JEOS profile then we should prefer SSH keys.
> Essentially my view is that passwords are just there as a fallback
> for disaster recovery when you have to boot single user mode to
> fix a broken system. Even then we don't really need that for
> VMs, because virt-rescue provides a backdoor into any broken
> VM.
> 
> The key place where passwords are needed are when we are doing
> graphical installs. The user connects to the graphical console
> with VNC/SPICE, and password is the only option there really.
> 

Okay after mulling it over I think I've come full circle :) My plan:
bare --cloud-init stays as is: generate a root password and set it to
immediately expire. Anything ssh needs to be explicitly opt in. Never
plan to trigger any --cloud-init behavior automatically, require it to
be opt in.

Here's my thinking: the ssh case is too complicated to be the default.
We've covered it in previous mails; there's many ways it may not 'just
work'. It might be what some or most users want out of using cloud
images with virt-install/virt-manager, but those users likely 1) already
have some cloud-init or cloud image specific knowledge, and 2) have a
good understanding of ssh keys. So, I'm comfortable letting those users
figure out themselves that they need --cloud-init ssh... or similar.

The root-password-generate=on case, combined with using `--autoconsole
text` to give terminal login access to the VM is the best mix I can come
up with of simple and reliable. It should always work, and doesn't take
much special explanation besides 'this is your root password, copy it
into the login prompt', and then set a new password on first login. I
understand that we shouldn't really be encouraging password auth but I
think it's unavoidable here as a 'please just let me log in to this disk
image' hammer

Another benefit of doing this as the sorta recommended --cloud-init
option for virt-install is that it will be closer to what I think we
should provide in virt-manager UI: some checkbox like 'This is a cloud
image' that gives an option to set a custom root password. The user can
log in to the imported VM easy enough through the graphical console.

I was making a big deal out of this bare '--cloud-init' option because I
was presuming that some day we may want to use it by default if we
detect we are passed a cloud image. But I've changed my mind a bit. Even
if today we could perfectly detect if the user provided disk image is a
distro cloud image, I think all we should do for virt-install is print a
warning like: WARNING: distro cloud image detected. you may not be able
to log in without specifying --cloud-init options. try FOO or BAR...`.
This way the user always has to explicitly request some kind of
--cloud-init interaction. I think this is the only way avoid a lot of
ambiguity,  and possibly avoid undesirable behavior in some subtle
scenarios.

The one case I could see us using some --cloud-init values by default,
is if we use libosinfo to point us to a distros's cloud image which we
download on demand. Say something like 'virt-install --install
fedora31,variant=cloud`. In that case we know we are dealing with a
cloud image, which distro, we know the image is pristine, so it would
make sense to default to --cloud-init as well.

But in the end, bare `--cloud-init` will still be the recommendation for
someone who justs wants to log in to that distro disk image they
downloaded. If the user is cloud-init savy and knows they want to get
ssh keys into the VM, they likely know enough to figure out how to make
it work with the explicit options we will provide.


Here's a finer grained task list. But for those who have read this far,
I'm not planning on working on this in the near term, probably not
before the end of the year at least

* Add --autoconsole none|default|text|graphical
* Any --cloud-init usage implies --autoconsole text.
* Add --cloud-init ssh-key=agent or similar: the ssh-add -l behavior. If
ssh-add -l fails, print some hints and fail
* Add `--cloud-init none` as future proof incase we ever choose to make
--cloud-init implied for some usecases
* make sure root-password-generate=on doesn't pause output unless run
interactively
* test a bunch of different distro cloud images, make sure our settings work
* man page document it all
* possibly virt-manager UI for it
* stretch: possibly virt-install --install fedora30,variant=cloud install
* stretch: libosinfo track default cloud image usernames. may come in
handy at some point

Thanks,
Cole