[virt-tools-list] [virt-manager PATCH][v2][RFC] Introduction of cloud-init configuration in virt-install

[Cole Robinson]

On 6/25/19 6:17 AM, Daniel P. Berrangé wrote:
> On Mon, Jun 24, 2019 at 01:53:38PM -0400, Cole Robinson wrote:
>> On 6/24/19 12:40 PM, Daniel P. Berrangé wrote:
>>> On Mon, Jun 24, 2019 at 12:12:16PM -0400, Cole Robinson wrote:
>>>> On 6/21/19 1:18 PM, Athina Plaskasoviti wrote:
>>>>> I am sending a v2 of the patch for cloud init basic configuration during cloud image installation, applying previous suggestions.
>>>>>
>>>>> Fixed: 
>>>>>
>>>>> -the way the installation is triggered (no_install is no longer required):
>>>>> 	--install is_cloud=yes ... --import
>>>>>
>>>>> -added tmpfile for the files that are being generated in the installation process 
>>>>> (meta-data, user-data, config.iso)
>>>>>
>>>>> -changed condition in installerinject.py from label='cloud' to cloudinit=True
>>>>>
>>>>
>>>>
>>>> I did some digging into cloud-init behavior to try and get a better idea
>>>> of what the minimal viable config is, and get ideas for future work.
>>>>
>>>>
>>>> cloud-init behavior
>>>> ===================
>>>>
>>>> The patch generates an .iso with volume label 'cidata' which is the
>>>> magic to make cloud-init search it for config data. cloud-init must find
>>>> a 'meta-data' and 'user-data' file in order to consider the iso a valid
>>>> data source. There isn't any requirements on the contents though. They
>>>> can both basically be empty AFAICT.
>>>>
>>>> This patch is setting meta-data 'instance-id' and 'hostname' fields but
>>>> neither are required: instance-id seems to only be needed on real cloud
>>>> providers (and cloud-init fills in a 'nocloud' value for us by default).
>>>> hostname will predictably set the VM hostname but that's not something
>>>> we need to set by default. So in fact meta-data is fine to leave empty
>>>> and I think that should be our default behavior.
>>>>
>>>> Every distro cloud image has some default config in
>>>> /etc/cloud/cloud.cfg. This is generated from cloud-init.git
>>>> tools/render-cloudcfg which generates cloud.cfg from
>>>> config/cloud.cfg.tmpl. Basically the only input to the template is the
>>>> distro name. For example fedora generates in in the RPM spec with simply:
>>>>
>>>> python3 tools/render-cloudcfg --variant fedora >
>>>> $RPM_BUILD_ROOT/%{_sysconfdir}/cloud/cloud.cfg
>>>>
>>>> cloud images seem to have this default behavior:
>>>> - ssh is enabled and running
>>>> - root account is locked
>>>> - cloud.cfg will explicitly disallow ssh to the root account, via
>>>> disable_root. Attempts to ssh to root will print an error message
>>>> pointing to the default account
>>>> - cloud.cfg will create a default account, locked, with full sudo access.
>>>>
>>>> The name of the default account depends on the value passed to the
>>>> tools/render-cloudcfg script. For Fedora it's 'fedora', Ubuntu is
>>>> 'ubuntu'. The openstack page here documents the common ones:
>>>> https://docs.openstack.org/image-guide/obtain-images.html
>>>>
>>>> An extra note: I was wrong when I said in other threads that fedora
>>>> images have a default password, they just have a default account name
>>>> which is locked by default.
>>>
>>> That good, so I was in fact right in remembering that.
>>>
>>
>> Yeah, sorry for the confusion.
>>
>>>> virt-install cli
>>>> ================
>>>>
>>>> I think the virt-install entry point for requesting cloud-init stuff
>>>> should actually be its own top level option, like --unattended, because
>>>> we may want to add suboptions and I don't want to mix those in with
>>>> --install. So I'm thinking:
>>>>
>>>> # does whatever we choose for default behavior, like --install is_cloud
>>>> virt-install --cloud-init ...
>>>>
>>>> # User provides their own user-data file. Maybe for meta-data too?
>>>> virt-install --cloud-init user-data=/path/to/user-data
>>>>
>>>> # Specify password for the default user. Yeah I know this isn't
>>>> # a good pattern to put a password on the cli but we could combine it
>>>> # with immediate password expiration so the user has to change it
>>>> # on first log in
>>>> virt-install --cloud-init password=FOOBAR
>>>>
>>>> # Something like this to inject an ssh key? But honestly I'm not
>>>> # convinced we should go down that route and instead just make users
>>>> # manually set this themselves like they would for any other VM
>>>> virt-install --cloud-init sshkey=/path/to/key
>>>
>>> I think ssh keys is the preferred way to setup cloud images in many
>>> cases as no one sane should ever be using passwords for ssh auth these
>>> days given the continual brute force attacks they face.
>>>
>>> Saying users must set a password and then manually copy their ssh key
>>> in afterwards is crazy when cloud init supports getting the ssh key
>>> in straight away without needing passwords.
>>>
>>> I would not be surprised to even find cloud images which have
>>> explicitly disabled SSH passwords entirely leaving SSH keys as the
>>> only option. Some organizations will have a complete ban on use of
>>> passwords for login purposes too.
>>>
>>
>> My impression is that most people trying to use these images are just
>> trying to get quick access to a distro VM and think a small
>> pre-installed image is the way to go about it. They expect it to 'just
>> work' like say a virt-builder image but don't realize that cloud-init is
>> going to get in their way. They see the VM hang for a while then drop to
>> a login prompt and they don't know how to log in.
>>
>> I think the goal of this work should be to make the images work like a
>> typical virt-install/virt-manager import, not to mirror the expected
>> cloud image interaction model. If a user is coming to these images with
>> knowledge of cloud-init they can probably google their way out of the
>> problem. We can give them the tools like an sshkey= option to make it
>> work like they want
> 
> I think the opposite actually. If I'm running a cloud-image locally I
> want to experiance it in exactly the same way I'll experiance it when
> running it on OpenStack / AWS later. This means I'll want to interact
> with it via SSH using public keys.  Traditional text or graphical
> consoles are not something I'd ever care about for cloud images except
> when things go badly wrong.
> 

My perception is based on hearing many complaints with the confusion I
mention above, of people coming to the cloud images without any
cloud-init knowledge.

I did some more googling though ('fedora cloud image virt-install' and
similar variations) and there's definitely a decent mix of both cases:
people just overwriting the user password, and people setting the ssh
keys. So I think we are both right :)

> I do accept though that we have a bit of a mis-match in paradigms for
> virt-manager though, since that tool is inherantly tailored around the
> idea of graphical console access.  So for virt-manager I think it does
> make sense to have a stronger focus on password based access.
> 
> virt-install though I think should be aiming to facilitate usage in
> the same way as with normal cloud software like OpenStack/AWS since
> it is a console based tool.
> 
>> Trying to facilitate ssh by default opens this whole can of worms. You
>> won't be able to login by default with virt-viewer/virt-manager or virsh
>> console. We have to have some logic for finding user ssh keys, figure
>> out how to deal with multiple keys, and error if the user doesn't have a
>> key. If we want to autolaunch ssh from virt-install, that means the user
>> sees no boot output, we need to wait for the guest IP address which
>> means validating that's possible with the selected network method, it
>> would only work remotely if we used an ssh tunnel. There's probably more
>> issues. Honestly I don't think it's worth trying to figure out this
>> whole new paradigm for a reasonably niche case like this.
> 
> I don't think any of those problems need solving.
> 

I wasn't trying to imply they need to be solved or are hard deal
breakers. They are things we need to consider in this area because I
would bet a lot of money each one would generate a confused bug report
within a couple years. That could just be warning messages

> This is a different usage from normal installs since you already have
> the guest OS provisioned. virt-install in this case is really just being
> used as a convenient tool to write the libvirt XML & create the cloud-init
> config. Thus I don't see any need to interact with the console when
> importing a pre-built image, so there's no need for virt-install to
> autolaunch SSH client, and thus need need to worry about network access
> methods, etc.
> 
> If I was using this then I'd just be runnig SSH myself afterwards, relying
> on the libvirt NSS module for the hostname eg
> 
> $  virt-install --name fedora30 \
>                 --import /var/lib/libvirt/images/fedora30.qcow
>                 --cloud-init ssh-key=$HOME/.ssh/id_rsa.pub
> 		--no-auto-console
> ....wait 10 seconds...	       
> $  ssh fedora30
> 

I doubt many people have libvirt-nss set up, plus it only works for
virtual networks. But maybe for the ssh key case we default to
--noautoconsole with a printed message trying to point the user to a way
to proceed. Unfortunately out of the box that's not exactly a simple
thing to explain, grabbing ip from virsh domifaddr. virt-install could
wait a minute for the VM to start, try to grab the IP ourselves, then
print out the ssh command for them to use. If IP lookup fails, we print
out some instructions pointing to domifaddr.

That might be too magic, I'll need to think about it more. Maybe if
someone is passing an ssh key we just expect them to know how to get VM
IP addresses. Still it would be annoying to launch virt-viewer or virsh
console in this case when we know they aren't going to be able to log in
that way

> Rather than pointing to my id_rsa.pub most commonly though I would need
> to use the keys known to my SSH agent, because the id_rsa.pub is on my
> laptop, but my VMs are all luanched on remote servers. This would want
> support for
> 
>    --cloud-init ssh-key=agent
> 
> We can implement this by using "ssh-add -L" which displays the public
> keys the agent has.
> 
> In fact the question of "default" behaviour doesn't need solving for
> virt-install as we would always require something to be specified with
> the --cloud-init arg.
> 
> ie, we have a range of possible options
> 
>    --cloud-init passwd=generate
>    --cloud-init passwd=/path/to/file
>    --cloud-init ssh-key=agent
>    --cloud-init ssh-key=/path/to/file
>    --cloud-init root-passwd=generate
>    --cloud-init root-passwd=/path/to/file
>    --cloud-init root-ssh-key=agent
>    --cloud-init root-ssh-key=/path/to/file
> 
> They're not exclusive - we can provide passwords + keys at the same
> time, both for the default user account and for root.
> 

Thanks, these are interesting ideas

> 
>>>> The simplest: use the guest name as the default password. It's not
>>>> hardcoded, it's simple to type. Less safe though because the VM is
>>>> exposed to ssh for that first bootup with a possibly predictable password.
>>>
>>> I would consider any use of a predictable default password to be a
>>> security flaw. Guest name is predictable and visible to any process
>>> on the host so is not safe.
>>>
>>
>> That's fair. But I'm really not into the idea of trying to make ssh work
>> automagically. Providing explicit options to use cloud-init ssh key
>> injection is fine with me though, for the cli at least. UI I'm unsure
>>
>> But for the default, I think that leaves generating a random password
>> for root, similar to what is done with virt-builder. root is not
>> accessible over ssh, so we drop them into 'virsh console' and they can
>> copy/paste it if they want. Then the password is expired and they set
>> their own on first login.
> 
> As above I don't htink there's any notion of "default" behaviour for
> virt-install as the user would always have to pass some values to the
> --cloud-init arg to specify what todo.

I still think it's useful to have bare '--cloud-init' map to some kind
of default behavior that we know works well. That can be
root-passwd=generate and connect to the text console. That gives us
future flexibility to tweak that default behavior, if some distros need
specific config to work out of the box, or if we just want to change it
for any reason

- Cole