[virt-tools-list] [PATCH virtio-win-pkg-scripts v3 0/2] helpers to standardize driver directory layout

[Cole Robinson]

On 02/08/2016 09:23 AM, Roman Kagan wrote:
> On Sat, Feb 06, 2016 at 06:20:42PM -0500, Cole Robinson wrote:
>> Sorry for the delay, been caught up in other stuff. I'll be focusing on
>> virtio-win stuff this coming week though
>>
>> I've committed these patches now. I added a few commits on top to make my
>> pylint[1] setup happy. No functional changes.
> 
> Great, thanks.
> I've accumulated a couple of fixes and enhancements in the meantime;
> I'll rebase and post them soon.
> 

So I'm full of crap and didn't actually touch virtio-win after sending that
mail, except trying to page this stuff back into memory. But it's my main
priority for the next bit of time.

>> I'm realizing now that the input bits to make-driver-dir.py
>> (virtio-win/qxl/qemu-ga windows builders from RH's internal build system)
>> aren't published anywhere so there isn't any way for you to reproduce the full
>> workflow for the public RPM. I'll work on getting those mirrored on
>> fedorapeople.org
> 
> Actually we started putting together our stuff, too, and encountered a
> number of issues I'd be interested to discuss.
> 

Can I ask what you plan to do with virtio-win RPM/ISO on your side? I'm trying
to spec out in my head all the bits that need to change, so understanding what
you're needs are will help my get things straight.

Are you just interested in extending the RHEL RPM, or the public one as well?
What types of bits are you adding? drivers, agents, etc

> 1) since the drivers can only be built on a Windows machine, there's a
>    need to define the artifact(s) produced on a Windows machine and used
>    as source(s) for rpmbuild on a Linux machine (koji).
> 
>    We thought the most natural responsibility split (as applied to
>    kvm-guest-drivers-windows) is to collect the stuff in "Install"
>    subdirectories of every driver upon buildAll.bat execution in a
>    single zip archive and use it as that artifact; the rest will be
>    taken care of from under rpmbuild on a Linux machine.  One benefit is
>    that it's going to be easy to intervene manually, e.g.  when doing
>    signatures or whatever else.
> 
>    This differs from what's currently in virtio-win srpm where the
>    drivers are received in a tarball whose contents was obviously
>    subject to certain manipulations after building on Windows.
> 

I'm not really familiar with the windows build process, so I'm not sure I
follow all of that. But the idea of pushing some of this rework down into the
kvm-guest-drivers-windows build infrastructure sounds like a good one to me,
since as you say that's the root source.

Regarding handling everything in rpmbuild, I'm not entirely sure how I feel.
Doing all the processing at rpmbuild time is probably okay as long as the
logic doesn't live entirely in the .spec file, and exists as external scripts
that can be tested independently of actually triggering 'rpmbuild'. Then say
virtio-win-pkg-scripts-gitXXX.tar.gz should just be about SOURCE in the spec
file. That would certainly streamline the process a bit and make it more clear
exactly what is going on.

Though on the RHEL side that will be a bit difficult, because the shipped
drivers are not composed of output from a single virtio-win build, but from
_many_ virtio-win builds.

For example, the latest winxp viostor WHQL submission might have used drivers
from internal build virtio-win-prewhql-26, and the latest win7 vioser WHQL
submission may have used bits from virtio-win-prewhql-52.

Now a new internal build pops up, virtio-win-prewhql-78. It's used to feed
win10 WHQL balloon submission. That build may ship have a newer winxp viostor
driver and a newer win7 vioser driver. But we aren't going to ship them,
because we are tied to the content that was used for the last WHQL submission.

Basically we have an internal git repo that tracks all whql submissions, their
associated prewhql input (which is raw kvm-guest-windows-drivers build
output), and we use the mappings to create a frankenstein symlink tree that
matches the driver layout you see on the virtio-win iso, or from upstream
make-driver-dir.py output. The public and RHEL build process then sync's up
the rest of the way.

So not sure how to handle that in rpmbuild without jumping through more hoops.

That said, a fix to sync up the processes a bit more is to generate a symlink
tree that matches the same layout as the kvm-guest-windows-drivers build
output. Then the public and RHEL build processes would be identical, except
for RHEL's prep step of splicing in the WHQL output.

> 
> 2) build scripts for kvm-guest-drivers-windows (haven't checked qxl yet)
>    don't adhere to the conventions assumed by the scripts I posted.
>    E.g. here's the relevant excerpt from Balloon/sys/packOne.bat:

Snipping this since I don't know enough to comment

> 
> 
> 3) current virtio-win package includes several loosely related packages:
>    virtio drivers, qxl, qga; we add our own, too.  They have independent
>    release cycles and build procedures (e.g. qga builds nicely with
>    mingw on Linux and can be turned into a normal rpm suitable for
>    shipping in e.g. Fedora)
> 

Certainly for the public virtio-win distribution there's no good reason to be
dependent on internal RH builds for qemu-guest-agent. It only works like that
because I haven't gotten fixing it.

There's a question then of where we get our qemu-guest-agent build. I don't
know if having a mingw-qemu-guest-agent RPM in fedora would be acceptable,
shipping a windows binary like that. It could be a separate RPM that is
distributed in the virtio-win.repo but it'll take a bit of scripting to set up.

But while it seems feasible for the public distribution, it's less so for
RHEL. RHEL doesn't have mingw and likely never will. The qemu-ga builds we use
in RHEL are actually performed on a windows machine, so aren't distributed in
an RPM. So this bit may end up as a diverging point for RHEL vs public/fedora
virtio-win RPM building

>    Therefore we're considering splitting the rpm into several
>    independent ones with the stuff installed onto the host filesystem as
>    a directory structure; the iso and floppy images would be generated
>    from that right on the host (e.g. by rpm triggers or by a tool like
>    supermin) rather than brought in pre-generated.

Hmm now that's an interesting idea! A little scary though :) I wouldn't want
to have to install the RPMs locally just to get an iso or vfd image to play
with or test. And not sure how that will work for RHEL if we don't end up
RPMifying qemu-ga (or eventually spice agent) for example

Certainly splitting the RPM at least into sub-RPMs is a no brainer, though I'd
need to check with RHEL consumers and documentation to make sure any
dependencies are updated. Maybe 'virtio-win' becomes a meta package that pulls
in virtio-win-drivers (/usr/share/virtio-win/drivers) and virtio-win-media
(iso and vfds), that would probably make things back compat. But that doesn't
help with the separate release cycle point you mention.

Thanks,
Cole