[virt-tools-list] setup.py and packaging for distribution

[Cole Robinson]

On 03/27/2013 09:50 AM, Gene Czarcinski wrote:
> On 03/26/2013 05:48 PM, Cole Robinson wrote:
>> On 03/26/2013 11:59 AM, Gene Czarcinski wrote:
>>> OK, I have been looking into how packaging and distribution works for example
>>> of when you want to get a snapshot of your development.
>>>
>>> 1. python ./setup.py build seems to work
>>> 2. python ./setup.py rpm fails looking for some patch file (?) but does create
>>> a tarball in ./dist/
>> 'python setup.py rpm' works for me... can you show the exact error?
> I found the problem.
> 
> I have my own rpmbuild tree under my home directory and a tailored .rpmacros. 
> When I temporarily renamed ".rpmmacros", the "python setup.py rpm" worked.
> 
> I consider this a but with setup.py.
> 
> I was also surprised to find that setup.py had created its own rpmbuild tree
> under my home directory.  This was a very unexpected and undesirable side
> effect! ... another "bug" in my opinion!
> 
> That said, I really have no need or want to have the "noarch binary" rpms
> produced directly from the git clone repository.  What I really what is a
> tarball with a valid spec file (that works with rpmbuild -ts).  There may
> exist others but I have yet to encounter a project (with a git repository)
> which creates the binary rpms directly from the git.  I have my own rpmbuild
> tree and mock to create the binary rpms and I suspect that this is the case
> with most others.
> 

Take a look at the rpm command in setup.py. All it does is

python setup.py sdist
rpmbuild -ta dist/virt-manager-$VERSION.tar.gz

So not sure why it isn't picking up your rpmmacro config


> More below.
>>
>>> 3. python ./setup.py sdist works and creates a tarball in ./dist/
>>> 4. in both of the above cases the tarball is completely unsatisfactory!  They
>>> contain everything in the directory.
>>>
>>> Suggestions:
>>>
>>> 1. Use "git archive" since it will only gather files that branch currently
>>> manages
>>>
>>> 2. This is a snapshot.  The tarball name should be something like
>>> virt-manager-git20130326.tar.gz
>>>
>>> 3.  Add more tags.  I know you have the RELEASE-0.9.4-1, etc. tags but you
>>> need some additional one for things such as release-candidates,
>>> testing-candidates, etc.  For example, maybe the current gtk3.2 branch should
>>> have a tag something like gtk3.2-t1 [test 1].
>>>
>>> Why the above suggestions:
>>> 1. From the git-scribe man page:
>>>>         The command finds the most recent tag that is reachable from a
>>>> commit.
>>>>         If the tag points to the commit, then only the tag is shown.
>>>> Otherwise,
>>>>         it suffixes the tag name with the number of additional commits on top
>>>>         of the tagged object and the abbreviated object name of the most
>>>> recent
>>>>         commit.
>>> 2. Use this tag-info to set the virt-manager "version"
>>>
>>> 3. Bring back a bit of the v-m.spec.in file.  Set the Version as the "date"
>>> from one specified on the tarball using "sed".  Then you can add the tag-info
>>> as the "release" or "extra-release" using "sed". The output of "sed" should be
>>> the .spec file which is NOT managed by git but the spec.in is.
>>>
>>> 4. When you run "git archive", create the tarball (.tar file) but not
>>> compressed (.gz).  Now run "tar -rf" to add the spec file into the tarball.
>>> Now you can run gzip to compress the tarball.
>>>
>>> 5. Skip creating rpms ... source or "binary".  Create a mechanism to produce a
>>> snapshot tarball.  To get an rpm, rpmbuild -ts works just fine.
>>>
>>> I have implemented something like the above as a bash script for another
>>> project.  If you want, I will take a shot at implementing this as part of
>>> setup.py ... just point me where to look and where to change things for the
>>> virt-manager --version.
>>>
>> I started replying to things individually here but I got confused: what's the
>> end goal exactly? 'python setup.py rpm' generates a uniquely named rpm based
>> on the current commit? Who does this benefit?
>>
>> If it's beneficial, I'd rather just make it a new subcommand like 'python
>> setup.py gitrpm'. You could likely overwrite the RPM version values with
>> rpmbuild options, rather than revive virt-manager.spec.in mangling which I'd
>> like to avoid if possible.
>>
>> But really if this is about using RPMs as part of the development process, I'd
>> rather spend effort on making sure running virt-manager from the source tree
>> works.
>>
>> Thanks,
>> Cole
>>
>>
> OK, let me try again to explain my position.
> 
> 1. During development in a git-clone repository I will generally have some
> non-git-managed files such as patch files or directories with patch files. 
> Unfortunately, "python setup.py sdist" includes all of this in the tarball it
> creates.  This is why I am suggesting the use of git-archive instead of tar to
> create the archive image. The resulting tarball only includes the "offical"
> git-managed files which is what I believe should be in a distribution
> (snapshoot or release) tarball.  Yes, ".gitignore" is included but that is a
> nit. This tarball should be sufficient to be used as a source for "rpmbuild
> -ts" or, when expanded, to build and install the product. If I really wanted a
> tarball which included everything, I can easily create that.
> 

I agree with this, the sdist bit just grabbing everything is not ideal.
Finding a way to use git archive is one option, though we would also want to
error out if the working directory is dirty, since someone may try building an
RPM with unapplied local changes, and archive won't pick them up. And git
archive doesn't work from a tarball.

> 2. The current sdist/rpm tarball uses "0.9.4-1" to version the tarball and
> rpms.  This is useless since this is this same "version" will be used for
> tarballs created in the master branch or the gtk3.2 branch.
> 

Sure, that's temporary pain though. And 'make dist' has the same behavior in
every autotools project out there so there's plenty of precedent.

> 3. Alternative version: Use something like "git20130327" for the version. 
> Better, at lease I can differentiate between a tarball produced last week and
> one produced today.  Better, but not really a winner.
> 
> 4. A really good alternative is to use "git describe --tags" produce the
> version.  Two examples:
> 
>       $ git checkout -b gc-0.9.4-1 RELEASE-0.9.4-1
>       $ git describe --tags
>       RELEASE-0.9.4-1
> 
>      $ git checkout gtk3.2
>      $ git describe --tags
>     RELEASE-0.9.4-1-86-gc61d9af
> 
> You can see that you get unique identifiers.  If this is used as the
> "version", then it will not matter when you create the tarball, there will be
> a different version if the contents of the tarball is different but the same
> version if they are the same.  I like this!
> 
> Of course, before doing the archive (creating the tarball), you must make sure
> there are no uncommitted changes.
> 
> 5.  So, now I have produced a good tarball with a version based on content (if
> the content is different, the the version will be different). But there are
> two other things that need to use this new version: the value given when the
> user enters "virt-manager --version" and the spec file.
> 
> 6.  I am not sure where/what to change but the effect result is to use the
> value returned by "virt-manager --version" to be  the value from "git describe
> --tags".  The basic place is line 44 in cliconfig.py but the is a cli.cfg
> referenced and that is likely where this should go.  I am just not sure where
> is is or when this cfg file is created.
> 
> 
> 7. For the spec file, I want to slightly revert to having a spec.in file which
> will be managed by git and a spec file which will not be managed by git.  Then
> setup.py needs to do "sed" to replace /@VERSION@/ with the result of "git
> describe --tags".  I also suggest that the release be set to "0" since most of
> these will be snapshots.  Once the updated spec file is created, it can be
> added to the tarball (tar -rf) and the new tarball compressed with gzip.
> 

Changing 'make rpm' to have a unique release number is fine. But I don't want
to revive virt-manager.spec.in if it can be avoided. You should be able to do
it all from setup.py: generate the new release number, then invoke rpmbuild
with --define '_release $RELEASENUM'

But I don't see the value in altering virt-manager --version to spit out a git
commit.

> Have I explained this any better?  I hope so.
> 
> BTW, there is still some clutter left with the remove of autotools which
> should also be deleted (e.g., autobuild.sh).  Do you want me to create patches
> for that or is that already being handled?

autobuild.sh is not autotools related, it's about a separate project for doing
continuous builds. It just hasn't been adapted to the new build process yet.

Thanks,
Cole