[virt-tools-list] [virt-convert] ec2 export module

[Joey Boggs]

Hi Joey,
> Thanks for the patch. However, I think it needs a lot of cleanup to even
> facilitate a good review. Some overall comments:
>
> - There are very few comments in the code, it is pretty difficult to
> decode what exactly is going on. A comment before every step in
> parsers/ec2.py would help, as well as comments for each function in the
> ec2 helper files. Also, a paragraph explaining how exactly conversion to
> ec2 works from start to finish (either in the code or when you repost
> the patch), since this is quite different from the other virtconv examples.
>
> - Please run this patch through 'python setup.py check' in the virtinst
> repo directory and fix all ec2 related errors (you'll need to install
> pylint).
>
> - We cannot call 'fail' from virtconv. Everything in the virtconv
> directory is an API: where you use 'fail', we should be raising an
> exception and letting virt-convert handle it. Same goes for using
> 'print', use either logging.info or logging.debug please.
>
> - Please move the virtconv/ec2* files to something like
> virtconv/parsers/ec2helpers/*. virtconv/parsers/ec2.py is fine where it is.
>
> - The code needs to be mindful of cleanup. There seems to be a lot of
> mounting, downloading, and directory creation going on, and no attempt
> at error recovery cleanup.
>
> - On a basic smoketest, this errored immediately with invalid imports
> (virtconv.cli and virtconv.rpmcheck). Please test your code before
> submitting.
>
> When you repost, an example virt-convert invocation with the command
> line output would also be helpful.
>
> Thanks,
> Cole
>   

Round 2

- ran setup.py check and cleaned up the code alot
- Added exceptions and logging.error in place of fail() sections
- moved helper files under virtconv/parsers/ec2helpers

Description of how this module works:

The EC2 module expects a single disk file from any other input 
format(vmx/virt-image), more than 1 with throw an error. Adding support 
for 2 disks should work fine out of the box but I've never tested it. 
Once the base module is in I have a few more items to add, multiple 
disks is one of them.
- the input image is attached to a loop device and kpartx is run to 
create device maps for each partition.
- e2label is ran to grab the file system labels on the input disk
- the list is sorted and begins mounting those file systems in the 
correct order to a temp directory
- du -sh is ran on the temp directory and we add 30% free space to the 
minimum amount of space required
- new file system is created using above calculation
- rsync copies the fully mounted filesystems to the new single loopback 
filesystem.
- EC2 standardizations added to the configuration
- eth0
- rc.local (to download public key of user on boot, and ami tools, to 
allow rebundling)
- kernel modules (pulls in the latest matching Fedora kernel that EC2 
support, for net/block device modules)
- api tools ( to allow rebundling and upload to S3)
- fstab (rootfs/swap/S3)

Once the configuration is done, we unmount the new filesystem and all of 
the input disk image partitions.

Normal operation of virt-convert continues
- We skip the disk format convert since we already accomplished this above
- New disk image is moved into the correct location for the user to take 
and bundle with the EC2's ami-tools or others.


-----------------
Sample output

The output is mostly quiet since it looks like logging.info isn't sent out.
-----------------

[jboggs at localhost jeos]$ sudo virt-convert jeos.xml -o ec2
Generating output in 'ec2' format to jeos/
469+0 records in
469+0 records out
491782144 bytes (492 MB) copied, 8.93075 s, 55.1 MB/s
mke2fs 1.41.4 (27-Jan-2009)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
120360 inodes, 480256 blocks
24012 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
59 block groups
8192 blocks per group, 8192 fragments per group
2040 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8027k 100 8027k 0 0 339k 0 0:00:23 0:00:23 --:--:-- 115k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16.5M 100 16.5M 0 0 1200k 0 0:00:14 0:00:14 --:--:-- 1220k
Done.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ec2-virt-convert-module
URL: <http://listman.redhat.com/archives/virt-tools-list/attachments/20091008/b591d9a6/attachment.ksh>