[virt-tools-list] How does the memory usage calculated?

[Cole Robinson]

On 11/18/2015 10:29 PM, 张强 wrote:
> Hi Cole, Richard,
> 
> Thanks for your answers.
> 
> Cole, I looked at your commit, I think we should increase required libvirt-python version to use 
> SetMemoryStatsPeriod. 
> 
> This function seems fairly new, my libvirt is 0.10.x, and still it doesn't support this function, let 
> alone the 0.4.0 we listed in pre-requisite in the INSTALL file... I think adding install_requires to 
> setup.py is a good idea.
> 

Notice the added check to support.py in that commit. We won't use the API if
the unless the following bits support it: libvirt daemon version,
libvirt-python, hypervisor.

I'm guessing we don't work well anymore with libvirt-python-0.4.0 but we
definitely will work fine with libvirt less than 1.1.1 where this API was
introduced

- Cole

> 
> -----邮件原件-----
> 发件人: Cole Robinson [mailto:crobinso at redhat.com] 
> 发送时间: 2015年11月19日 10:26
> 收件人: Richard W.M. Jones; 张强
> 抄送: virt-tools-list at redhat.com
> 主题: Re: [virt-tools-list] How does the memory usage calculated?
> 
> On 11/18/2015 09:54 AM, Cole Robinson wrote:
>> On 11/18/2015 09:44 AM, Richard W.M. Jones wrote:
>>> On Wed, Nov 18, 2015 at 01:04:32PM +0000, 张强 wrote:
>>>> Hi all,
>>>> I’m studying the source code, and I saw this in domain.py around line 164:
>>>> …
>>>>         curmem = stats['rss']
>>>>         totalmem = stats['actual']
>>>> except libvirt.libvirtError, err:
>>>>     logging.error("Error reading mem stats: %s", err)
>>>>
>>>> pcentCurrMem = curmem * 100.0 / totalmem pcentCurrMem = max(0.0, 
>>>> min(pcentCurrMem, 100.0)) …
>>>>
>>>> But using this algorithm, I’m always getting results that are greater than 100:
>>>>>>> dom.memoryStats()
>>>> {'actual': 16777216L, 'rss': 17062900L}
>>>>
>>>> 17062900 * 100 / 16777216 = 101.7028…
>>>>
>>>> Is this normal?
>>>
>>> The VIR_DOMAIN_MEMORY_STAT_RSS statistic returned for qemu/KVM guests 
>>> is the resident set size (RSS) of the qemu process.  Of course that 
>>> includes all the overhead of qemu, such as host-side structures used 
>>> when emulating devices.  Plus the guest RAM, which is just a regular 
>>> malloc-style allocation inside qemu and so also contributes to the RSS.
>>>
>>> The 'actual' field seems to come from the libvirt 
>>> VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON statistic which is found by 
>>> sending the query-balloon monitor command to the guest.  The returned 
>>> value is the guest's get_current_ram_size() (or less if the balloon 
>>> is "inflated", but for the majority of guests the balloon is never used).
>>>
>>> Given all that, it seems reasonable that rss > actual, and so you'd 
>>> get a number > 100%.  Sometimes.  It also seems to me that if the 
>>> guest RAM had been allocated but not accessed, you might see rss < actual.
>>>
>>> rss / actual seems like a fairly meaningless number from a 
>>> virt-manager point of view.
>>>
>>
>> Yeah I think this calculation is leftover from when virt-manager's 
>> memory reporting just tracked memory ballooning, so pcentCurrMem was 
>> <currentMemory> / <memory>. I'll look into cleaning it up
>>
> 
> We still need to do the 'rss / actual' allocation to get meaningful memory stats from the lxc driver, but I pushed a commit to virt-manager.git now that uses qemu's proper guest coordinated memory stats. In order to get these, we need to call DomainSetMemoryStatsInterval so qemu actually does the polling for us
> 
> commit e68efe810375826a0e72864f4e60b3ee3e0d9bb8
> Author: Cole Robinson <crobinso at redhat.com>
> Date:   Wed Nov 18 21:17:22 2015 -0500
> 
>     domain: Use setMemoryStatsPeriod for QEMU
> 
>     Since that's what is required to get cooperative memory stats from
>     the guest balloon driver.
> 
> 
> - Cole
>