VM - a plenty of inactive memory

[Andreas Grimm]

Hello everybody,

i got a weird problem with one of my servers. It's a Intel SR2500AL
with 32GB of RAM.
Looking at the memory usage of the system, something is going totally
wrong. The crucial numbers from /proc/meminfo are:

MemTotal: 33265916 kB
MemFree: 416168 kB
Inactive: 24630428 kB (24GB? whooaaa)

Another system with only 16GB, same amount of users and load, shows a
more normal behaviour:

MemTotal: 16619808 kB
MemFree: 6912676 kB
Inactive: 1774364 kB

Why does the 32GB-System have this plenty of inactive memory. Is there
a way to find out, what the kernel is holding in readiness (that's the
definition of inactive memory afaik)?

OS: SLES 10 SP1
Kernel : 2.6.16.27-0.9-bigsmp

Thanks in advance.

Andreas Grimm

[Johannes Weiner]

Hi,

Post by Andreas Grimm
Hello everybody,
i got a weird problem with one of my servers. It's a Intel SR2500AL
with 32GB of RAM.
Looking at the memory usage of the system, something is going totally
MemTotal: 33265916 kB
MemFree: 416168 kB
Inactive: 24630428 kB (24GB? whooaaa)
Another system with only 16GB, same amount of users and load, shows a
MemTotal: 16619808 kB
MemFree: 6912676 kB
Inactive: 1774364 kB
Why does the 32GB-System have this plenty of inactive memory. Is there
a way to find out, what the kernel is holding in readiness (that's the
definition of inactive memory afaik)?

Inactive pages are marked in use but haven't been touched for some time.
These are candidates for memory reclaiming.

If nothing memory consuming happens, the kswapd should reclaim them back
eventually.

Hannes

[Andreas Grimm]

Hi Johannes,

i know this. But why the kernel locks that memory for a so long time
(2 days now)? Is there a way to enforce the reclaiming? And how can i
find out, which process owns that memory. The problem is, that i can't
accept, that the free memory fell down to 50MB, when i have 24GB in
the nirvana. The system was recently very close to the awkward
situation to swap to disk, and i bet it will do so in the next few
days, because it happened before. Unintelligible, if one got that much
ram.

Bye, Andreas

Post by Johannes Weiner
Hi,

Post by Andreas Grimm
Hello everybody,
i got a weird problem with one of my servers. It's a Intel SR2500AL
with 32GB of RAM.
Looking at the memory usage of the system, something is going totally
MemTotal: 33265916 kB
MemFree: 416168 kB
Inactive: 24630428 kB (24GB? whooaaa)
Another system with only 16GB, same amount of users and load, shows a
MemTotal: 16619808 kB
MemFree: 6912676 kB
Inactive: 1774364 kB
Why does the 32GB-System have this plenty of inactive memory. Is there
a way to find out, what the kernel is holding in readiness (that's the
definition of inactive memory afaik)?

Inactive pages are marked in use but haven't been touched for some time.
These are candidates for memory reclaiming.
If nothing memory consuming happens, the kswapd should reclaim them back
eventually.
Hannes

[Dan Noe]

Post by Andreas Grimm
i know this. But why the kernel locks that memory for a so long time
(2 days now)? Is there a way to enforce the reclaiming? And how can i
find out, which process owns that memory. The problem is, that i can't
accept, that the free memory fell down to 50MB, when i have 24GB in
the nirvana. The system was recently very close to the awkward
situation to swap to disk, and i bet it will do so in the next few
days, because it happened before. Unintelligible, if one got that much
ram.

Have you investigated the sys.vm.swappiness tunable? It allows you to
control how readily the kernel swaps out unused mapped memory. This
seems to be relevant to your issue.

See http://lwn.net/Articles/83588/

Cheers,
Dan

[KOSAKI Motohiro]

Hi

Post by Andreas Grimm
i know this. But why the kernel locks that memory for a so long time
(2 days now)?

maybe, page cache consume it.
it isn't mean that waste memory.

Post by Andreas Grimm
Is there a way to enforce the reclaiming?

following command drop all cache. (but I don't reccomend it)

# echo 3 > /proc/sys/vm/drop_caches

Post by Andreas Grimm
And how can i
find out, which process owns that memory.

please use free command and see second line(+ buffer/cache'ed free mem).

Post by Andreas Grimm
The problem is, that i can't
accept, that the free memory fell down to 50MB, when i have 24GB in
the nirvana. The system was recently very close to the awkward
situation to swap to disk, and i bet it will do so in the next few
days, because it happened before. Unintelligible, if one got that much
ram.

may be, it is not happend forever.

[Andreas Grimm]

Hi,

the output of free:
-/+ buffers/cache: 30718212 2547704
Information in frees man page is very sparse, what's the
interpretation? I will try the drop_cache thing in the night.
Thanks!

Post by KOSAKI Motohiro
Hi

Post by Andreas Grimm
i know this. But why the kernel locks that memory for a so long time
(2 days now)?

maybe, page cache consume it.
it isn't mean that waste memory.

Post by Andreas Grimm
Is there a way to enforce the reclaiming?

following command drop all cache. (but I don't reccomend it)
# echo 3 > /proc/sys/vm/drop_caches

Post by Andreas Grimm
And how can i
find out, which process owns that memory.

please use free command and see second line(+ buffer/cache'ed free mem).

Post by Andreas Grimm
The problem is, that i can't
accept, that the free memory fell down to 50MB, when i have 24GB in
the nirvana. The system was recently very close to the awkward
situation to swap to disk, and i bet it will do so in the next few
days, because it happened before. Unintelligible, if one got that much
ram.

may be, it is not happend forever.

[Andreas Grimm]

Hi Kosaki,

Post by KOSAKI Motohiro
# echo 3 > /proc/sys/vm/drop_caches

that frees exactly the 1,5Gigs in the buffers and the cache. Inactive
memory is still at 25GB.
Any other hint?

Andreas

[Johannes Weiner]

Hi,

Post by Andreas Grimm
Hi Kosaki,

Post by KOSAKI Motohiro
# echo 3 > /proc/sys/vm/drop_caches

that frees exactly the 1,5Gigs in the buffers and the cache. Inactive
memory is still at 25GB.
Any other hint?

Could you supply the rest of /proc/meminfo?

Hannes

[Andreas Grimm]

Hi Johnannes,

of course:
MemTotal: 33265916 kB
MemFree: 438764 kB
Buffers: 98544 kB
Cached: 1837180 kB
SwapCached: 0 kB
Active: 7086300 kB
Inactive: 25168628 kB
HighTotal: 32631216 kB
HighFree: 429252 kB
LowTotal: 634700 kB
LowFree: 9512 kB
SwapTotal: 2104504 kB
SwapFree: 2104504 kB
Dirty: 14480 kB
Writeback: 0 kB
Mapped: 515524 kB
Slab: 506636 kB
CommitLimit: 18737460 kB
Committed_AS: 16388640 kB
PageTables: 37036 kB
VmallocTotal: 112632 kB
VmallocUsed: 15024 kB
VmallocChunk: 95268 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

Here the meminfo from a "normal" system, nearly same amount of users,
and stress, but only 16GB of ram:

MemTotal: 16619808 kB
MemFree: 5497672 kB
Buffers: 422592 kB
Cached: 7018072 kB
SwapCached: 0 kB
Active: 8769608 kB
Inactive: 1979508 kB
HighTotal: 15854036 kB
HighFree: 5484136 kB
LowTotal: 765772 kB
LowFree: 13536 kB
SwapTotal: 4200988 kB
SwapFree: 4200828 kB
Dirty: 16956 kB
Writeback: 0 kB
Mapped: 864284 kB
Slab: 309456 kB
CommitLimit: 12510892 kB
Committed_AS: 14933584 kB
PageTables: 33248 kB
VmallocTotal: 112632 kB
VmallocUsed: 9980 kB
VmallocChunk: 102480 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

Thanks,
Andreas

Post by Johannes Weiner
Hi,

Post by Andreas Grimm
Hi Kosaki,

Post by KOSAKI Motohiro
# echo 3 > /proc/sys/vm/drop_caches

that frees exactly the 1,5Gigs in the buffers and the cache. Inactive
memory is still at 25GB.
Any other hint?

Could you supply the rest of /proc/meminfo?
Hannes

[Peter Zijlstra]

Post by Andreas Grimm
Hi Johannes,
i know this. But why the kernel locks that memory for a so long time
(2 days now)?

It should not be locked (being on the inactive list does not imply that
in any way). Being on the inactive list just means its first in line to
be looked at when memory needs to be reclaimed.

Post by Andreas Grimm
Is there a way to enforce the reclaiming?

Why would you want to do that for? Would you not rather have a copy of
some page in memory than having to go back to disk to fetch it? Free
memory is a waste, better have something in it that might potentially be
used again.

Post by Andreas Grimm
And how can i
find out, which process owns that memory. The problem is, that i can't
accept, that the free memory fell down to 50MB, when i have 24GB in
the nirvana.

See above, free memory is a waste. Really, you don't want 24GB of free
memory.

Post by Andreas Grimm
The system was recently very close to the awkward
situation to swap to disk, and i bet it will do so in the next few
days, because it happened before. Unintelligible, if one got that much
ram.

If it really would have hit swap then it just means you have a _lot_ of
anonymous memory and little page cache pages on that inactive list.
Check Cached and AnonPages.

[Andreas Grimm]

Hi everybody,

i investigated this further. The tunables swappiness, drop_caches etc.
are no options to solve this. The problem is becoming very unpleasent,
because the system isn't able to cache that much:

The 16GB system (expected behaviour):
MemTotal: 16619808 kB
MemFree: 4490032 kB
Cached: 6929448 kB
Inactive: 1670812 kB

The 32GB system :
MemTotal: 33265916 kB
MemFree: 600000 kB
Cached: 1561124 kB
Inactive: 25873128 kB

Don't you think this is insane? Wasting 25GB at the expense of caching
(compare the cached value)? I'm clueless about this issue. Who (which
process) owns that memory? Is there a way to flush this inactive
memory? I assume that this memory hasn't been reclaimed for days,
because the amount of inactive memory is at this level for days now.

All values in /proc/sys/vm are at the defaults again, on both systems.

Any help would be appreciated.

Andreas

Post by Johannes Weiner
Hi,

Post by Andreas Grimm
Hello everybody,
i got a weird problem with one of my servers. It's a Intel SR2500AL
with 32GB of RAM.
Looking at the memory usage of the system, something is going totally
MemTotal: 33265916 kB
MemFree: 416168 kB
Inactive: 24630428 kB (24GB? whooaaa)
Another system with only 16GB, same amount of users and load, shows a
MemTotal: 16619808 kB
MemFree: 6912676 kB
Inactive: 1774364 kB
Why does the 32GB-System have this plenty of inactive memory. Is there
a way to find out, what the kernel is holding in readiness (that's the
definition of inactive memory afaik)?

Inactive pages are marked in use but haven't been touched for some time.
These are candidates for memory reclaiming.
If nothing memory consuming happens, the kswapd should reclaim them back
eventually.
Hannes

[linux-os (Dick Johnson)]

Post by Andreas Grimm
Hi everybody,
i investigated this further. The tunables swappiness, drop_caches etc.
are no options to solve this. The problem is becoming very unpleasent,
MemTotal: 16619808 kB
MemFree: 4490032 kB
Cached: 6929448 kB
Inactive: 1670812 kB
MemTotal: 33265916 kB
MemFree: 600000 kB
Cached: 1561124 kB
Inactive: 25873128 kB
Don't you think this is insane? Wasting 25GB at the expense of caching
(compare the cached value)? I'm clueless about this issue. Who (which
process) owns that memory? Is there a way to flush this inactive
memory? I assume that this memory hasn't been reclaimed for days,
because the amount of inactive memory is at this level for days now.
All values in /proc/sys/vm are at the defaults again, on both systems.
Any help would be appreciated.
Andreas

Wasting 25GB? It looks to me as though the kernel cached
(didn't write pages to disk yet) just about all it needed,
and there is lots of memory that it just doesn't need
--yet! This means that if you have tasks that need memory
in a hurry, they'll get it without any disk accesses.

The kernel doesn't expand to use all the memory just to
spread itself all over the place. It uses what it needs,
caches some buffered data, and keepts track of all the
memory it has. If you want to use more cache, do '-lsR /'.
That'll read every directory into memory. It probably
won't go away until someody needs more memory.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.29 BogoMips).
My book : http://www.AbominableFirebug.com/
_


****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to ***@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

[Robert Hancock]

Post by Andreas Grimm
Hi Johannes,
i know this. But why the kernel locks that memory for a so long time
(2 days now)? Is there a way to enforce the reclaiming? And how can i
find out, which process owns that memory. The problem is, that i can't
accept, that the free memory fell down to 50MB, when i have 24GB in
the nirvana. The system was recently very close to the awkward
situation to swap to disk, and i bet it will do so in the next few
days, because it happened before. Unintelligible, if one got that much
ram.
Bye, Andreas

The memory should not be locked. If the kernel has no reason to reclaim
the memory, it won't, though.