I am running Debian Wheezy with kernel 3.0.0 on a thin client turned into a NAS. The device has only 128MB of RAM so I also use a 256MB of swap just in case. Upon boot the memory looks like that:
total used free shared buffers cached Mem: 105 24 81 0 0 7 -/+ buffers/cache: 16 89 Swap: 243 0 243 When I start to copy a file from a samba server on this device, 'cached' quickly starts to grow. As soon as cached fills up almost all of the memory (~90MB) three things happen: kswapd kicks in, smbd starts to be swapped away and the file copy performance drops.
edit: I have confirmed this with iostat. As soon as kswapd kicks in, w_await jumps from 0 to ~500 and %util to 99%.
Interestingly, this happens regardless of the setting of vm.swappiness and vm.vfs_cache_pressure (e.g. swappiness 0 or 1 and vfs_cache_pressure 500)
In my scenario, caching 90MBs of a file read from over 2TB attached storage is quite pointless. What's even worse, disk cache is given priority over application memory.
If I clear the cache during the copy process to prevent it from filling up the memory (by repeatedly doing echo 1 > /proc/sys/vm/drop_caches) I suffer from no performance drop. Hardly a solution to the problem, though.
It appears that this particular problem is known in the mysql community (here or here ) but I can't help thinking that it is me who is doing something wrong.
TL;DR can't make overzealous disk caching go away.
Please advise.
edit: came across this, seems abandoned
edit2: did some more reading on the issue, I suppose that patching samba (or perhaps writing a vfs module?) to use POSIX_FADV_NOREUSE could work, assuming it's implemented already.
However, I still don't understand why setting swappiness doesn't seem to work and the kernel happily swaps out running processes.
[link] [9 comments]