bugThe GNU Hurd - Bugs: bug #28730, Bad Mach object cache reuse

 
 

bug #28730: Bad Mach object cache reuse

Submitter:  Samuel Thibault <sthibaul>
Submitted:  Sun 24 Jan 2010 10:28:15 PM UTC
   
 
Category:  Meta Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Originator Name: 
Open/Closed:  Open Reproducibility:  None
Size (loc):  None Planned Release:  None
Effort:  0.00
Wiki-like text discussion box: 

Pagers only flush pages from their memory objects in special situations like when truncating/removing a node, to be able to use them as cache. So, for most cases, resident_page_count will be > 0.

* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 05 Sep 2011 11:38:12 PM UTC, comment #2: 

Here is a patch I had been experimenting with, which should simply drop
the limit, and let the pageout daemon eliminate objects as their pages
get dropped. It however leads Debian buildds to run out of memory,
apparently the daemon does not drop pages fast enough?


(file #23951)

Samuel Thibault <sthibaul>
Group administrator
Fri 30 Apr 2010 12:03:13 AM UTC, comment #1: 

I think we should replace the cache system for memory objects (at least for the ones with external pagers). How about something like this:

  • GNU Mach
    • Get rid of vm_object_cached_*. If an object can persist, it'll persist forever (or until a pager changes its attributes).
    • Separate pages of memory objects with external pagers into a different queue. Treat it like cache. If we want to free some, ask the translator (by using one of its pagers) to unallocate some objects (translators can select the ones they want to, Mach will only give them a hint on how many space should be freed). This requires a new interface.
    • (Optional) Maintain cached pages statistics per task (pager's owner). Use it as foundation for cache quotas.


  • Hurd
    • Keep track of every action that could alter the contents of a pager. Take special attention with mmap(). Use a lru queue with pointers to the pagers. For everyone of those actions, mark the object as dirty and update queue.
    • Sync only objects marked as dirty.
    • When GNU Mask ask us to unallocate some objects, start running through the queue until the hint provided by the kernel is archieved.


Sergio López <k0ro>
Group Member
Sun 24 Jan 2010 10:28:15 PM UTC, original submission:  

/var/tmp/hurd-20090404 holds 4142 files totalling 77MB
/var/tmp/sudo* holds 434 files totalling 26MB

vm_object_cached_max is set to 4000 in my gnumach, thus not enough for
the whole hurd-20090404

I have 256MB memory, thus plenty for cache

$ time rgrep foobarbaz /var/tmp/hurd-20090404
1m5
$ time rgrep foobarbaz /var/tmp/sudo*
15s
$ time rgrep foobarbaz /var/tmp/sudo*
10s
$ time rgrep foobarbaz /var/tmp/sudo*
8s
$ time rgrep foobarbaz /var/tmp/sudo*
5s
$ time rgrep foobarbaz /var/tmp/sudo*
4s
$ reboot

$ time rgrep foobarbaz /var/tmp/sudo*
15s
$ time rgrep foobarbaz /var/tmp/sudo*
4s

This means filling the cache with hurd-20090404 first gets in the way
of getting sudo* into the cache.

I've dug a bit, the problem lies in the periodic sync of ext2fs. If I
pass --sync=60 to ext2fs, and then retry

$ time rgrep foobarbaz /var/tmp/hurd-20090404
1m5
$ ... wait for ext2fs sync to happen, then as soon as it's over,
$ time rgrep foobarbaz /var/tmp/sudo*
15s
$ time rgrep foobarbaz /var/tmp/sudo*
4s

What I can notice is that on ext2fs sync, all its cached objects get
out of the object cache (for the sync) and then put again (probably
because of the call to memory_object_lock_request()). In the first
scenario, it means the following happening on the object cache (put in
brackets, objects on the left were queued first).

[]
$ time rgrep foobarbaz /var/tmp/hurd-20090404
[ ---------------- hurd-20090404 objs ------------------- ]
$ time rgrep foobarbaz /var/tmp/sudo*
[ hurd-20090404 objs ------------------------ | sudo objs ]
# ext2fs sync
[ ------ mixture of hurd-20090404 objs and sudo objs ---- ]
# rgrep continues, new sudo objs push the mixture out, and some sudo
# objs get dropped
[ mixture of hurd-20090404 objs and sudo objs | sudo objs ]
# ext2fs sync
[ ------ mixture of hurd-20090404 objs and sudo objs ---- ]
[ mixture of hurd-20090404 objs and sudo objs | sudo objs ]
15s

I.e. the ext2fs sync mixes old useless objects with sudo objs,
according to hash functions (see diskfs_node_iterate() or
ports_bucket_iterate()), and in the end a lot of the sudo objs have
been pushed out and dropped from the cache, which is unfortunate. When
raising the period to 60s, the rgrep has the time to finish before its
sudo objects gets scrambled with the hurd objects, and then cache hit
can happen.

To summarize, the issue is that write_all_disknodes() touches the
cached objects too heavily so that GNU Mach believes they are being
used, while they're not.  Raising vm_object_cached_max is not a
solution since that won't prevent the scrambling, just reduce its
effect.  Tinkering with memory_object_lock_request() is not so easy
either, the ground issue being that ext2fs issues syncs for all objects
in a basically random order.  Could there be a way for ext2fs to know
which objects need a sync?

Samuel Thibault <sthibaul>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #23951:  patch-cache added by sthibaul (2KiB - application/octet-stream - experimental patch to simply drop the limit)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by k0ro (Posted a comment)
  • -email is unavailable- added by sthibaul (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-09-07 k0ro Wiki-like text discussion box Pagers only flush pages from their memory objects in special situations like when truncating/removing a node, to be able to use them as cache. So, for most cases, resident_page_count will be > 0.
    2011-09-05 sthibaul Attached File- Added patch-cache, #23951

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code