Comments on: Reservoir sampling and performance considerations http://huddledmasses.org/reservoir-sampling-and-performance-considerations/ You can do more than breathe for free... Fri, 03 Feb 2012 03:27:25 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Joel 'Jaykul' Bennett http://huddledmasses.org/reservoir-sampling-and-performance-considerations/comment-page-1/#comment-138675 Joel 'Jaykul' Bennett Wed, 23 Jan 2008 14:49:57 +0000 http://HuddledMasses.org/reservoir-sampling-and-performance-considerations/#comment-138675 Actually, I did run every test in it's own process, sequentially rather than all at once, to make sure they had a situation that was as similar to each other as possible ... I was watching memory use in the Task Manager and checking Working Set, and Peak Working Set, as well as the Private Working Set. In this case (and, I suspect, ain any case like this) the numbers are essentially the same for the Working Set at the end of the script and the Peak Working Set -- at most a few Kb difference on a process using over 180Mb -- I just went with what I thought was the simple number because the differences were so small. Actually, it's an interesting side effect of non-deterministic garbage collection (and having plenty of RAM) that it takes a couple of minutes for the Working Set memory to start coming down from it's peak unless you start thrashing RAM with some other application. Actually, I did run every test in it’s own process, sequentially rather than all at once, to make sure they had a situation that was as similar to each other as possible … I was watching memory use in the Task Manager and checking Working Set, and Peak Working Set, as well as the Private Working Set.

In this case (and, I suspect, ain any case like this) the numbers are essentially the same for the Working Set at the end of the script and the Peak Working Set — at most a few Kb difference on a process using over 180Mb — I just went with what I thought was the simple number because the differences were so small.

Actually, it’s an interesting side effect of non-deterministic garbage collection (and having plenty of RAM) that it takes a couple of minutes for the Working Set memory to start coming down from it’s peak unless you start thrashing RAM with some other application.

]]> By: Dmitry Sotnikov http://huddledmasses.org/reservoir-sampling-and-performance-considerations/comment-page-1/#comment-138269 Dmitry Sotnikov Mon, 21 Jan 2008 14:36:54 +0000 http://HuddledMasses.org/reservoir-sampling-and-performance-considerations/#comment-138269 Quick question Joel, why not use PeakWorkingSet rather than just WorkingSet when detecting the memory consumption? This would show peak consumption regardless of how Windows frees up memory, right? Of course you'll need to start the tests in different processes to make sure that you don't get the same peak over and over. ;) Just my two cents. Dmitry Quick question Joel, why not use PeakWorkingSet rather than just WorkingSet when detecting the memory consumption? This would show peak consumption regardless of how Windows frees up memory, right?

Of course you’ll need to start the tests in different processes to make sure that you don’t get the same peak over and over. ;)

Just my two cents.

Dmitry

]]>