• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7848 for generic doc work for TWiki-6.1.1. Use View topic Item7851 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

OK. Some results. And they surprise me a lot.

So I was again testing with a very simple page using pattern skin.

I measure with

ab -n 10 http://merlin.lavrsen.dk/twiki/bin/view/Motion/TimBernersLee (Cairo)

and

ab -n 10 http://merlin.lavrsen.dk/dakar/bin/view/Motion/TimBernersLee

You can see the same page here on my externally available server.

http://www.lavrsen.dk/twiki/bin/view/Motion/TimBernersLee

Skin is Pattern skin also in Dakar (not available to the public). I am using the same Motion web (copied).

1.8 GHz machine with Fedora 4, Apache 2.0.55, mod_perl 2.01 and CPAN CGI::Session 4.03 from 2005-09-16 downloaded from CPAN.

Cairo (no mod_perl)

  • 770 ms at 10 session files in /tmp
  • 774 ms at 1070 session files in /tmp
Note. Hardly any change !!

Dakar 7311 without mod_perl

  • 875 ms at ~10 session files in /tmp
  • 1130 ms at 1050 session files in /tmp
Now many session files cost performance

Dakar 7311 with mod_perl

  • 294 ms at ~10 session files in /tmp
  • 528 ms at 1050 session files in /tmp
Sessions files cost performance also here

On my Motion TWiki I have 600-800 unique visitors per day. I get 2000-4000 session ID files per day. The search engines produces quite many of these (no cookies). So it is worth finding out why it cost more and more per session file in Dakar and not in Cairo.

-- KennethLavrsen - 04 Nov 2005


I know CC can use the classic skin results as well.

Cairo with CLASSIC SKIN

  • 689 ms at ~10 session files
  • 687 ms at 1070 session files

Dakar 7311 with mod_perl and CLASSIC SKIN

  • 152 ms with 20 session files
  • 385 ms with 1050 session files

Dakar 7311 without mod_perl and CLASSIC SKIN

  • 708 ms with ~10 session files
  • 977 ms with 1050 session files

Note: With classic and no session files. 689 vs 708. Almost same performance.

So if you can figure out the Session part and we can get a simplified Pattern Skin as default we are back at Cairo performance.

-- KennethLavrsen - 04 Nov 2005

Some notes:

  1. ab is very inaccurate. Running ab -n 10 once is no good - you have to run it at least 5 times, because of variance.
  2. the only functional difference between Cairo and Dakar session handling is the fact that Dakar expires dead sessions. Perhaps this is the cause of the speed difference, though it is only supposed to happen after the query has already been responded to.
  3. Because ab doesn't support cookies, each request sent by ab will be unique and will generate a new session. To avoid this effect, enable MapIP2SID when running ab benchmarks.

CC

  • I do run ab round 5 times when I report my numbers and use the fastest number assuming that the higher numbers comes from misc other things loading the CPU. I have built a special test machine for my testing of Dakar. It does nothing else. Benchmarking on my production machines is worthless. I get consistant results on my test machine. That is how I noticed that each time I ran ab -n 10 the time was a few milliseconds longer. When I run now I delete sessions files so I can comparable results. And the results are very consistant. But I always run ab - n 10 some times to verify that I get consistant results. And I go back between Cairo and Dakar to ensure that my results are repeatable. I would say that my numbers are repeatable by +/- 20 ms. I am a hardware engineer (closet software engineer) so I am used to measuring stuff and checking repeatability.
  • So maybe the expiry of sessions should be something that either
    • Gets rewritten to work better.
    • Simple change so one can disable this and purge old sessions using a cron job like you had to do with Cairo. That actually works well but is less user friendly. Does the value 0 disable the expiry feature?
  • I understand why the session files are created by ab. The MapIP2SID is not very relevant. AT least not for me. Linking sessions to IP addresses is not a good solution because so many users are sharing IP addresses through NAT routers. On my Motion TWiki I get approx 2000 new sessions files per 24 hours. So with a 24 hour expiry that makes Dakar half the speed of Cairo because of this detail - so it is worth doing something about. The many session files are not a product of testing with ab. They are part of real life. And with a TWiki installation more popular than my little Motion project - you will maybe have 10000 session files - even with an 8 hour expiry. Quite many of the session files are created by search engines and misc email harvesters, hacker tools etc.
    • You are missing the point; use MapIP2SID (literally: "map IP address to session ID") for benchmarking, because it re-uses the existing session without having to use a cookie. You can still use cookies with the production install.

I am actually very positive now with respect to Dakar. This session issue is for sure something that can be fixed. If nothing else it can be brought back to work like the original SessionPlugin. And by simplifying a few things related to the default Pattern Skin (something that someone else than CC can work on - including myself) we are back at Cairo performance which is not impressive but acceptable. That is what my last numbers showed. And for those that can run mod-perl (I will for sure) TWiki is almost a speed devil with this session thing fixed.

  • Well, that's encouraging, thank you - I was starting to get a bit despondent.

I tried the obvious, and forked a subprocess to do the session expiry. My platform is just too variable to use for meaningful benchmarking, so I present the patch here for you to try. Please let me know if it makes any difference.

Index: Client.pm
===================================================================
--- Client.pm   (revision 7321)
+++ Client.pm   (working copy)
@@ -261,7 +261,16 @@
     $this->{cgisession}->flush()
       if $this->{cgisession}; # just to make sure ...

-    _expireDeadSessions();
+    # NOTE: Beginning with v5.6.0, Perl will attempt to flush
+    # all files opened for output before forking the
+    # child process, so the network output should be flushed
+    # even before session cleanup starts.
+    my $child = fork();
+    if( defined $child && $child == 0 ) {
+        # Child process
+        _expireDeadSessions();
+        exit(0);
+    }
 }
Note that while output streams are flushed, it's not clear whether STDOUT needs to be explicitly closed to disconnect the client. If the client is not disconnected then the stream will be held open while the sessions are expired. This shouldn't affect interactive performance, but may make ab report slower numbers.

-- KennethLavrsen - 05 Nov 2005

Expiry of sessions seems to me like something we'd only want to do when the server is not busy. As long as they get cleared before the disk fills too much is there a problem that demands this to be run on every hit? How about a background process that always runs, sleeps for 10 mins and then wakes again. A cron job / remote job running every 30 mins could perhaps check that the process is still running. A single background process could coordinate the dispatch of webstatistics, mailnotify and workflow steps, etc.

Alternatively, run it randomly: roll a random digit 1 to 6 and if it is a 6, clear the sessions. etc.

-- MC


For Cairo and for the early Dakar Beta that still used the Sessionplugin, I had a cron job that cleaned out the old session files.
Its easy enough to wrap such a job in a script that makes sure it only runs if the load factor is low. This script will give the most recent load factor. Do your own arithmetic for what is acceptable.
uptime | sed 's/^.*load average: \([0-9\.]*\), .*$/\1/'

(Waiting for feedback on the subprocess tactic) CC


In case you did not notice. I gave feedback on your patch on http://twiki.org/cgi-bin/view/Codev/DakarPerformanceIssues. I should have done it here. Sorry.

Short summary.

  • Forking off the expiry does not work. It still creates load. And in mod-perl it spawns httpd processes that stay behind and a ab -n 1000 is enough to make the server hang up and deny service.
  • Disabling session cleanout (put # in front of the call to _expireDeadSessions()) does indeed remove the extra load. With this disabled it does not matter how many session files you have in /tmp.
  • My proposal is two changes
    1. {Sessions}{ExpireAfter} = 0 should disable the expiry. Then you can choose this if you'd rather use cron.
    2. _expireDeadSessions() should only be called when a new session is created. That will ensure that only when a browser connects first time and misc bots and RSS readers the server gets the extra load. This is a fair compromize.

-- TWiki:Main.KennethLavrsen - 07 Nov 2005


Yes, that's an excellent solution. I shall make it so (well (1) at least. (2) is too complex and, I feel, unnecessary. Dead session expiry should always apply to all sessions, or we will miss some)

SVN 7348

CC


Thanks for the implementation. I updated my Dakar (SVN is for sure nice here for testing also). And the modified feature works at it should.

I wonder why you say number 2 is complex. The idea seems simple - at least as plain words. If a page view causes a new session ID to be made call _expireDeadSessions if the {Sessions}{ExpireAfter} > 0.

But what you have implemented now does the job. And this is the solution I would have chosen anyway so I am happy. Thanks again. You're the best.

-- TWiki:Main.KennethLavrsen - 07 Nov 2005

ItemTemplate
Summary Many sessions files cause major slowdown - KENNETH
ReportedBy KennethLavrsen
AppliesTo Engine
Component

Priority Urgent
CurrentState Closed
WaitingFor

Checkins 7348
Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View |  Raw edit | More topic actions
Topic revision: r11 - 2005-11-07 - KennethLavrsen
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback