• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7848 for generic doc work for TWiki-6.1.1. Use View topic Item7851 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

Item5704: Performance tuning of HtPasswdUser for large number of users

Item Form Data

AppliesTo: Component: Priority: CurrentState: WaitingFor: TargetRelease ReleasedIn
Engine   Normal Confirmed   major 5.0.0

Edit Form Data

Reported By:
Applies To:
Current State:
Waiting For:
Target Release:
Released In:


As you might now, we have performance issues with the TWiki:Codev/NewTWikiOrgServer2008. A Sun specialist found out that reading the .htpasswd file with 36K entries is very slow. Two suggestion to improve performance:

1. Performance tune reading the data

Here is the patch Sun came up with:

I have modified the program based on a suggestion from a Compiler Expert and the response time with our test has come down to 3.5 sec - a whole second down

here is the diff

--- HtPasswdUser.pm.save1    2008-01-21 19:18:55.000000000 -0800
+++ HtPasswdUser.pm    2008-06-12 10:27:36.510762000 -0700
@@ -121,9 +121,9 @@
   open( IN_FILE, "<$TWiki::cfg{Htpasswd}{FileName}" ) ||
     throw Error::Simple( $TWiki::cfg{Htpasswd}{FileName}.' open failed: '.$! );
-    local $/ = undef;
-    foreach my $line ( split(/\r?\n/, <IN_FILE>) ) {
-        if( $line =~ /^(.*?):(.*?)(?::(.*))?$/ ) {
+    my $line = '';
+    while (defined ($line =<IN_FILE>) ) {
+        if( $line =~ /^(.*):(.*)(::(.*))?/ ) {
           $data->{$1}->{pass} = $2;
           $data->{$1}->{emails} = $3 || '';

He has made more suggestion in rearranging the data structure which gives very good response time based on a microbenchmark that i wrote (attached - the commented ones are the various options i used)

2. Lazy loading of user data

There is no reason to load all users at init time if the user is a guest. The user data should be load late, only when needed, e.g. when an access control check is done, or when a login name needs to be converted to WikiName.

-- TWiki:Main/PeterThoeny - 16 Jun 2008

Or, if you have a large userbase, use a random access file (like DBM) or database backed User Store. TWiki:Plugins.HTTPDUserAdminContrib supports both DBM and database.

The htpasswd data was lazyloaded, but there was a backwards compatibility problem wrt groups or something that forced me to revert that frown

marking as urgent and checked in speedup for 4.2.1

-- TWiki:Main.SvenDowideit - 16 Jun 2008

note that the patch as provided by Sun is incorrect and will prevent some users from being able to login - Micha fixed it pretty fast.

-- SvenDowideit - 25 Jun 2008

At the 07 Jul 2008 release meeting it was agreed that this bug has a fix that takes care of the urgency. We keep it open for a more general fix (which requires a rewrite of the user code) but with Normal priority.

-- KennethLavrsen - 09 Jul 2008

no, this bug has not had a fix applied. I incorrectly assumed that the Sun patch worked, and applied it, Micha had to back it out again to be able to login. Therefore, it is still urgent, needs actual analysis, and a proper fix. It is also likely that the reason Sun got a speedup, is because their tested regex is broken.

-- SvenDowideit - 28 Jul 2008

We knew this when we made the decision.

I do not see it realistic to fix this and get it stable in the 5 days we have before the 4.2.1 release.

The alternative - to let people download the buggy 4.2.0 is worse.

I downgrade to normal. Nothing prevents anyone to fix it for 4.2.1 even if the bug is normal graded.

But not releasing 4.2.1 now means this bug will still be in 4.2.0 so it does not resolve anything delaying the 4.2.1 release again.

-- TWiki:Main.KennethLavrsen - 28 Jul 2008

Summary Performance tuning of HtPasswdUser for large number of users
ReportedBy TWiki:Main.PeterThoeny
Codebase 4.2.0, 4.2.1, ~twiki4
SVN Range TWiki-5.0.0, Sun, 01 Jun 2008, build 16865
AppliesTo Engine

Priority Normal
CurrentState Confirmed

Checkins TWikirev:16906 TWikirev:16907 TWikirev:16908 TWikirev:16909
TargetRelease major
ReleasedIn 5.0.0
Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View |  Raw edit | More topic actions
Topic revision: r12 - 2008-07-28 - KennethLavrsen
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback