• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7848 for generic doc work for TWiki-6.1.1. Use View topic Item7851 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

Item4291: User mapping causes entire LDAP download.

Item Form Data

AppliesTo: Component: Priority: CurrentState: WaitingFor: TargetRelease ReleasedIn
Extension LdapContrib Urgent Closed   n/a  

Edit Form Data

Summary:
Reported By:
Codebase:
Applies To:
Component:
Priority:
Current State:
Waiting For:
Target Release:
Released In:
 

Detail

TWiki 4.1.2/June 22, 2007 download of LdapContrib/Linux/Apache 2.0.52...

I'm trying to get LdapContrib to do auth and also create WikiNames from another LDAP field. These things are reasonably well documented.

My problem comes when I try to enable user mapping... the code attempts to load into the cache file ALL of our LDAP directory (130,000 entries in our case) every time I click on a function such as "Login".

After looking at the code for a couple hours, it appears that the issue is that "loadLdapMapping" in only called with $this -- rather than $this and $name (needs to build a more specific search filter).

Am I missing something simple? I can't find any parameters to limit the cache (or turn it off). It seems that the intent would be to add individuals to the cache as they do somthing, not pull down the whole thing...

-- TWiki:Main/CrisRhea - 23 Jun 2007

I'm having the same problem. Is there a workaround or fix?

-- TWiki:Main.CurtWilhelm - 23 Jul 2007

My workaround is to set the cache refresh rate to every couple of days.

All in all, LdapContrib is an invaluable contrib, but I find the cache implementation very strange. In my installation I'd actually have less slowdown with no caching at all than with the default cache life of 10 minutes. My LDAP server can provide a single entry with essentially no delay, but it takes 10-15 seconds to provide all 6000 entries. Causing a page load to hang that long every 10 minutes just isn't acceptable.

Really, the cache should be re-implemented so that LDAP entries are only refreshed one at a time. And until we get that feature, we need some way to turn caching off.

-- TWiki:Main.IsaacRabinovitch - 22 Aug 2007

Yes, the current LdapContrib is (a) not caching efficiently and (b) buggy.

There have been lots of people asking me why LdapContrib tries to download so many records at once. Let me try to find a lame excuse explain.

TWiki (we are talking about 4.1.2, the current stable release) calls for user information internally on a couple of occasions. A user logs in, he is authenticated agains LDAP, infact no user info is needed for that, just a bind to the server. Once a user is logged in, his attributes are fetched from LDAP to construct his WikiName besides email info and the like. This is one record and this information is needed on every click he does, so thats worth caching. Let's not consider the case of multiple users acting on the site simultaneously. Surely, this is at least on LDAP record each. Let's stay with this one user who just logged in and now finds his WikiName pointing to his TWiki homepage. So far we only did authentication and some housekeeping. Now comes authorization: users are member of groups and there are ACLs everywhere. To check authorization TWiki now requests more information: groups and their members. TWiki-4.1.2 constructs a user object for every user and every group (this has been reworked in TWiki-4.2), and thereafter checks if the current user is member of that group. Imagine that it does not suffice to only request the current user's LDAP record but those of most if not all others now. Now comes the reason why things are so slow: while in the first use case we only needed one record for the current user to let him in and hop around, we now have the second use case for the code: to perform authorization. The bad part is, that this is all force through the same API. LdapContrib can't distinguish those two use cases. It does not know if TWiki is about to request info about one user, or if it is in the middle of an authorization process where a bunch of user infos are needed. TWiki is implementing the membership check on its own application level, whereas this check could have been done on the backend level with fewer overhead.

The current caching in LdapContrib is bad. Full stop. What it does is it requests chunks of user records while they are requested from the backend and caches them. It does this as longs as the actually requested user record was not found. The next call will either benefit from the work done by the previous call, or continue to request chunks of user records. This process is done in the middle of the rendering pipeline, that is, when the user clicks around. This process is time critical and must be as lean as possible by all means. TWiki needs more than just one user record per click. Worstcase use case is visiting Main.TWikiGroups: all users of all groups are requested and formatted into one big html page, thus dossing the LDAP server (if these records haven't been fully cached yet), the TWiki server constructing perl objects for each user and last not least the user's browser that has to render the resulting large page. Well, but that can be dealt with by removing the %GROUPS% tag from Main.TWikiGroups and replace it with a paginated %LDAP{}% query.

So the solution for LdapContrib on TWiki-4.1.2 is to remove the caching mechanism from the rendering pipeline and make it a separate thing. LDAP records don't change that often that there is a need to populate the cache from scratch every 10 minutes automatically. Much more likely, LDAP admins add a new record and then want this change to be visible on the TWiki side. It would not harm to add an extra "Refresh LDAP cache" button to the LDAP admin's workflow to make sure the LDAP cache of TWiki is up to date. What I am going to say is, as TWiki-4.1.2 needs a pretty broad knowledge about its users anytime, caching should be a one shot manual thing and not be done automatically anymore. If the LDAP data changes every 24 hours or so, go with a cronjob that runs midnight and refreshes TWiki's LDAP cache.

As far as I know this is the best we can do wrt LDAP & TWiki-4.1.2.

-- TWiki:Main.MichaelDaum - 22 Aug 2007

Not sure I really understand that, except the part about it not getting fixed soon. So let's talk about workarounds. One idea is set the cache refresh to > 1 day, then cause a manual refresh every night when usage is low. Question: how to refresh manually? The button on LdapContrib does nothing that I can see. It just invokes /twiki/bin/view/TWiki/LdapContrib?refreshldap=on, which returns immediately — even though a refresh on my network takes at least 30 seconds. Also, no debug output; there should be about 8000 retrieval messages.

-- TWiki:Main.IsaacRabinovitch - 25 Aug 2007

I will be working on LdapContrib next week as I need a fixed version on september 3.

-- TWiki:Main.MichaelDaum - 25 Aug 2007

Fixed in 2.0 release.

-- TWiki:Main.MichaelDaum - 11 Oct 2007

ItemTemplate
Summary User mapping causes entire LDAP download.
ReportedBy TWiki:Main.CrisRhea
Codebase 4.1.2
SVN Range TWiki-4.1.2, Thu, 21 Jun 2007, build 14235
AppliesTo Extension
Component LdapContrib
Priority Urgent
CurrentState Closed
WaitingFor

Checkins TWikirev:15231 TWikirev:15232
TargetRelease n/a
ReleasedIn

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r9 - 2007-10-11 - MichaelDaum
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback