Enter %GROUPS% in a Sandbox topic. The view script takes a big load for 10 sec, then redirects to the view script that eats 99% CPU for indefinite time. The view process needs to be killed. Using Apache login with .htpasswd file.

-- TWiki:Main/PeterThoeny - 31 May 2007

Same if you visit Main.TWikiGroups that has the %GROUPS% variable.

-- PTh

do you mean on this server, twiki.org, or under what setup ? Its working on my servers, even when running twiki.org's user and group topics, so i need more info to proceed with this.

-- SvenDowideit - 02 Jun 2007

This is on my own development server with latest MAIN.

Linux server 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:59:02 EST 2004 i686 i686 i386 GNU/Linux

Perl, v5.8.0 built for i386-linux-thread-multi

-- PTh

still insufficient information to proceed.

-- SvenDowideit - 02 Jun 2007

Any suggestion what more details to provide?

-- PTh

I get instant view. So something is different on your server. We need to be able to reproduce it to fix it. Peter you need to provide details of how many groups you have. How about starting with a screen shot of the resulting GROUPS

KJL

Here are my groups:

Main/NobodyGroup.txt

   * Set GROUP =

(empty list)

Main/PtestGroup.txt

   * Set GROUP = Main.PeterThoeny, Main.NonAdminTest

Main/TWikiAdminGroup.txt

   * Set GROUP = ATWikiUser, PeterThoeny

Main/TestLargeGroup.txt

   * Set GROUP = TestUser1, TestUser2, TestUser3, ...., TestUser59999, TestUser59998, TestUser60000

(a very large group with 60K entries)

The last group kills the server.

If I move that group away into the Sandbox web I can load the TWikiGroups topic normally.

BTW, here is the time it takes for Cairo and TWiki 4.2 pre-beta to start loading the Sandbox.TestLargeGroup topic. (Time between accessing Sandbox.TestLargeGroup and starting to load the topic)

• Cairo: 11 sec
• TWiki 4.1.2: 1 min 59 sec
• TWiki 4.2: 1 min 44 sec

Looks like we need to do a lot of performance optimization in TWiki 4.2

-- PTh

I assume it is the Cairo that takes 11 sec and TWiki 4.2 that takes 1:44

• Yes, flipped -- PTh

The large group detail was surely missing in the original bug report.

I was under the impression that all the new user code was going to be faster. There is no way we are going to accept further slowdown of TWiki for any reason. And naturally a large TWiki with large groups should work flawlessly and fast.

-- KJL

yes, a bug report needs to contain enough information that anyone can read it, and know precisly how to reproduce it. While knowing that you have a 60K large group should really be enough for me to go on with, can you also add relevant configuration settings, and which svn rev of MAIN we're talking about? There are major differences between those too.

while you're at it, I'd like to know times 4.1.2 gives you.

Peter - once you've added that info, just put it waiting for me.

just to give you both a little reassurance - so far there is no optimisation at all, I have foccussed on correctness first - the next commit should be to do with legacy forms of users in rcs, topics etc, and after that we should be placed to make this fast.

-- Sven

We are mixing two issues here:

1. Performance of loading a particilar large topic. More specifically, time it takes to start display a large topic
2. Performance of loading a topic that has a %GROUPS% if there is a very large group.

In regards to item 1:

• TWiki 4.2 was SVN rev 14054.
• I updated the timing above with TWiki 4.1.2 loading the page in the Sandbox web

In regards to item 2:

• Time between accessing Main.TWikiGroups and starting to load the topic:
  • Cairo: N/A (no %GROUPS% variable)
  • TWiki 4.1.2: 1 min 57 sec
  • TWiki 4.2: 2 min 15 sec
• CPU use is 99% on the view script during the load time (server has no other load)

-- PTh

ok, commit 14099 contains only one change, that halves the time to render for the test i'm using (the fixed version of tools/create_large_groups.pl that was commited

Peter, can you please have a test with your setup and tell me how it goes? I expect many other simple tweks to come out of the Monitor code, most of which should really use TWiki::Cache, but I hope that just this change will make 4.2 more capable than 4.1

-- SvenDowideit - 12 Jun 2007

See attached the large group topic. Please use this for performance testing.

Time to start loading Main.TWikiGroups in latest SVN checkout: 2 min 17 sec.

-- PTh

does this mean that you are creating a large group topic, of users that don't have either .htpasswd entries, not user topics? because I user the MAIN/tools/create_large_groups.pl script to make 100,000 users, and the 6 groups - ie, one with 50,000, one with 25,000 ... etc and then GROUPS still took less then 2 minutes to render on my p4m 1.8 notebook. After that my browser was maxed out for ages, but on the server side it 'coped'.

I'm gathering timing stats using time ./view TWikiGroups > /dev/null otherwise the time taken by the shell to output the data is much higher.

Also - can you start stating what 'latest SVN checkout' is - ie, the revision number when you svn up - otherwise it unclear in future, and unclear now if it was before or after certain changes.

-- SvenDowideit - 15 Jun 2007

Why are you guys so anal on server performance when nobody wants go get such huge pages. Delivering the GROUPS information paginated in smaller chunks makes much more sense wrt performance and usability. Server and browser crack the shit out of the bits just to present the user a monster.

-- TWiki:Main.MichaelDaum - 15 Jun 2007

I am interested in using %GROUPS% as a stress test, as well as to debug what groups are actually defined as, but I have to admit that I was expecting that some UI person would work on making the output nicer. The fact that Peter's testing is giving quite different results to mine, also makes for an important thing - until all the permutations are known, and resolved (server side) we don't know how the code performs.

-- SvenDowideit - 15 Jun 2007

GROUPS is a worst case scenario for every system on the way. It is no valid use case! We know already that a request for all of the database information is a huge performance burden. Sure, you can measure an environment how slow this is. Much more important is to omit this operation by the architecture underneath in normal use cases (authentication and authorization).

-- TWiki:Main.MichaelDaum - 15 Jun 2007

One pragmatic solution is to cut the group at 256 or so members for display only, with a linked ... at the end, linking to the group topic.

-- PTh

That's fine if there is a group topic. If it's an LDAP group, there isn't.

Something along those lines might be possible, though.

CC

FFS, I am Still waiting for Peter to answer the question from the 15th - If you want to discuss changing the UI, please make yourselves a new bug item.

does this mean that you are creating a large group topic, of users that don't have either .htpasswd entries, not user topics?

I use the MAIN/tools/create_large_groups.pl script to make 100,000 users, and the 6 groups - ie, one with 50,000, one with 25,000 ... etc and then GROUPS still took less then 2 minutes to render on my p4m 1.8 notebook. After that my browser was maxed out for ages, but on the server side it 'coped'.

I'm gathering timing stats using time ./view TWikiGroups > /dev/null otherwise the time taken by the shell to output the data is much higher.

Also - can you start stating what 'latest SVN checkout' is - ie, the revision number when you svn up - otherwise it unclear in future, and unclear now if it was before or after certain changes.

-- SvenDowideit - 15 Jun 2007

Peter actually wrote TWiki 4.2 was SVN rev 14054

But Peter you owe Sven an answer to does this mean that you are creating a large group topic, of users that don't have either .htpasswd entries, not user topics?

-- TWiki:Main.KennethLavrsen - 18 Jun 2007

no, I also asked Peter to re-test after the 14099 commit, and the remaining details of that setup. So I still need svn revision info - especially as since then, things have changed again :/.

-- SvenDowideit - 18 Jun 2007

I tested it again. Now it takes 23 sec to display the TWikiGroups topic, a very big improvement. Tested on svn 14407 with a large TestLargeGroup topic (attached) containing 60K users. Actual user homepages and .htpasswd entries do not exist.

The "Members" field is empty for the TestLargeGroup, which is confusing. How about showing up to 32 users, and if more have a linked "..." pointing to the group topic? That way there is a clear indication that there are more users in the group.

-- PeterThoeny - 16 Jul 2007

Changed status to 'Waiting for Feedback' from Sven; it was 'Being worked on' with no-one in the 'WaitingFor'.

CC

Peter, I think your test might not be valid. When I fixed the user code so that the TWiki.org setup works, I removed the reliance on the TWikiUsers topic if allowloginnames=false. Instead, it checks user existance by looking for an entry in the .htpasswd file.

Thus, for a user to be in a group, they need to be

1. in the .htpasswd file (if allowloginnames=false) OR
2. in the TWikiUsers topic (if allowloginnames=true)

I did correct the tools/create_large_groups.pl script to add the user to the .htpasswd file - which is how i was previously doing the test.

mmm, 23 seconds is rather better than i expected - as I thought i'd made the code slower when I wrote the compatibility code.

OK - I intend to look at limiting the number of users rendered in the list before we release, but please note that this will only affect the rendering output of %!GROUPS% - it will still parse, expand and check the full list - at least until we re-do the iterators in a later release

-- SvenDowideit - 18 Jul 2007

I've just hard coded it to only show the first 32 users. paging and settings will have to wait for a later release (I'll open a new bug for that)

-- SvenDowideit - 19 Jul 2007