• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7848 for generic doc work for TWiki-6.1.1. Use View topic Item7851 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

Item4178: New query search is far slower than equivent regex meta search

Item Form Data

AppliesTo: Component: Priority: CurrentState: WaitingFor: TargetRelease ReleasedIn
Engine   Urgent Closed   minor 4.2.0

Edit Form Data

Summary:
Reported By:
Codebase:
Applies To:
Component:
Priority:
Current State:
Waiting For:
Target Release:
Released In:
 

Detail

I have started testing the new tql based search. Ie. the type="query" search tested in practical

And I went for the simple use case first. I like the simple syntax. Much more logical than doing the same with the regex meta hack we have been used to.

The test I am doing is replacing some formatted searches where I only search for one value in one field.

But the new search has a severe performance problem. I have not timed it very accurately. Just counting the seconds. My meta regex searches take round 7 seconds. The tql search takes 12 seconds. Almost double up. That needs to get optimized. There is no good reason why such a simple search should be half the speed in the twl version as I assume it is in practical still implemented as a regex search.

Take a look for yourself.

I tested with the crond stopped. I repeated tests at least 10 times for each page. Back and forth. There is no doubt. The new search has a performance problem. Hopefully one that can be resolved.

If you benchmark on my server then remember that from XX:00 - XX:05 and XX:30 - XX:35, cron runs svn update, remove links, chown, pseudo-install and this makes the server very slow and impossible to use for timing tests. Also remember that the server runs on an ADSL line. When I test I have a gigabit network. That can mean a big difference. So you need to create a similar environment on your local machine to recreate.

-- TWiki:Main/KennethLavrsen - 01 Jun 2007

"I assume it is in practical still implemented as a regex search" - no. I spent all the time I had planned to spend on optimisation answering alternative syntax proposals, as it was clear that getting the syntax right was a greater issue than performance of a (potentially broken) implementation. At the moment the implementation is based on a linear, brute-force search through the topics. I did some experiments with using RE's to accelerate it, but decided that RE optimisation wasn't practical in the time I had available.

There are three possible approaches I am considering:

  1. "Winnowing" the set of topics that have to be brute-force searched by extracting and applying static RE's derived from the query
  2. A DBCache based search engine (raises the requirements on the install, but up to 10X faster than an RE search)
  3. A DBI store, which can be searched using SQL

-- CrawfordCurrie - 02 Jun 2007

Kenneth, please repeat your benchmarks, and close this report when you are happy with the performance.

CC

Looks very good. Measurements of the "Motion bug search" using ab 10 times.

  • Patch04x01 regex search (more or less 4.1.2): 5.4 seconds
  • MAIN regex search SVN 14112: 5.4 seconds.
  • MAIN query search SVN 14112: 5.4 seconds.

Exactly the same. So excellent work. Closing.

KJL 13 Jun 2007

ItemTemplate
Summary New query search is far slower than equivent regex meta search
ReportedBy TWiki:Main.KennethLavrsen
Codebase ~twiki4
SVN Range TWiki-4.1.2, Fri, 18 May 2007, build 13796
AppliesTo Engine
Component

Priority Urgent
CurrentState Closed
WaitingFor

Checkins TWikirev:14110 TWikirev:14111
TargetRelease minor
ReleasedIn 4.2.0
Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r9 - 2007-06-13 - TWikiUserMapping_KennethLavrsen
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback