• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7848 for generic doc work for TWiki-6.1.1. Use View topic Item7851 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

Item5960: Incorrect Content-length breaks HTTP headers, a.o. pound fail results

Item Form Data

AppliesTo: Component: Priority: CurrentState: WaitingFor: TargetRelease ReleasedIn
Engine HTTP / CGI Urgent Closed TWiki:Main.SopanShewale patch 4.3.0

Edit Form Data

Reported By:
Applies To:
Current State:
Waiting For:
Target Release:
Released In:


There seems to be a problem with the code that 'hacks' a Content-length header into the HTTP response. It sort of appends it, like this (from TWiki.pm, line 602 in 4.2.2 tarball):

602 my $len = length($text);

603 $hdr =~ s/\n$/Content-Length: $len\n\n/s if $len;

However, this does not really work, since headers are \r\n delimited, so the result is that you get something like:

Content-Type: text/html; charset=iso-8859-1\r\n\rContent-Length: 1234

Ie.: badness!! HTTP fail!! On multiple levels!!

My quick fix has been:

602 my $len = length($text);

603 $hdr =~ s/\r\n$/Content-Length: $len\n\n/s if $len;

That's not the end of it though. Section 4.4 of RFC 2616 (that's the HTTP/1.1 rfc) states:

Messages MUST NOT include both a Content-Length header field and a
non-identity transfer-coding. If the message does include a non-
identity transfer-coding, the Content-Length MUST be ignored.

Now, what i'm seeing is that twiki responses have a 'chunked' transfer-coding, thus the Content-length field MUST NOT be there (in fact, having both headers present can lead to all sorts of security implications, opening up avenues to poison caches etc..).

Why I care about all this? Because twiki currently breaks when put behind pound (a load balancer / SSL endpoint tool).

Lesson learned: don't mess with HTTP headers if you're not sure what you're doing.

What is beyond me, by the way, is that one of my twiki instances does not generate a Content-length header, whereas the other does. There are numerous differences between the various installs (the former is on an apache with mod_perl installed, though not used with twiki, the latter is on an apache with chrooted fastcgi, etc..). The differences are numerous, and it's late so i'm not very inclined to get to the bottom of this. Perhaps one of the TWiki guru's can shed some light on this peculiar bit of code, and on why the Content-length header is present at all.

(ps: as you might know, i'm quite out of the twiki scene at the moment, and i intend to keep it that way. if anyone cares to communicate with me about this bug, please contact me on irc: gmc on the freenode network, or send me an email, as i'm not likely to check the bug database anytime soon).

-- TWiki:Main/KoenMartens - 25 Aug 2008

Note, btw, that 'MUST NOT' means that it is an absolute prohibition, ie: twiki is non-RFC compliant.

-- TWiki:Main.KoenMartens - 25 Aug 2008

Need to address this at release meeting. Or better. Here in the bug item. I think Koen needs feedback from the experts.

-- TWiki:Main.KennethLavrsen - 18 Sep 2008

I discussed with the experts on irc already, we concluded that it needs further investigating: while it is twiki that sets the Content-Length, it should only do so if the Content-Encoding is identity. However, this is not decided by TWiki and (as a preliminary scan revealed) also not by the CGI module. I suspect it might be as high up as the webserver (which would make sense). What needs to be investigated is when and why the Content-Encoding is set and whether twiki can detect this or maybe force the issue somehow.

-- TWiki:Main.KoenMartens - 19 Sep 2008

I've applied the quick fix to the 4.2 branch as someone else just reported the problem to me.

-- SvenDowideit - 14 Oct 2008

AFAIK the rationale for the Content-length header was "performance" - letting the browser know how much content is coming. From my own tests, I can't see that it makes a blind bit of difference - though that may be different with another server/browser combination (I tested on Apache 2 + FF 3). My vote is to drop the content-length header completely.

I see that Sven checked a fix into the release branch but not into trunk. Why?

-- TWiki:Main.CrawfordCurrie - 24 Oct 2008

iirc, its because this code is no longer present in trunk in this form - but things were moved around, so i may well have just failed to find them.

-- TWiki:Main.SvenDowideit - 24 Oct 2008

In trunk this kind of thing is responsibility of each engine. I also plan to move things like generateHTTPHeaders to runtime engines. Core code shouldn't worry about this. IIRC CGI engine (and others) generate the header using TWiki::Engine->CRLF.

-- TWiki:Main.GilmarSantosJr - 24 Oct 2008

http://foswiki.org/Tasks/Item5960 has a potential fix.

I never understood why the content length was added. TWiki was OK for many years without the content length in the header. Because of the content length some speed optimization had to be taken out. For example, search cannot output results as it finds hits, it has to wait until the whole search is finished because of the content length. This is a design flaw in recent TWiki.

-- TWiki:Main.PeterThoeny - 25 Nov 2008

We should have Content-length added to the TWiki.

Following is quote from http://www.mnot.net/cache_docs/

Generate Content-Length response headers. Itís easy to do, and it will allow the response of your script to be used in a persistent connection. This allows clients to request multiple representations on one TCP/IP connection, instead of setting up a connection for every request. It makes your site seem much faster.

In a few examples - like directly allowing download images/attachments from pub/... may avoid having Content-Length (i am talking about way of giving attachments away without using viewfile API). This is taken care by webserver/or it really does not matter if Content-length is present/absent.

Other Applications like Drupal support "Content-Lenght" (http://drupal.org/node/145037 - read the content of the article, dont conclude by reading title).

Ok.. I think we should modify the code to use CGI API to form the header than using plain Regular expressions, concatenate the strings and then print. We should be using something similar to $query->header("-type"=>$type, "-Content-length"=>$length, "-Content-Disposition"=>$dispo,) in the code.

If you guys agree to have the header, i can modify the code to add $query->header support in "release" branch. Later on, i can check the "trunk", i guess Respons.pm plays great role in forming the header in "trunk"-this must be handled in the code.

-- TWiki:Main.SopanShewale - 06 Jan 2009

The code was already working - thanks Sven for coding this. I have modified the code, made $contentLength part of generateHTTPHeaders subroutine.

Changing status to "Waiting for Release"

-- TWiki:Main.SopanShewale - 21 Jan 2009

Summary Incorrect Content-length breaks HTTP headers, a.o. pound fail results
ReportedBy TWiki:Main.KoenMartens
Codebase 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.2.1, 4.2.2, 4.2.3
SVN Range TWiki-5.0.0, Mon, 18 Aug 2008, build 17431
AppliesTo Engine
Component HTTP / CGI
Priority Urgent
CurrentState Closed
WaitingFor TWiki:Main.SopanShewale
Checkins TWikirev:17623 TWikirev:17624 TWikirev:17790
TargetRelease patch
ReleasedIn 4.3.0
Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View |  Raw edit | More topic actions
Topic revision: r18 - 2009-04-08 - PeterThoeny
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback