Item4747: TinyMCEPlugin mangles TML because it eats newlines

<blockquote style="background-color:#f0f0f0">
%ATTACHURL%/WebHome.jpg
</blockquote>

After editing with TinyTMCEPlugin, the text was changed to

<blockquote style="background-color:#f0f0f0">%ATTACHURL%/WebHome.jpg </blockquote>

and, therefore, did not show the attached picture any longer...

TW

Release blocker. This could be any HTML tag round TML. It is a general problem that the TMCE keeps on eating newlines everywhere.

We open one severe bug after the other related to this.

KJL

Let's please try and focus on the problem here, and stop jumping to conclusions. TMCE does not "eat" newlines. it is an HTML editor, and HTML is insensitive to newlines. The problem here is in the conversion of HTML back to TML. TML requires a whitespace before a URL to recognise it as a URL, and it is that space that is getting lost. The fact that it was originally a newline is irrelevant - it could be any whitespace.

The problem is that there's no way to know that a TWiki variable will expand to a URL (or a wikiword for that matter). We can't just put spaces before every TWiki variable, and we can't put spaces inside every HTML tag - either approach will break far more than it fixes. We can't "recognise" > either, because that also breaks in more cases than it works. The translator is clever, but it doesn't have second-sight

I'm not sure there is any solution to this. We may just have to live with it, and accept that if you want an image, you will have to use a <img tag. or, perhaps better, extend the definition of TML such that > is a valid delimiter for a URL or wikiword.

CC

You want users to use <img tag ???

That gives a TWiki:Codev.NerdoMeter score round 8 or 9.

I do not care what kind of editor TMCE is. It eats newlines like it was Pacman. And it causes problems. And normal users will never understand it.

When you open a topic in TMCE and it contains HTML already - then it should leave that HTML alone. Or at least leave the white space alone.

When it eats newlines - TML that requires either newlines or white space are destroyed. The Wysiwyg/TMCE needs to deal with that.

-- TWiki:Main.KennethLavrsen - 01 Oct 2007

TMCE eats whitespace because, in its HTML universe, leading and trailing whitespace of elements is irrelevant. It wouldn't actually need to do so, but hey, that's how it is implemented.

But that's only for leading and trailing whitespace in an element, every other whitespace is just normalized to one single space. So, there's no need to detect the space before variable expansion: As Crawford suggests, I'd think that recognizing >/http://... in lib/TWiki/Render.pm should be reasonably robust. After a literal > you are not within a HTML element, and everywhere else expansion of an explicit URL should be desirable. That would be a simple one-liner.

-- TWiki:Main.HaraldJoerg - 01 Oct 2007

Each time this topic is edited some vital info is changed.

When I answered Crawfords proposal it said that the user should write the less than and larger than as html entities. Ie. a < was written as & l t ; (ignore spaces).

That is what made me say NerdoMeter score 8.

Now I am in doubt what Crawford actually meant to have written.

I hope we all can agree that asking to user to write html entities to hide the tags from TMCE is not a solution.

But that said.

I have several TWiki Applications where I have a TWiki Variable which defines a line in a table. If TMCE removes the new line in front of this tag the twiki application is toast.

I need to be able to trust new lines.

-- TWiki:Main.KennethLavrsen - 01 Oct 2007

This is a very simplified version of my Twiki app. Not at all the real thing but illustrates. And it actually works

• Set CHANGEHISTORY = | Text | Number | Text with another Twiki variable |

So this is what puzzles me most. Why does TMCE not eat the new line before %CHANGEHISTORY but it eats it before %ATTACHURL ??

The reason seems to be that it eats white space around html tags or anything that looks like an html tag. If this is a TMCE "feature" then we need to modify TMCE - OR - We need to somehow protect those newlines so they do not get eaten.

-- TWiki:Main.KennethLavrsen - 01 Oct 2007

Because %CHANGEHISTORY is after a table, and a TWiki table is always terminated by a newline.

This is not a "feature" of TMCE as much as a "feature" of the X(HT)ML parser in the browser. This parser is free to do what it likes with whitespace, as evidenced by the fact that FF, IE and Safari all have quite different treatments of whitespace. Protecting whitespace isn't simple.

-- TWiki:Main.CrawfordCurrie - 02 Oct 2007

Kenneth - Each time this topic is edited some vital info is changed. Sorry, that was me. Revision 6 of this topic, in which Crawford wrote his ideas about <img, develop.twiki.org suffered from over-escapitis and converted that to <img. I repaired that in Crawfords suggestion and in your code, only realizing later that it was exactly that escapitis which you took for real. I am pretty sure that Crawford meant <img, which is still too high on the nerdometer.

Crawford - you are still missing a point which disturbs Kenneth: Because %CHANGEHISTORY is after a table, and a TWiki table is always terminated by a newline. doesn't quite catch it, because there was a newline after Thomas' initial blockquote element in the bug report. It is a bit more subtle: For a table, TMCE could eat the newline as well. It is the final HTML to TML translation which will make sure that every table row, and the table itself, will be on a line of its own: The translator generates a newline, regardless of whether there was one in the HTML text, before and after a table.

BTW: It is possible to break TWiki tables with TMCE as well.

| *Who* | *Rev* | *Description* |
| pre_capture | 1 | include images before first motion detect |
| post_capture | 2 | append images after last motion detect |
%CHANGEHISTORY%
%CHANGEHISTORY%

After an edit/save cycle with TMCE, there will be a fourth table row with six elements. I have such stuff, where the table body comes from a search, and a table footer from a preference variable. Duh.

It would be really interesting to find out where that newline between the two spans around the TWiki variables gets lost. If I view the HTML text with TMCE, the original lines are preserved!

For the scope of this bug report I suggest to implement the one-liner in Render.pm as Crawford suggested, if only because it would have been the right way to do it from the start. I would prefer not to abuse a bug report for general musings about how to translate HTML to TML.

-- TWiki:Main.HaraldJoerg - 02 Oct 2007

Harald, when I edit and save that topic there is no fourth table row. Are you confused by a plugin?

The newlines are indeed visible in FF. You will not necessarily see the same whitespace in other browsers, however, so the translator has to treat newlines in the post-edit HTML as whitespace.

-- TWiki:Main.CrawfordCurrie - 03 Oct 2007

I can't TMCE at develop.twiki.org, but can reproduce it with plain 4.2 Beta 1. After saving, the topic contains %CHANGEHISTORY%%CHANGEHISTORY%, i.e. not a single space between the two variables instead of a newline. For TMCE these are just two adjacent span elements, and it shows one whitespace in edit view and a newline in HTML view. HTML2TML can't know without variable expansion that in fact they are table rows, and can only try to Do The Right Thing if TMCE passes the whitespace between the spans as newline.

So, I don't understand your last sentence: the translator has to treat newlines in the post-edit HTML as whitespace. Whitespace doesn't necessarily mean "space"! While I don't mind that HTML2TML collapses multiple whitespaces because they're irrelevant in HTML, it should not convert a single whitespace (whether space, tab, or newline) into a single space character. And I've not found out why the file has no whitespace between the variables at all (the code isn't that easy to read).

--TWiki:Main.HaraldJoerg - 03 Oct 2007

Trying again

-- TWiki:Main.KennethLavrsen - 10 Oct 2007

The above edit was entered in Raw Edit and saved. And then opened and saved in TMCE/IE7. And as can be seen the newline between the two CHANGEHISTORY variables has been eaten.

I have plenty of twiki apps where I merge results of twiki variable such a searches into larger tables. And we have other issues that come up all the time because TMCE eats new lines. Eating normal multiple spaces leaving one is normally OK but it must not eat newlines.

If there was a newline in the original topic opened for editing then it should also be there when I save it unless the user has deleted it. Otherwise the editor will destroy TML based apps. We keep opening new similar bugs all the time and I have actually stopped adding more cases because they all have the same root cause. Disabling TMCE when you have applications is not a solution because we always have some degree of intelligent TML in most topics. That is why we use TWiki instead of the other wikis.

The argument that it is TMCE that is coded this way is not an excuse our users can use for much. We either have to work around it or make a special TWiki version of TMCE that does not eat newlines. The old Kupu editor had same problem and the eating of newlines was the most important reason why I had to remove it from our installation. This has to be fixed.

Title of this bug altered to reflect the generic issue.

-- TWiki:Main.KennethLavrsen

Kenneth, it's not a flaw in TMCE, it's a flaw in the browsers. There was never any TinyMCE is coded this way argument. I doubt that you are suggesting we make a special version of IE/FF/Safari for this.

As with anything it may be possible to work around this, but it is likely to reduce usability, as we will have to make some gross assumptions about topic formatting. A shame, really, as this is such an obscure edge case.

-- TWiki:Main.CrawfordCurrie - 13 Oct 2007

It is not edge cases.

The eating of newlines affect anything I try to edit. I just don't bother to open new bug reports on it anymore. When I was testing EditTablePlugin I noticed more cases where my topics are mangled by TMCE. It is a huge problem.

-- TWiki:Main.KennethLavrsen - 13 Oct 2007

If you don't report them, I don't know about them. if I don't know about them, I can't fix them. Impedimentum partis dimidium est

CC

I have spent too much time on this one bug already. I decided to start testing other features with real topics and the same installation as I have at Motorola and I have opened urgent and normal bugs like hell this weekend. I will power open bugs against TMCE as I see them but first I need some of the urgent bugs fixed that prevents the thing from running in the first place.

-- TWiki:Main.KennethLavrsen - 15 Oct 2007

Closing this because I have addressed all of the specific problems described in it. Please open new bugs for any problems that you think are the same thing, because most of the time they are not.

[CC]]