Item5248: TinyMCE breaks unicode (was breaks euro currency sign)

Enter a euro currency sign (€) into the editor and save the topic. The Euro Sign shows up correct in the topic. Edit the Topic again and the euro sign is converted into the string %u20AC.

This doesn't happen to other special chars like german umlaute (äüö)

-- TWiki:Main/GerhardHeeke - 14 Jan 2008

When I try this on my home server I have no problem with the Euro.

But if I copy the raw text from here the umlauts fail instead.

And when I edit this topic the d.t.o. server fails with this in the error log when try to edit it with Wysiwyg.

[Mon Jan 14 15:40:34 2008] [error] [client 83.91.41.235] Wide character in print at /home/twiki4/twikisvn/lib/TWiki.pm line 612., referer: http://develop.twiki.org/~twiki4/cgi-bin/edit/Bugs/Item5248?t=1200346816
[Mon Jan 14 15:40:34 2008] [error] [client 83.91.41.235]  at /home/twiki4/twikisvn/lib/TWiki.pm line 160, referer: http://develop.twiki.org/~twiki4/cgi-bin/edit/Bugs/Item5248?t=1200346816
[Mon Jan 14 15:40:34 2008] [error] [client 83.91.41.235] \tTWiki::__ANON__('Wide character in print at /home/twiki4/twikisvn/lib/TWiki.pm...') called at /home/twiki4/twikisvn/lib/TWiki.pm line 612, referer: http://develop.twiki.org/~twiki4/cgi-bin/edit/Bugs/Item5248?t=1200346816
[Mon Jan 14 15:40:34 2008] [error] [client 83.91.41.235] \tTWiki::writeCompletePage('TWiki=HASH(0x9df2b3c)', '<!--WYSIWYG content - do not remove this comment, and never u...') called at rest line 181, referer: http://develop.twiki.org/~twiki4/cgi-bin/edit/Bugs/Item5248?t=1200346816
[Mon Jan 14 15:40:34 2008] [error] [client 83.91.41.235] Premature end of script headers: rest, referer: http://develop.twiki.org/~twiki4/cgi-bin/edit/Bugs/Item5248?t=1200346816

This looks serious. Very serious. This looks like a show stopper.

-- TWiki:Main.KennethLavrsen - 14 Jan 2008

I have made an over simplified test topic with only the one unicode character from above.

Here on d.t.o the rest scripts faults

On my home server the unicode turns to garbage

http://develop.twiki.org/~twiki4/cgi-bin/view/LitterTray/Item5248Test

This is a situation which is static and reproduceable and seems to affect anyone that types a unicode character. Here is was a german reporter. Not a japanese or chinese.

We have a serious issue here.

There is a TODO in TWiki.pm where the code fails.

Crawford you wrote this todo. What is it that should have been done here?

-- TWiki:Main.KennethLavrsen - 14 Jan 2008

if there's no-one to fix it, then i don't think delaying release helps anyone Imagining that we released last week (for eg) and then had this bug found, we would not pull the release

This is how i see 'show stopper' at this late point - we really really need to find i8n help.

-- TWiki:Main.SvenDowideit - 15 Jan 2008

I am OK with the i18n part of this bug not being fixed. I am concerned about the code crashing.

Is d.t.o setup to fail on warnings? If this is the case then maybe with a warning all that happens is the unicodes becoming garbage.

If you type some German in the Wysiwyg don't you get correct text? I do when I type Danish.

It is the crashing I am concerned about. Not that the unicode fails. We know that already

-- TWiki:Main.KennethLavrsen - 15 Jan 2008

We now know why it fails on d.t.o. d.t.o runs with some extra checks because it is a development server (LocalLib.cfg has $ENV{TWIKI_ASSERTS}=1;)

When you disable that d.t.o does not fail. But instead it alters the unicode into garbage. It happens when the Wysiwyg editor reads the topic. Not when you save.

So there is a bug but it is too risky to try and fix it before 4.2.0.

The Eurosymbol when entered using the menu or html entity as it was in this bug report seems to work. This was for sure broken a few weeks ago so if the reporter runs with RC2 then they will see the old bug

Downgrading to Normal now we know normal installations will not see crashed code.

-- KennethLavrsen - 15 Jan 2008

Ignoring the details of this bug: if you want to use common Western European accented characters and the Euro sign, you must set your locale to something ending in iso8859-15 and also set the charset to ISO-8859-15 - both in configure. See InstallationWithI18N for the details.

The Euro is not part of ISO-8859-1 which is what the original reporter may be using, but is part of ISO-8859-15.

-- TWiki:Main.RichardDonkin - 15 Jan 2008

Nice idea, but the problems were much, much deeper

Fixed now (as of 16650)

-- CrawfordCurrie - 12 Apr 2008