• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7700 for generic doc work for TWiki-6.0.2. Use View topic Item7703 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

Item5457: TinyMCE breaks Cyrillic Encoding (cp1251 or KOI8-R)

Item Form Data

AppliesTo: Component: Priority: CurrentState: WaitingFor: TargetRelease ReleasedIn
Extension WysiwygPlugin Normal Closed   patch 4.2.1

Edit Form Data

Summary:
Reported By:
Codebase:
Applies To:
Component:
Priority:
Current State:
Waiting For:
Target Release:
Released In:
 

Detail

If I am using Raw Edit, everything works just fine.. If I am trying to use WYSIWYG the cyrillic symbols going to be broken.

test2.PNG

My setting in LocalSite.cfg.

$TWiki::cfg{Site}{Locale} = 'ru_RU.cp1251';
$TWiki::cfg{Site}{CharSet} = 'cp1251';

I have also tried with KOI8-R.

-- TWiki:Main/OlegButovich - 19 Mar 2008

This is representative of the general problem with UTF-8 encodings, reported elsewhere. Duplicate of Item5314.

-- TWiki:Main.CrawfordCurrie - 27 Mar 2008

CP1251 and KOI8-R are single-byte encodings BTW, not unlike ISO-8859-1, except that almost every character in Cyrillic will have the high bit set.

-- TWiki:Main.RichardDonkin - 27 Mar 2008

I think the same... As far as I understand, the same problem should have, for example, Germans with umlauts... like Œ€�... May be I am doing something wrong?... Will try to test with umlauts.

-- TWiki:Main.OlegButovich - 28 Mar 2008

OK, I didn't know that, thanks Richard. I'm pretty sure that 8 bit encodings work, as the characters in the symbols drop-down are all 8-bit.

А Б В Г Д Є Ж Ѕ З И І К Л М Н О П Р С Т Ф Х Ц Ч Ш Щ Ъ Ы Ь

-- TWiki:Main.CrawfordCurrie - 28 Mar 2008

I have tested now with euro locale on my server without any problem... I can switch in any order between WYSIWYG and Raw editor, the special symbols are not broken.. That means, there is definitely a problem with russian 8-bit locale :(.. for example, if you look on your АБВ symbols in Raw editor, or just switch from WYSIWYG editor to raw and back, you will see this effect.. Ok, this is understandable on this site, were russian locale is not set... but believe, if will set it, it will not help a lot.. frown You will face the problem I've described in original bug report. With Euro locales it works perfect, with Russian not..

-- TWiki:Main.OlegButovich - 30 Mar 2008

I've now changed this back to waiting for feedback from Crawford based on Oleg's comment.

-- RichardDonkin - 30 Mar 2008

Thanks Richard.

I see what Oleg is talking about now; the encoding is wierd. Not sure where that is happening. This is not the same as Item5314, it is nothing to do with the locale, and it needs some quality debugging time. Confirmed.

-- CrawfordCurrie - 31 Mar 2008

I believe I finally found the solution. I ended up having to convert octets to UTF-8 to stop the HTML::Parser falling over, then converting UTF-8 wide chars to HTML entities to stop the print falling over (STDOUT is not opened :utf8). I was able to engineer the fix without needing to touch the core code.

Many thanks to TWiki:Main/ThYang, TWiki:Main.TimothyChen, TWiki:Main/OlegButovich and TWiki:Main.RichardDonkin for exploring around the problem and proposing investigative procedures.

-- CrawfordCurrie - 31 Mar 2008

Now it not convert in entities, but convert it in UTF-8 (but I see 8 bit string like P `` T `` T) after saving and after reload page(and firefox set codepage in UTF8 if i reload page ), when i set cp1251 in config frown

-- TWiki:Main.VictorKasatkin - 21 May 2008

I try old version (31 Mar 2008) and its work fine. smile something broken after this date frown

-- TWiki:Main.VictorKasatkin - 22 May 2008

Unless you open a new bug report this one will go into a back hole in a matter of 1 -2 days (then it can no longer be seen among the last 10 changed bug reports)

-- TWiki:Main.KennethLavrsen - 23 May 2008

Please run the test script atatched to TestEncodings. If the tests pass, then I'm pretty sure i can fix these problems, but I need you to confirm it works for you.

-- CrawfordCurrie - 23 May 2008

Never mind, I ran the necessary tests myself, and fixed the plugin.

-- CrawfordCurrie - 25 May 2008

Im run script with cp1251( under suse10.3/apache2/twiki412 altlinux/apache13/twiki420 - firefox2) and set range 0-256, and i not see any russian chars - only \x{HHHH}...

-- TWiki:Main.VictorKasatkin - 28 May 2008

At frist time I can not to see bug, butnow Im find it (31.04.2008 version): when i click on raw button it convert all russian chars to the entities, but save it normaly(after save i see russian chars in source by firefox.)

-- TWiki:Main.VictorKasatkin - 28 May 2008

unfortunattely it saved encoded in entities in data folder(with 31.03.2008 version). Now i set last version plugin and utf8 in config. And all work much better. If another plugin(ExcelExportImport, SVG) will work, i try to forget cp1251 encoding.

-- TWiki:Main.VictorKasatkin - 29 May 2008

ItemTemplate
Summary TinyMCE breaks Cyrillic Encoding (cp1251 or KOI8-R)
ReportedBy TWiki:Main.OlegButovich
Codebase 4.2.0
SVN Range TWiki-5.0.0, Sun, 09 Mar 2008, build 16496
AppliesTo Extension
Component WysiwygPlugin
Priority Normal
CurrentState Closed
WaitingFor

Checkins TWikirev:16598 TWikirev:16599 TWikirev:16830 TWikirev:16831
TargetRelease patch
ReleasedIn 4.2.1
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng test1.PNG r1 manage 8.8 K 2008-03-19 - 14:35 OlegButovich  
PNGpng test2.PNG r1 manage 11.4 K 2008-03-19 - 14:53 OlegButovich  
Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r23 - 2008-08-04 - KennethLavrsen
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback