Item5457: TinyMCE breaks Cyrillic Encoding (cp1251 or KOI8-R)

My setting in LocalSite.cfg.

$TWiki::cfg{Site}{Locale} = 'ru_RU.cp1251';
$TWiki::cfg{Site}{CharSet} = 'cp1251';

I have also tried with KOI8-R.

-- TWiki:Main/OlegButovich - 19 Mar 2008

This is representative of the general problem with UTF-8 encodings, reported elsewhere. Duplicate of Item5314.

-- TWiki:Main.CrawfordCurrie - 27 Mar 2008

CP1251 and KOI8-R are single-byte encodings BTW, not unlike ISO-8859-1, except that almost every character in Cyrillic will have the high bit set.

-- TWiki:Main.RichardDonkin - 27 Mar 2008

I think the same... As far as I understand, the same problem should have, for example, Germans with umlauts... like ÌÀ�... May be I am doing something wrong?... Will try to test with umlauts.

-- TWiki:Main.OlegButovich - 28 Mar 2008

OK, I didn't know that, thanks Richard. I'm pretty sure that 8 bit encodings work, as the characters in the symbols drop-down are all 8-bit.

А Б В Г Д Є Ж Ѕ З И І К Л М Н О П Р С Т Ф Х Ц Ч Ш Щ Ъ Ы Ь

-- TWiki:Main.CrawfordCurrie - 28 Mar 2008

I have tested now with euro locale on my server without any problem... I can switch in any order between WYSIWYG and Raw editor, the special symbols are not broken.. That means, there is definitely a problem with russian 8-bit locale :(.. for example, if you look on your АБВ symbols in Raw editor, or just switch from WYSIWYG editor to raw and back, you will see this effect.. Ok, this is understandable on this site, were russian locale is not set... but believe, if will set it, it will not help a lot.. You will face the problem I've described in original bug report. With Euro locales it works perfect, with Russian not..

-- TWiki:Main.OlegButovich - 30 Mar 2008

I've now changed this back to waiting for feedback from Crawford based on Oleg's comment.

-- RichardDonkin - 30 Mar 2008

Thanks Richard.

I see what Oleg is talking about now; the encoding is wierd. Not sure where that is happening. This is not the same as Item5314, it is nothing to do with the locale, and it needs some quality debugging time. Confirmed.

-- CrawfordCurrie - 31 Mar 2008

I believe I finally found the solution. I ended up having to convert octets to UTF-8 to stop the HTML::Parser falling over, then converting UTF-8 wide chars to HTML entities to stop the print falling over (STDOUT is not opened :utf8). I was able to engineer the fix without needing to touch the core code.

Many thanks to TWiki:Main/ThYang, TWiki:Main.TimothyChen, TWiki:Main/OlegButovich and TWiki:Main.RichardDonkin for exploring around the problem and proposing investigative procedures.

-- CrawfordCurrie - 31 Mar 2008

Now it not convert in entities, but convert it in UTF-8 (but I see 8 bit string like P `` T `` T) after saving and after reload page(and firefox set codepage in UTF8 if i reload page ), when i set cp1251 in config

-- TWiki:Main.VictorKasatkin - 21 May 2008

I try old version (31 Mar 2008) and its work fine. something broken after this date

-- TWiki:Main.VictorKasatkin - 22 May 2008

Unless you open a new bug report this one will go into a back hole in a matter of 1 -2 days (then it can no longer be seen among the last 10 changed bug reports)

-- TWiki:Main.KennethLavrsen - 23 May 2008

Please run the test script atatched to TestEncodings. If the tests pass, then I'm pretty sure i can fix these problems, but I need you to confirm it works for you.

-- CrawfordCurrie - 23 May 2008

Never mind, I ran the necessary tests myself, and fixed the plugin.

-- CrawfordCurrie - 25 May 2008

Im run script with cp1251( under suse10.3/apache2/twiki412 altlinux/apache13/twiki420 - firefox2) and set range 0-256, and i not see any russian chars - only \x{HHHH}...

-- TWiki:Main.VictorKasatkin - 28 May 2008

At frist time I can not to see bug, butnow Im find it (31.04.2008 version): when i click on raw button it convert all russian chars to the entities, but save it normaly(after save i see russian chars in source by firefox.)

-- TWiki:Main.VictorKasatkin - 28 May 2008

unfortunattely it saved encoded in entities in data folder(with 31.03.2008 version). Now i set last version plugin and utf8 in config. And all work much better. If another plugin(ExcelExportImport, SVG) will work, i try to forget cp1251 encoding.

-- TWiki:Main.VictorKasatkin - 29 May 2008