Freetag - 8bit-tags list in entry-editor

LazyBadger · Post by **LazyBadger** » Sun Jun 13, 2010 11:50 am

OK, I can understand, why pure 8bit tags are big problem for european developers, but - English isn't single (and even most used) language in the world.
But everybody with 8 or more bits alphabet want to see tags on his native language sorted and ordered in the same style as it happens with english tags, which he can't get for now (updated to 3.12.2 and re-checked), because it's smth. like this

Code: Select all

|A: adobe, |B: bluetooth, |D: DVCS, |F: FIDO, |G: Git, |H: headsets, |M: Mercurial, |N: nokia, |�: я.ру, шутки, телефоны, фразы, фантастика, филология, фото, русский язык, религия, |P: pdf, pdf reader, |S: SCM, sony, |W: wireless, |�: Анонс, АИ, Россия, Тарковский, Яндекс, аудио, аудиофильское, агрегатор, алармизм, анонс, бред, байки, блоги, гарнитуры, детектив, злое, история России, мысли, масскульт, мифы, мониторы, на злобу дня, новость, псевдолитература, па, История,

and even worse - tags output is browser-dependent (Safary, FireFox, Opera,QTWeb tried)
I don't ask fix it ASAP, I just ask "Where I can read code, which generate such output?" (tried to read sources, no luck)

Post by **garvinhicking** » Sun Jun 13, 2010 12:49 pm

Hi!

I believe the problem is that the first character is used with substr instead of checking for mb_substr(). The sorting of the keys is done with PHPs sorting, which in fact really might not properly order by UTF-8 bytemarks.

I'll try to look into this the next days!

Regards,
Garvin

LazyBadger · Post by **LazyBadger** » Sun Jun 13, 2010 3:05 pm

OK. Please kill also lowercasing of tags, or made it as option (will most possible choices)

Post by **Don Chambers** » Sun Jun 13, 2010 5:09 pm

LazyBadger wrote:OK. Please kill also lowercasing of tags, or made it as option (will most possible choices)

Forcing tags to be lower case already IS an option in the freetag event plugin.

LazyBadger · Post by **LazyBadger** » Sun Jun 13, 2010 10:44 pm

Forcing tags to be lower case already IS an option in the freetag event plugin

Mea culpa, found it now. Tnx

Post by **garvinhicking** » Wed Jun 16, 2010 10:08 am

Hi!

The freetag plugin in fact already uses mb_strtolower. Can you check that your PHP has the mb* functions enabled?

Regards,
Garvin

LazyBadger · Post by **LazyBadger** » Wed Jun 16, 2010 4:34 pm

Garvin, I'll slightly lazy to write code, but - phpinfo() tells me:
mbstring
Multibyte Support enabled
Multibyte string engine libmbfl
Multibyte (japanese) regex support enabled
Multibyte regex (oniguruma) version 4.4.4
Multibyte regex (oniguruma) backtrack check On

Post by **garvinhicking** » Wed Jun 16, 2010 10:36 pm

Hi!

Can you create a test.php file with:

Code: Select all

<?php
echo mb_strtolower('teststring');

?

Regards,
Garvin

LazyBadger · Post by **LazyBadger** » Wed Jun 16, 2010 11:55 pm

garvinhicking wrote: Can you create a test.php file

Yes, but it's not so easy for UTF8-texts. All OK with 8bit ANSI. Maybe you'll tell me, whichpart of of code (and where) I can verify and I'll write full report here?

Post by **garvinhicking** » Thu Jun 17, 2010 10:16 am

Hi!

I'm not sure I understand. Does the test.php code work or does it produce errors? Does it not properly lowercase your UTF-8 characters?

If a UTF-8character cannot be lowercased, it should be shown as the original character...

Regards,
Garvin

LazyBadger · Post by **LazyBadger** » Thu Jun 17, 2010 4:53 pm

garvinhicking wrote: I'm not sure I understand. Does the test.php code work or does it produce errors? Does it not properly lowercase your UTF-8 characters?

I can't understand results of my test too - and it drive me nuts
Test code (extended version)

Code: Select all

<?php
echo('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">');
echo('<HTML>');
echo(' <HEAD>');
echo('  <TITLE> Test of mb_ functionality v.1.5</TITLE>');
echo('  <META NAME="Generator" CONTENT="EditPlus">');
echo('  <META NAME="Author" CONTENT="LazyBadger">');
echo('  <META NAME="Keywords" CONTENT="test, mb_ functions">');
echo('  <META NAME="Description" CONTENT="Test Page">');
echo('  <META HTTP-EQUIV="CONTENT-TYPE" CONTENT="TEXT/HTML; CHARSET=UTF-8" />');

echo(' </HEAD>');

echo(' <BODY>');
if (is_callable('mb_strtolower'))
                    echo mb_strtolower('mb_strtolower is callable- Пробные РаЗные ТЕксты; ');
                else
                    echo('mb_strtolower NOT callable!!! ');

echo ('Pure UTF8 - Пробные РаЗные ТЕксты; ');
echo strtolower('strtolower - Пробные РаЗные ТЕксты');
 echo('</BODY>');
echo('</HTML>');
?>

saved as "UTF8 without BOM" file (yes, stupid code, but it must work AFAIK for our needs) doesn't show lowercased texts as readable texts - see http://www.lazybadger.ru/test.php

If a UTF-8character cannot be lowercased, it should be shown as the original character...

Not in my case. Is it hosting-related issue? If yes, we can close my bug-report with "not a bug" resolution

Post by **garvinhicking** » Thu Jun 17, 2010 5:36 pm

Hi!

Let's not give up yet. Your code is good to make further tests.

Try to add a:

Code: Select all

mb_strtolower('...your strings...', 'UTF-8');

This should instruct mb_strolower to recognize your original encoding as UTF-8 code. It seems as if your server is setup to have a internal mb-encoding not equal to UTF-8.

Also you might want to add a:

Code: Select all

$cur = mb_internal_encoding();
mb_internal_encoding('UTF-8');
echo "Internal encoding set to UTF-8, was: $cur<br />\n";

before any calls to mb_strtolower.

Regards,
Garvin

LazyBadger · Post by **LazyBadger** » Thu Jun 17, 2010 7:34 pm

Code: Select all

Internal encoding set to UTF-8, was: ISO-8859-1

Result (OK)

Code: Select all

mb_strtolower is callable- пробные разные тексты; Pure UTF8 - Пробные РаЗные ТЕксты;

Second parameter for for mb_strtolower without using mb_internal_encoding() produce the same result as above (good)

Post by **garvinhicking** » Thu Jun 17, 2010 8:44 pm

Hi!

Excellent. I patched the plugin to add that call:

http://php-blog.cvs.sourceforge.net/vie ... sion=1.141

Regards,
Garvin

LazyBadger · Post by **LazyBadger** » Fri Jun 18, 2010 1:28 am

garvinhicking wrote: Excellent. I patched the plugin to add that call:

Patch applied, but nothing changed in output. Local browser's cache was cleaned before testing, sure

Freetag - 8bit-tags list in entry-editor

Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor

Re: Freetag - 8bit-tags list in entry-editor