charset problems (utf8) after upgrading php (5.3->5.5)

Found a bug? Tell us!!
Post Reply
User avatar
kleinerChemiker
Regular
Posts: 765
Joined: Tue Oct 17, 2006 2:36 pm
Location: Vienna/Austria
Contact:

charset problems (utf8) after upgrading php (5.3->5.5)

Post by kleinerChemiker » Sat Feb 08, 2014 12:11 pm

Some days ago I changed php from 5.3 to 5.5. Now I have a problem with the germanumlauts. But only umlauts, that are saved in the database, not umlauts that are stored in the lang-files. After changing back to 5.3 the problem is gone. With 5.4 the problem is the same.
blog: blog.kleinerchemiker.net

So, daoes anybody havean idea, what's causing this problem?
tia

User avatar
garvinhicking
Core Developer
Posts: 30009
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by garvinhicking » Sat Feb 08, 2014 5:13 pm

Hey,

that's really odd. I would understand when changing from MySQL version to another, but PHP should not be involved in umlauts getting changed.

Is your PHP 5.3 or 5.5 maybe using overloaded mbstring functions? Do the php.ini files differ?

Actually, only the $serendipity["dbNames"] should influence umlauts. Apart from that, maybe try using livehttpheaders or some other browser issue to inspect the HTTP headers of your blog, see if the Charset differs from one PHP to the other?

It could be that one PHP version is using a different mysql libclient, so maybe changing the dbNames value could influence that...

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

User avatar
kleinerChemiker
Regular
Posts: 765
Joined: Tue Oct 17, 2006 2:36 pm
Location: Vienna/Austria
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by kleinerChemiker » Sat Feb 08, 2014 6:18 pm

Thanks for your help, I found the difference. 5.3 ist speaking with mysql in latin1 but 5.5 ist speaking in utf8. Both use mysqli. Aktivating "Datenbank-Zeichensatzkonvertierung aktivieren" doesn't help. Maybe its possible to add an config option for the used charset and set it with "mysqli_set_charset($mysqli, "utf8")" after the connection is established.

User avatar
garvinhicking
Core Developer
Posts: 30009
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by garvinhicking » Sat Feb 08, 2014 7:22 pm

Hi!

That would mean your database contains latin1 charsets? If you enable that option while using 5.3, supposedly you'd get the same charset trouble, right?

A solution would be to fix the encoding of your database. What you could try is to use mysqldump with the latin1 charset, then edit the resulting .sql file and replace latin1 occurences with utf8_general_ci inside the collations, and then reimport the dump. Maybe, for safety, better into a seperate database so that you keep the original one intact.

(Yes, this will be a hassle, happens with installations with older mysql/php version that had latin1 default charsets and then were updated with utf-8 versions...)

Some other suggestions maybe here: http://stackoverflow.com/questions/5842 ... et-charset

Maybe even you can fix the tables with little PHP scripts like these: http://www.inmotionhosting.com/support/ ... e-to-utf-8 or http://wiki.typo3.org/UTF-8_support#Con ... e_to_UTF-8 (this applies not only to TYPO3 but generally to any PHP/MySQL app).

(I want to stress - always test this on copies of your current database, once you dumpe the files you might dump them in the wrong charset and by overwriting the db you might never get the original back...)

HTH,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

User avatar
kleinerChemiker
Regular
Posts: 765
Joined: Tue Oct 17, 2006 2:36 pm
Location: Vienna/Austria
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by kleinerChemiker » Sat Feb 08, 2014 7:27 pm

Thanky, I will have a look and report ih/how I fixed it. Afair there could be the problem, that the data in utf8 colums need doule the space of latin columns, but I will see.

User avatar
garvinhicking
Core Developer
Posts: 30009
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by garvinhicking » Sat Feb 08, 2014 7:36 pm

Yeah; the upside of this is that the data inside your DB should already be UTF-8 encoded (else it wouldn't show up properly in a PHP 5.3 environment) - so what you'll need to fix is only the mechanism that instructs mysql to transform data again into UTF-8 that already is UTF-8 (which is what the "SET names utf8" does, mysql detects a collation "latin1" field, and converts to UTF-8, no matter if the data therein already is utf8).

HTH,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

User avatar
kleinerChemiker
Regular
Posts: 765
Joined: Tue Oct 17, 2006 2:36 pm
Location: Vienna/Austria
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by kleinerChemiker » Sun Feb 09, 2014 8:24 pm

The database is utf8. The problem seems to be the connection between httpd and mysqld. I was latin1 and now is utf8. Somehow this makes problems. I wil try to install phpmyadmin with php 5.3, export the database and import it with php5.5. Hope that helps.

Only other way is to set the charset on connecting to mysqld, but than I have to edit the core files.

User avatar
garvinhicking
Core Developer
Posts: 30009
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by garvinhicking » Mon Feb 10, 2014 3:31 pm

Please check out the stuff I wrote about the collations, this should be your real issue. The mysql metadata needs to be set to UTF-8.

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

orpobanana
Posts: 1
Joined: Sun Nov 02, 2014 5:38 pm

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by orpobanana » Sun Nov 02, 2014 5:42 pm

I had exactly same problem and I was almost downgrading to PHP 5.3. This problem was really frustrating to debug. I figured out that if you use second parameter (link identifier) in any mysql-commands they don't work in PHP 5.5. I removed them and everything started to work again. Both

mysql_set_charset('utf8');
mysql_query("SET NAMES utf8");

did the job. I was using link identifier parameter only in these which made debugging even harder.

urbano
Posts: 1
Joined: Thu Jan 12, 2017 10:24 pm

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by urbano » Thu Jan 12, 2017 10:29 pm

In my case the problem was of the function php htmlspecialchars that in the 5.3 takes by default the charset of the system or the one defined in defaul_charset in the php.ini and between 5.4 and 5.6 is UTF8 if nothing is indicated to him.

The solution is to find all the occurrences of that function and add the charset parameter.

User avatar
garvinhicking
Core Developer
Posts: 30009
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany
Contact:

Re: charset problems (utf8) after upgrading php (5.3->5.5)

Post by garvinhicking » Wed Jan 25, 2017 12:31 pm

Just as a sidenote, in recent Serendipity versions that is fixed so that it works properly with PHP 5.5...
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

Post Reply