Project

General

Profile

Actions

Bug #2920

closed

Unicode characters not being displayed correctly

Added by Edwin Bentley almost 10 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
-
Start date:
04/09/2014
Due date:
% Done:

0%

Estimated time:

Description

Hi,

We are currently experiencing issues where unicode characters aren't getting displayed correctly.

When we save them to the database, they are correct, and when we read them from the database and wcout them, they are also correct.

When we set the text of a WText, and check that value, they're there, but when it comes to displaying them on the webpage, they are not. We tried changing the textformat of the WText to PlainText, but that didn't solve the issue. I'm guessing the issue is somewhere in where the characters get rendered to html?

Actions #1

Updated by Wim Dumon almost 10 years ago

  • Status changed from New to Feedback
  • Assignee set to Wim Dumon

Hello Edwin,

What is your OS, and what database backend are you using?

Are the strings correctly rendered in a WText before you save them to the database?

What kind of string (Wt::WString, std::string, std::wstring) are you storing in the database?

BR,

Wim.

Actions #2

Updated by Edwin Bentley almost 10 years ago

Hi Wim,

Developing on Opensuse 12.3, and running the http server from there too.

The database backend is mysql and is installed on CentOS 6.3

I've just quickly tried putting some of the problem characters directly into a WText

root()->addWidget(new WText(WString::fromUTF8("(aka "symmetric mode")\")));

The problem characters being the special quotation marks ""

That shows fine in the page oddly enough...

We are saving the string in the database as a Wt::WString.

I'm now wondering whether the issue could be due to some slightly convoluted string conversion, where we basically do -

Some manipulation within QString and QByteArrays.

Save into a Wt::WString via WString::fromUTF8(QString().toUtf8().data()) or WString::fromUTF8(QByteArray().data())

Where if it is in a QByteArray, it's definitely utf8 encoded characters.

We then save that WString into the database - where this all looks fine when we use an mysql tool to view the database.

We then retrieve the WString from the database, and convert back into QString, using QString::fromUtf8(WString().toUTF8()), do some manipulation, and then finally convert it back into a WString using the method above, and set the WText->setText() with it.

As I say, a little complex, but having debugged at all stages of conversion - it looks like the utf8 encoding is being maintained, though maybe you can spot something there that we're doing wrong.

Regards,

Edwin

Actions #3

Updated by Wim Dumon almost 10 years ago

That looks good. What do you see, what kind of 'incorrect' display do you have?

Can you check if your mysql database is configured to store UTF8?

SHOW VARIABLES LIKE 'collation%';

SHOW VARIABLES LIKE 'character_set%';

To convert an existing database I use this method:

http://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8

ALTER DATABASE databasename CHARACTER SET utf8 COLLATE utf8_general_ci;

ALTER TABLE tablename CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

BR,

Wim.

Actions #4

Updated by Edwin Bentley almost 10 years ago

Yeah, the database is using utf8 encoding for both.

where the text should be (aka "symmetric mode")

it's being displayed as (aka “symmetric mode”)

Regards,

Edwin

Actions #5

Updated by Edwin Bentley almost 10 years ago

Hi Wim,

Just found the cause of the issue, it was a problem with our conversion, there was one place where we were reading from a QByteArray and placing into a QString, but not calling QString::fromUtf8().

Sorry for the false alarm, and thanks for the help!

Regards,

Edwin

Actions #6

Updated by Wim Dumon almost 10 years ago

  • Status changed from Feedback to Closed

Great to hear it works :)

BR,

Wim.

Actions

Also available in: Atom PDF