Encode utf8 strings. Fixes errors when non-ascii chars are used.

[frizzby]

No description provided.

[avarabyeu]

@frizzby could you please resolve conflicts to be able to merge PR

[krasoffski]

I don't have a problem context and robot framework internal implementation, thus I can be wrong.
From my perspective we need to convert all to Unicode first, than build string from parts and then convert single string to utf8 bytes (encode in python3 returns bytes type).

On python35

>>> s = "Русский"
>>> print(s)
Русский
>>> type(s)
<class 'str'>
>>> rb=s.encode("utf8")
>>> repr(rb)
"b'\\xd0\\xa0\\xd1\\x83\\xd1\\x81\\xd1\\x81\\xd0\\xba\\xd0\\xb8\\xd0\\xb9'"
>>> repr("{0}".format(rb))
'"b\'\\\\xd0\\\\xa0\\\\xd1\\\\x83\\\\xd1\\\\x81\\\\xd1\\\\x81\\\\xd0\\\\xba\\\\xd0\\\\xb8\\\\xd0\\\\xb9\'"'

On python27

>>> s = u"Русский"
>>> print(s)
Русский
>>> type(s)
<type 'unicode'>
>>> rb=s.encode("utf8")
>>> repr(rb)
"'\\xd0\\xa0\\xd1\\x83\\xd1\\x81\\xd1\\x81\\xd0\\xba\\xd0\\xb8\\xd0\\xb9'"
>>> repr("{0}".format(rb))
"'\\xd0\\xa0\\xd1\\x83\\xd1\\x81\\xd1\\x81\\xd0\\xba\\xd0\\xb8\\xd0\\xb9'"

Need to be checked for py2 and py3.
Also python format is able to work with bytes type.

[ailjushkin]

@frizzby This is common problem with Robot Framework, but it should be fixed not with encode.

The reason

It is because Robot Framework works only and only with unicode strings.
Meanwhile, doing "some string {}".format(robot_framework_value) will cause encoding error because python trying to encode unicode (robot_framework_value) to ascii maybe because of some internatioalization options (correct me please).

How to fix it properly

• Please, remove all .encode calls.
• Change all strings to unicode strings. For instance (file model.py):

Before:

    def get_name(self):
        assignment = "{0} = ".format(", ".join(self.assign)) if self.assign else ""
        arguments = ", ".join(self.args)
        full_name = "{0}{1} ({2})".format(assignment, self.name, arguments)
        return full_name[:256]

After:

     def get_name(self):
        assignment = u"{0} = ".format(u", ".join(self.assign)) if self.assign else ""
        arguments = u", ".join(self.args)
        full_name = u"{0}{1} ({2})".format(assignment, self.name, arguments)
        return full_name[:256]

[avarabyeu]

@frizzby @krasoffski Guys, can anyone comment on the state of this RP? Should i close it?

[krasoffski]

Hi @avarabyeu,
Need to check for python3 because it might contain an issue with unicode encoding/decoding.

[krasoffski]

@avarabyeu one question here,
Name is limited by 255 Unicode chars or 255 bytes?

@frizzby,
How to verify/reproduce problem? Non-ascii chars for keyword/docstring/tags?

[avarabyeu]

@frizzby @krasoffski guys, could you please handle this PR. Merge or close it, please.

[ailjushkin]

I leaved above working examples. format cannot concatenate ascii and unicode. robot framework works only with unicode. so we just have to have unicode strings to format it with robot framework values.

[ailjushkin]

@frizzby This is common problem with Robot Framework, but it should be fixed not with encode.

The reason

It is because Robot Framework works only and only with unicode strings.
Meanwhile, doing "some string {}".format(robot_framework_value) will cause encoding error because python trying to encode unicode (robot_framework_value) to ascii maybe because of some internatioalization options (correct me please).

How to fix it properly

• Please, remove all .encode calls.
• Change all strings to unicode strings. For instance (file model.py):

Before:

    def get_name(self):
        assignment = "{0} = ".format(", ".join(self.assign)) if self.assign else ""
        arguments = ", ".join(self.args)
        full_name = "{0}{1} ({2})".format(assignment, self.name, arguments)
        return full_name[:256]

After:

     def get_name(self):
        assignment = u"{0} = ".format(u", ".join(self.assign)) if self.assign else ""
        arguments = u", ".join(self.args)
        full_name = u"{0}{1} ({2})".format(assignment, self.name, arguments)
        return full_name[:256]