UnicodeDecodeError

[anonymous]

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 148, in dispatch
    populate_hdf(req.hdf, self.env, req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 68, in populate_hdf
    hdf['trac'] = {
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, util.escape(value))
  File "/usr/lib/python2.4/site-packages/trac/util/html.py", line 95, in escape
    text = unicode(text).replace('&', '&') \
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 4: ordinal not in range(128)

I use mod_python for trac-devel -r 3025. This is my apache setting.

SetHandler mod_python
PythonHandler trac.web.modpython_frontend
PythonOption TracEnv /my/TracRepo
PythonOption TracLocale "ko_KR.utf8"
SetEnv PYTHON_EGG_CACHE /my/EggCache

If i delete "PythonOption TracLocale "ko_KR.utf8"", The error message is not occured.

[rbrown@…]

This happens to me also, and rolling back to r3022 makes the problem go away.

We generally use the en_GB locale, however removing references to locale in the apache config did not cure the problem for us.

[cmlenz]

This is a problem in our date formatting

[cmlenz]

Patch

[cmlenz]

Can you try the attached patch and let me know whether it fixes the issue?

[tdkim@…]

Thanks for patch. I tested Trac -r 3032(head) and -r 3030.

Before appling the patch.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 148, in dispatch
    populate_hdf(req.hdf, self.env, req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 68, in populate_hdf
    hdf['trac'] = {
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, markup.escape(value))
  File "/usr/lib/python2.4/site-packages/trac/util/markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 4: ordinal not in range(128)

After appling the patch.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 162, in dispatch
    chrome.populate_hdf(req, chosen_handler)
  File "/usr/lib/python2.4/site-packages/trac/web/chrome.py", line 262, in populate_hdf
    for category, name, text in contributor.get_navigation_items(req):
  File "/usr/lib/python2.4/site-packages/trac/web/auth.py", line 70, in get_navigation_items
    yield ('metanav', 'login',
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 4: ordinal not in range(128)

Whether I switched off the "PythonOption TracLocale "ko_KR.utf8"" or not, The second error is occured.

I think that this is a new problem about UnicodeDecodeError.

[cmlenz]

I guess this is related to your login name containing non-ASCII characters, right?

[cmlenz]

Previously posted patch committed in [3035], which should fix the issue with date/time formatting.

[cmlenz]

Patch for the login name issue

[cmlenz]

I've attached another patch that should hopefully help with non-ASCII login names. Can you please try that again? Thanks.

[tdkim@…]

Thanks for your patch. But I use ASCII login names(AlphaNumeric?).

I tested it for trac -r 3040.

Before the patch, the same error is occured at auth.py. After the patch, the error message is only

'NoneType' object has no attribute 'decode'

Whether I switched off the "PythonOption TracLocale "ko_KR.utf8"" or not, the same error is occured.

[tdkim@…]

Please ignore my previous posting. I need more testing for the problem.

After testing, I will post the test result. Thank you...

[tdkim@…]

The test result using trac-3040 with the patch.

'NoneType' object has no attribute 'decode'

The test result using trac-3040 without the patch.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 283, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 170, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/wiki/web_ui.py", line 109, in process_request
    self._render_history(req, db, page)
  File "/usr/lib/python2.4/site-packages/trac/wiki/web_ui.py", line 343, in _render_history
    req.hdf['wiki.history'] = history
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 233, in add_value
    add_value('%s.%d' % (prefix, idx), item)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, markup.escape(value))
  File "/usr/lib/python2.4/site-packages/trac/util/markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 2: ordinal not in range(128)

[anonymous]

It seems like this is happening in many modules. I also have this with changesets:

Traceback (most recent call last):
  File "c:\python23\Lib\site-packages\trac\web\main.py", line 290, in dispatch_request
    dispatcher.dispatch(req)
  File "c:\python23\Lib\site-packages\trac\web\main.py", line 170, in dispatch
    resp = chosen_handler.process_request(req)
  File "c:\python23\Lib\site-packages\trac\versioncontrol\web_ui\changeset.py", line 218, in process_request
    diff_args, diff_options)
  File "c:\python23\Lib\site-packages\trac\versioncontrol\web_ui\changeset.py", line 456, in _render_html
    diffs = _content_changes(old_node, new_node)
  File "c:\python23\Lib\site-packages\trac\versioncontrol\web_ui\changeset.py", line 428, in _content_changes
    ignore_space_changes='-b' in options)
  File "C:\Python23\lib\site-packages\trac\versioncontrol\diff.py", line 188, in hdf_diff
    line = space_re.sub(htmlify, escape(line, quotes=False))
  File "c:\python23\Lib\site-packages\trac\util\markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

[Markus Tacker <m@…>]

Use comma+space in cc field

[PBruin]

Thanks for the correction.

I rolled back to r3022 and the problem is gone. 0Xc3 is indeed a character that appears in one of the text files that was changed in the changeset that caused the Oops.

[cmlenz]

This should be fixed in [3084].

[Markus Tacker <m@…>]

Sorry, the error still persists.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 175, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 86, in process_request
    resp = self._render_view(req, db, id)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 254, in _render_view
    req.hdf['%s.real' % prefix] = col[0]
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 218, in add_value
    self.hdf.setValue(prefix, markup.escape(value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 1: ordinal not in range(128)

[Johannes Rudolph <johannes_rudolph@…>]

I'm getting this error:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.3/site-packages/trac/web/main.py", line 175, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 241, in process_request
    diff_args, diff_options)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 479, in _render_html
    diffs = _content_changes(old_node, new_node)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/web_ui/changeset.py", line 451, in _content_changes
    ignore_space_changes='-b' in options)
  File "/usr/lib/python2.3/site-packages/trac/versioncontrol/diff.py", line 198, in hdf_diff
    line = escape(line, quotes=False).replace('\0', '<del>') \
  File "/usr/lib/python2.3/site-packages/trac/util/markup.py", line 114, in escape
    text = text.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid data

[cboos]

I can confirm that, for me at least, [3084] fixes the issue.

However, there's problably a better way to avoid the problem, as in this situation we try to explicitely convert both file contenst to utf8 (in _content_changes(old_node, new_node)).

As we already discussed, we should try to take into account the file encoding before comparing them. I'll see what I can do in this area.

[cboos]

Please try out the following patch, it works for me even without r3084:

Index: changeset.py
===================================================================
--- changeset.py	(revision 3090)
+++ changeset.py	(working copy)
@@ -420,16 +420,21 @@
             The list is empty when no differences between comparable files
             are detected, but the return value is None for non-comparable files.
             """
-            data = old_node.get_content().read()
-            if is_binary(data):
+            old_content = old_node.get_content().read()
+            if is_binary(old_content):
                 return None
-            old_content = mimeview.to_utf8(data, old_node.content_type)
 
-            data = new_node.get_content().read()
-            if is_binary(data):
+            new_content = new_node.get_content().read()
+            if is_binary(new_content):
                 return None
-            new_content = mimeview.to_utf8(data, new_node.content_type)
 
+            old_cset = mimeview.get_charset(old_content, old_node.content_type)
+            new_cset = mimeview.get_charset(new_content, new_node.content_type)
+            # character sets should be 'iso-8859-15' at the very least
+            assert old_cset and new_cset
+            old_content = unicode(old_content, old_cset)
+            new_content = unicode(new_content, new_cset)
+
             if old_content != new_content:
                 context = 3
                 options = diff_options[1]

I'll follow-up with a more complete patch, which will also take care of the textual diff view.

[cboos]

Patch on top of r3091 for doing the version control related diffs using unicode strings

[Johannes Rudolph <johannes_rudolph@…>]

OK, the small patch did certainly help. But I have some strange non-deterministic behaviour, sometimes it works without patch, sometimes it doesn't. My working copy seems to be somewhat corrupted...

[cboos]

... but with the patch, does it always work or do you still have errors?

[Johannes Rudolph <johannes_rudolph@…>]

Yes, that was the only version which worked all times.

[athomas]

I think this should also be applied to util.markup.Fragment:

trac/util/markup.py

@@ -317 +324 @@
                 yield escape(child, quotes=False)
 
     def __str__(self):
-        return Markup(''.join(self.serialize()))
+        return Markup(''.join(self.serialize())).encode('utf-8')
 
     def __add__(self, other):
         return Fragment()[self, other]

[cboos]

Well, Alec, I don't think so.

We have to convert unicode strings to plain str with a specified encoding only at clearly defined times, like when sending text to the browser (convert to utf8) or sending a generated mail (convert to the configured encoding), because we can't switch back and forth between encoded strings and unicode as the charset used for the encoding is not remembered.

If you make an object's __str__ return an UTF-8 encoded string, next time you'll call unicode on that, you'll most likely get an exception:

>>> class txt(object):
...    def __str__(self):
...      return u'été'.encode('utf-8')
...
>>> str(txt())
'\xc3\xa9t\xc3\xa9'
>>> unicode(txt())
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

with 'ascii' replaced by whatever is your sys.getdefaultencoding().

As for the patch above, I'll complement it with some fallback in the style of trac.util.to_utf8, in case there's still an UnicodeError exception raised. This can happen if a wrong charset has been associated to the file, using the svn:mime-type property.

[cboos]

The issues with TracChangeset and TracBrowser modules are fixed by r3092. It should be noted that with that fix, r3084 is not needed, so I wonder if we should revert it in or not, since assuming an 'utf-8' encoding could lead to or mask other errors...

I also saw that Markus reported errors with the report module, which are probably still there, so I'll leave this ticket opened.

[Markus Tacker <m@…>]

Right. In r3092 the error still persists.

[PBruin]

I confirm that the problems I had with TracChangeset are resolved with r3092. Thanks.

[athomas]

Christian, that makes sense. However, I've been testing with the PEC data and #2868 causes an exception at source:sandbox/workflow/trac/web/clearsilver.py on line 235. markup.Markup is treated explicitly, but markup.Element is not, which I believe it should be. My previous patch fixed this.

I'm seeing this in WorkFlow though, so I could be not encoding something correctly, but my thought was that if it's already got explicit handling for Markup it probably should for Element?

Traceback (most recent call last):
  File "/home/athomas/projects/trac/workflow/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/home/athomas/projects/trac/workflow/trac/web/main.py", line 175, in dispatch
    resp = chosen_handler.process_request(req)
  File "/home/athomas/projects/trac/workflow/trac/ticket/web_ui.py", line 188, in process_request
    self._insert_ticket_data(req, db, ticket, reporter_id, fields)
  File "/home/athomas/projects/trac/workflow/trac/ticket/web_ui.py", line 469, in _insert_ticket_data
    req.hdf['ticket.%s' % field['name']] = field
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 194, in __setitem__
    self.set_value(name, value, True)
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 236, in set_value
    add_value(name, value)
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 228, in add_value
    add_value('%s.%s' % (prefix, k), value[k])
  File "/home/athomas/projects/trac/workflow/trac/web/clearsilver.py", line 235, in add_value
    self.hdf.setValue(prefix, str(value))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 232-237: ordinal not in range(128)

[cboos]

As for a special treatment for Element (Fragment even), I'm all for it, but it should rather be done like this:

clearsilver.py

@@ -213 +213 @@
                 self.hdf.setValue(prefix, str(int(value)))
             elif isinstance(value, markup.Markup):
                 self.hdf.setValue(prefix, value.encode('utf-8'))
+            elif isinstance(value, markup.Fragment):
+                self.hdf.setValue(prefix, unicode(value).encode('utf-8'))
             elif isinstance(value, str):
                 if escape:
                     self.hdf.setValue(prefix, markup.escape(value))

[athomas]

Beautiful! Works perfectly.

[tdkim@…]

I have the problem. I tested it in Trac -r 3104.

Traceback (most recent call last):
 File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch\n    result = object(req)
 File "/usr/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 86, in handler\n    gateway.run(dispatch_request)
 File "/usr/lib/python2.4/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
 File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 311, in dispatch_request\n    req.hdf['error'] = {
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 194, in __setitem__\n    self.set_value(name, value, True)
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 238, in set_value\n    add_value(name, value)
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 230, in add_value\n    add_value('%s.%s' % (prefix, k), value[k])
 File "/usr/lib/python2.4/site-packages/trac/web/clearsilver.py", line 220, in add_value\n    self.hdf.setValue(prefix, markup.escape(value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\\uc774' in position 0: ordinal not in range(128)

If I apply the patch, there is no problem.

trunk/trac/web/clearsilver.py

@@ -217 +217 @@
                 self.hdf.setValue(prefix, unicode(value).encode('utf-8'))
             elif isinstance(value, str):
                 if escape:
-                    self.hdf.setValue(prefix, markup.escape(value))
+                    self.hdf.setValue(prefix, markup.escape(value).encode('utf-8'))
                 else:
                     self.hdf.setValue(prefix, value)
             elif isinstance(value, unicode):

Please, confirm the patch.

[cboos]

Thanks for the patch, applied in r3105.

[cboos]

The latest changeset (r3107) hopefully fixes all the remaining issues (it does fix the report/CSV issue, at least).

I've also sent a mail to Trac-Dev in order to discuss r3084.

[Markus Tacker <m@…>]

I still have the error with a report.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 300, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 176, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 86, in process_request
    resp = self._render_view(req, db, id)
  File "/usr/lib/python2.4/site-packages/trac/ticket/report.py", line 335, in _render_view
    prefix = 'report.items.%d.%s' % (row_idx, unicode(column))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

This is the report entry from the db

11||Termine|SELECT

   id AS ticket,
   strftime('%d.%m.%Y', d.value) AS Fälligkeit,
   round((julianday(d.value) - julianday('now')))||' Tage' AS verbleibend,
   summary AS Beschreibung,
   summary AS _summary,
   description AS _description_,
   strftime('%s', d.value) AS _changetime,
   p.value AS __color__

  FROM ticket t, enum p

  LEFT OUTER JOIN ticket_custom d ON
       (t.id=d.ticket AND d.name='due')

  WHERE status IN ('new', 'assigned', 'reopened')
  AND t.type = 'Termin'
  AND p.name = t.priority AND p.type = 'priority'
  AND d.value != ''
  AND d.value > date('now')

  ORDER BY d.value ASC, p.value|Zeigt alle Termine nach ihrem [wiki:TicketFaelligkeit Datum] sortiert an.

[cboos]

Patch on r3107 -- fixes the report error for me.

[cboos]

Thanks for giving us report entry. While testing it, I could reproduce the error, but I had a slightly different error than you.

Can you try the following patch: attachment:another_2905_fix.diff

First, there's a fix for cursor.description which used to give back an utf-8 encoded string (fixed pysqlite2 only so far).

Then, the "Fälligkeit" value was used in the HDF name, a situation rather unusual...

[Markus Tacker <m@…>]

Yeah, the patch attachment:another_2905_fix.diff works.

[cboos]

Ok, I applied a slightly different fix in r3109. Can you test it too, please?

[Markus Tacker <m@…>]

r3109 works too. Thanks a lot.

[Markus Tacker <m@…>]

Uh. Found another one using the TicketQuery Macro.

Error: Macro TicketQuery(type=Termin&status=new|assigned|reopened) failed
'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

[Markus Tacker <m@…>]

And a third on in ticket view

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 300, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 176, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/web_ui.py", line 222, in process_request
    ticket = Ticket(self.env, id, db=db)
  File "/usr/lib/python2.4/site-packages/trac/ticket/model.py", line 35, in __init__
    self.fields = TicketSystem(self.env).get_ticket_fields()
  File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 99, in get_ticket_fields
    for field in self.get_custom_fields():
  File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 118, in get_custom_fields
    field = {
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 835, in wiki_to_oneliner
    OneLinerFormatter(env, absurls, db).format(wikitext, out, shorten)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 765, in format
    result = re.sub(self.wiki.rules, self.replace, result)
  File "/usr/lib/python2.4/sre.py", line 142, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 610, in replace
    return getattr(self, '_' + itype + '_formatter')(match, fullmatch)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 342, in _lhref_formatter
    return self._make_link(ns, target, match, label)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 349, in _make_link
    escape(label, False)))
  File "/usr/lib/python2.4/site-packages/trac/util/markup.py", line 107, in escape
    text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

[cboos]

Markus, the error just above is related to #2972. There's a patch there which I believe will work.

For the error with the TicketQuery macro, I've not been able to reproduce it, but it's probably the same cause as the other one.

[Markus Tacker <m@…>]

Right, the patch in #2972 fixed both errors. Thanks.

[anonymous]

Removing myself from cc list.

[cboos]

I think that as of r3141, there are no errors remaining (particularly thanks to r3137, which should fix the error with TicketQuery above)

[tdkim@…]

I used trac-r3213(dev) of trunk.

When I set the due date to the schedule of milestone, the problem is occured.

I use "2006년 03월 30일" as due date(YYYY년 MM월 DD일). This is a korean date/time format.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 299, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 196, in dispatch
    raise HTTPInternalError(e.message)
  File "/usr/lib/python2.4/site-packages/trac/web/api.py", line 46, in __call__
    Exception.__init__(self, '%s %s (%s)' % (self.code, self.reason,
UnicodeEncodeError: 'ascii' codec can't encode character u'\ub144' in position 4: ordinal not in range(128)

[cboos]

Yes, I could reproduce this too.

Can you please upgrade to r3215 and try if that fix works for you?

[tdkim@…]

Thank you.

I tested the date input of "timeline", "due" and "completed" in r3215. The UnicodeEncodeError was fixed.

But new error is occured.

Internal Error
2006년 04월 30일 is not a known date format.

I think that it is similar to #2182

[cboos]

Yes, that's another issue then, cmlenz just scheduled it for 0.11, to be part of our i18n effort.

I'll close this one for now (once more :) ).

[freesoitj@…]

I am sorry to reopen this ticket but I have an issue with UnicodeDecodeError too with 0.10.2 on Fedora Core 6 (x86-64). I am french and I have used non ascii names for component (at least I suppose I have since I can not see them anymore).

When I upgraded , quite a long time ago, new ticket and roadmap stopped working :

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 387, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 238, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/trac/ticket/web_ui.py", line 129, in process_request
    ticket = Ticket(self.env, db=db)
  File "/usr/lib/python2.4/site-packages/trac/ticket/model.py", line 37, in __init__
    self.fields = TicketSystem(self.env).get_ticket_fields()
  File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 123, in get_ticket_fields
    options = [val.name for val in cls.select(self.env, db=db)]
  File "/usr/lib/python2.4/site-packages/trac/ticket/model.py", line 541, in select
    for name, owner, description in cursor:
  File "/usr/lib/python2.4/site-packages/trac/db/util.py", line 40, in __iter__
    row = self.cursor.fetchone()
  File "/usr/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line 73, in fetchone
    return row and self._convert_row(row) or None
  File "/usr/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line 69, in _convert_row
    return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0]
  File "/usr/lib64/python2.4/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: invalid data

When I go in WebAdmin, I can not edit components and when I use trac-admin, it is the same...

Trac [/var/trac/xxx]> component list
Command failed: 'utf8' codec can't decode bytes in position 1-2: invalid data
Trac [/var/trac/xxx]> 

Do you have any idea how I can recover?

[cboos]

(Sorry for the delay)

You have to do some manipulations at the SQLite db level. Delete and replace the faulty components, rename their occurrences in ticket and ticket_change tables.

We already discussed about being tolerant with wrongly encoded strings in the database, and jonas vetoed using replacement chars (the rationale being that those errors shouldn't be silently ignored and must be fixed at the db level).

(restoring the fixed status for the 0.10 milestone)