You can listen to it right here:
You can read Ben’s take on it as well, on Dev.to: What I learned from interviewing StackOverflow’s top Python contributor, Martijn Pieters
]]>logging module, but the logging module uses blocking I/O when emitting records.
Or does it?
The framework is very flexible in how records are emitted; it is up to the logging handlers that you install. And since Python 3.2, an interesting new handler has been included, the QueueHandler class, which comes with a corresponding QueueListener class. These were originally developed to handle logging in the child processes of the multiprocessing library, but are otherwise perfectly usable in an asyncio context.
The QueueListener class starts its own thread to watch a queue and send records to handlers it manages, so it will not affect the asyncio loop. The QueueHandler handler implementation simply puts records into the queue you specify, after a minimal clean-up operation to ensure records can be serialised easily. This makes this handler entirely non-blocking.
QueueListenerMy strategy is simply to move all root handlers to a QueueListener object before the main asyncio loop starts, and placing a QueueHandler object in their place. From there on out, all blocking operations in handlers are handled in a separate thread, freeing the asyncio loop from having to wait for log records to written out to files or network sockets.
I do customise the QueueHandler class a little, but only minimally so: there is no need to prepare records that go into a local, in-process queue, we can skip that process and minimise the cost of logging further:
import asyncio
import logging
import logging.handlers
try:
# Python 3.7 and newer, fast reentrant implementation
# without task tracking (not needed for that when logging)
from queue import SimpleQueue as Queue
except ImportError:
from queue import Queue
from typing import List
class LocalQueueHandler(logging.handlers.QueueHandler):
def emit(self, record: logging.LogRecord) -> None:
# Removed the call to self.prepare(), handle task cancellation
try:
self.enqueue(record)
except asyncio.CancelledError:
raise
except Exception:
self.handleError(record)
def setup_logging_queue() -> None:
"""Move log handlers to a separate thread.
Replace handlers on the root logger with a LocalQueueHandler,
and start a logging.QueueListener holding the original
handlers.
"""
queue = Queue()
root = logging.getLogger()
handlers: List[logging.Handler] = []
handler = LocalQueueHandler(queue)
root.addHandler(handler)
for h in root.handlers[:]:
if h is not handler:
root.removeHandler(h)
handlers.append(h)
listener = logging.handlers.QueueListener(
queue, *handlers, respect_handler_level=True
)
listener.start()
You could, of course, configure the logging module to use the queue handler to begin with, but I find the above pattern to work better when also using the logging.config to handle handler configuration elsewhere. The above can then simply take an already-configured system and make it suitable for an asyncio environment.
If you run into a problem with some API or Python code what do you do to solve it? I personally throw a few keywords into google, sometimes even before checking the full docs.
Why does this work? Because invariably an excellent conversation and answer from StackOverflow comes back with just what I need.
This week you’ll meet Martijn Pieters. One of the top Python contributors at StackOverflow with over 16,500 questions answered and a reputation of over 500,000.
Relevant links (apart from my social media links):
The specific questions I discussed:
When you want to create a class including a metaclass, making it compatible with both Python 2 and 3 can be a little tricky.
The excellent six library provides you with a six.with_metaclass() factory function that’ll generate a base class for you from a given metaclass:
from six import with_metaclass
class Meta(type):
pass
class Base(object):
pass
class MyClass(with_metaclass(Meta, Base)):
pass
The basic trick is that you can call any metaclass to produce a class for you, given a name, a sequence of baseclasses and the class body. six produces a new, intermediary base class for you:
>>> type(MyClass)
<class '__main__.Meta'>
>>> MyClass.__mro__
(<class '__main__.MyClass'>, <class 'six.NewBase'>, <class '__main__.Base'>, <type 'object'>)
This can complicate your code as for some usecases you now have to account for the extra six.NewBase baseclass present.
Rather than creating a base class, I’ve come up with a class decorator that replaces any class with one produced from the metaclass, instead:
def with_metaclass(mcls):
def decorator(cls):
body = vars(cls).copy()
# clean out class body
body.pop('__dict__', None)
body.pop('__weakref__', None)
return mcls(cls.__name__, cls.__bases__, body)
return decorator
which you’d use as:
class Meta(type):
pass
class Base(object):
pass
@with_metaclass(Meta)
class MyClass(Base):
pass
which results in a cleaner MRO:
>>> type(MyClass)
<class '__main__.Meta'>
>>> MyClass.__mro__
(<class '__main__.MyClass'>, <class '__main__.Base'>, <type 'object'>)
As it turns out, Jason Coombs took Guido’s time machine and added the same functionality to the six library last summer. Not only that, he included support for classes with __slots__ in his version. Thanks to Mikhail Korobov for pointing this out.
The six decorator is called @six.add_metaclass():
@six.add_metaclass(Meta)
class MyClass(Base):
pass
Whenever you need to process a file in-place, transforming the contents and writing it out again in the same location, you can reach out for the fileinput module and use its inplace option:
import fileinput
for line in fileinput.input(somefilename, inplace=True):
line = 'additional information ' + line.rstrip('\n')
print line
There are a few problems with the fileinput module, however. My biggest nitpick with the module is that it has an API that relies heavily on globals; fileinput.input() creates a global fileinput.FileInput() object, which other functions in the module then access. You can of course ignore all that and reach straight for the fileinput.FileInput() constructor, but fileinput.input() is presented as the main API entrypoint.
The other is that the in-place modus hijacks sys.stdout as the means to write back to the replacement file. Obstensibly this is to make it easy to use a print statement, but then you have to remember to remove the newline from the lines read from the old file.
Last, but not least, the fileinput version in the Python 3 standard library does not support specifying an encoding, error mode or newline handling. You can open the input file in binary mode, but output is always handled in text mode. This greatly diminishes the usefulness of this library.
So I wrote my own replacement, using the excellent @contextlib.contextmanager decorator. This version works on both Python 2 and 3, relying on io.open() to remain compatible between Python versions:
from contextlib import contextmanager
import io
import os
@contextmanager
def inplace(filename, mode='r', buffering=-1, encoding=None, errors=None,
newline=None, backup_extension=None):
"""Allow for a file to be replaced with new content.
yields a tuple of (readable, writable) file objects, where writable
replaces readable.
If an exception occurs, the old file is restored, removing the
written data.
mode should *not* use 'w', 'a' or '+'; only read-only-modes are supported.
"""
# move existing file to backup, create new file with same permissions
# borrowed extensively from the fileinput module
if set(mode).intersection('wa+'):
raise ValueError('Only read-only file modes can be used')
backupfilename = filename + (backup_extension or os.extsep + 'bak')
try:
os.unlink(backupfilename)
except os.error:
pass
os.rename(filename, backupfilename)
readable = io.open(backupfilename, mode, buffering=buffering,
encoding=encoding, errors=errors, newline=newline)
try:
perm = os.fstat(readable.fileno()).st_mode
except OSError:
writable = open(filename, 'w' + mode.replace('r', ''),
buffering=buffering, encoding=encoding, errors=errors,
newline=newline)
else:
os_mode = os.O_CREAT | os.O_WRONLY | os.O_TRUNC
if hasattr(os, 'O_BINARY'):
os_mode |= os.O_BINARY
fd = os.open(filename, os_mode, perm)
writable = io.open(fd, "w" + mode.replace('r', ''), buffering=buffering,
encoding=encoding, errors=errors, newline=newline)
try:
if hasattr(os, 'chmod'):
os.chmod(filename, perm)
except OSError:
pass
try:
yield readable, writable
except Exception:
# move backup back
try:
os.unlink(filename)
except os.error:
pass
os.rename(backupfilename, filename)
raise
finally:
readable.close()
writable.close()
try:
os.unlink(backupfilename)
except os.error:
pass
This context manager deliberately focuses on just one file, and ignores sys.stdin, unlike the fileinput module. It is aimed squarly at just replacing a file in-place.
Usage example, in Python 2, with the CSV module:
import csv
with inplace(csvfilename, 'rb') as (infh, outfh):
reader = csv.reader(infh)
writer = csv.writer(outfh)
for row in reader:
row += ['new', 'columns']
writer.writerow(row)
and the Python 3 version:
import csv
with inplace(csvfilename, 'r', newline='') as (infh, outfh):
reader = csv.reader(infh)
writer = csv.writer(outfh)
for row in reader:
row += ['new', 'columns']
writer.writerow(row)
To help with making a large and busy intranet website perform better, we’ve used a light sprinkling of ESI (via Varnish’s ESI support) to improve the cacheabilibty of pages in the site. By delegating assembly of parts of the page to the Varnish cache, pages become much more cacheable as frequently changing chunks such as the personal bar at the top are requested separately.
One such chunk we separated out is the right-hand portlets column. Varnish has been configured to set a special header so that we can detect that ESI is supported:
sub vcl_recv {
...
# Indicate that a varnish capable of doing ESI is in front...
set req.http.X-ESI = "esi";
...
}
Using this header we can then conditionally swap out the portlets column with an <esi:include> statement; this makes site development much easier as we do not have to run Varnish just to see the site working. Here is the relevant section from the main_template.pt file:
<td id="portal-column-two"
metal:define-slot="column_two_slot"
tal:condition="sr">
<div class="visualPadding"
tal:define="
esi_header request/HTTP_X_ESI | nothing;
base context/@@plone_context_state/current_base_url | nothing;
location python:base and base.rstrip('/').split('/')[-1].lstrip('@');
esi python:esi_header and (location not in (
'manage-portlets', 'manage-content-type-portlets'));
queryString request/QUERY_STRING;
queryString python: queryString and '?' + queryString or '';
">
<metal:portlets define-slot="portlets_two_slot">
<esi:include tal:condition="esi"
tal:attributes="src string:${context/absolute_url}/@@right-column${queryString}" />
<tal:noesi condition="not: esi"
replace="structure provider:plone.rightcolumn" />
</metal:portlets>
</div>
</td>
Note that we are making sure that ESI is also not applied when using the portlet management views.
The @@right-column view is simply a template:
<html tal:omit-tag="">
<body tal:omit-tag="">
<tal:block replace="structure provider:plone.rightcolumn" />
</body>
</html>
This whole setup was working swimmingly; we could cache pages for extended periods of times with things like the portlets updating much more frequently and with caching keyed to specific groups of users.
This being a large and complex intranet, it took some time for someone to notice that some lightly-used portlets were no longer showing up. These were portlets that depend on certain content being there, so their absence was not necessarily a problem. However, it was becoming clear that even when their specific conditions were being met, they were not being rendered still. This was quickly narrowed down to the ESI-included portlet rendering; if you bypassed the cache the portlets would show up.
So what went wrong?
Portlets are essentially rendered as part of the Zope viewlet framework. Viewlets are snippets of page output that are looked up by a key consisting of the current context, the current request, the current view and the viewlet manager. Portlets thus have access to these same pieces of information, and you can thus register portlets that only show for certain contexts (particular content types, marker interfaces, etc.), browser layers (usually themes), and even only for specific views or portlet managers (tying the portlet to the left, right or dashboard portlet wells).
With the lesser-known <plone:portletRenderer /> directive, you can also vary the way portlets are rendered for the above keys. Thus, a portlet can look different in different themes, different portlet managers, or when a certain extra marker interface is present on your content objects. This is what had happened to the vanished portlets here; they had been tied to specific views:
<configure
xmlns="http://namespaces.zope.org/zope"
xmlns:plone="http://namespaces.plone.org/plone"
/>
<plone:portlet
name="foobar.portlets.localcalendar"
interface=".localportlet.ILocalCalendarPortlet"
assignment=".localportlet.Assignment"
renderer=".localportlet.Hidden"
addview=".localportlet.AddForm"
/>
<!-- My HQ page -->
<plone:portletRenderer
portlet=".localportlet.ILocalCalendarPortlet"
class=".localportlet.Renderer"
view="foobar.types.browser.mychain.MyChainView"
/>
<!-- My Store page -->
<plone:portletRenderer
portlet=".localportlet.ILocalCalendarPortlet"
class=".localportlet.Renderer"
view="foobar.types.browser.store.StoreView"
/>
</configure>
The above plone:portlet declaration registers a portlet that is hidden by default. The two plone:portletRenderer declarations then assign new renderers when certain views are being used instead. This neat trick allows for the portlet to be targeted very specifically.
This all works great, unless you use a dedicated view for ESI rendering of the portlets. Suddenly the current view is no longer MyChainView or StoreView, but rather @@right-column. Thus the dedicated renderer is skipped in favour of the .localportlet.Hidden renderer, which does what it says on the tin: not render.
The solution is of course to reconstruct the whole context; the @@right-column view already had most things right, only the current view is wrong. With a simple set of TAL declarations we can set up a new value for the view variable when rendering the portlets. Here is the reworked main_template.pt code:
<td id="portal-column-two"
metal:define-slot="column_two_slot"
tal:condition="sr">
<div class="visualPadding"
tal:define="
esi_header request/HTTP_X_ESI | nothing;
base context/@@plone_context_state/current_base_url | nothing;
location python:base and base.rstrip('/').split('/')[-1].lstrip('@');
esi python:esi_header and (location not in ('manage-portlets', 'manage-content-type-portlets'));
viewContext string:?__view_context=${view/__name__};
queryString request/QUERY_STRING;
queryString python: queryString and viewContext + '&' + queryString or viewContext;
">
<metal:portlets define-slot="portlets_two_slot">
<esi:include tal:condition="esi"
tal:attributes="src string:${context/absolute_url}/@@right-column${queryString}" />
<tal:noesi condition="not: esi"
replace="structure provider:plone.rightcolumn" />
</metal:portlets>
</div>
</td>
We use a GET parameter to pass along the name of the view to look up; I’ve used a double-underscore prefix here to reduce the chances we clash with a query string parameter used elsewhere in the site. The @@right-column view then restores this view for portlet rendering (with a fallback to the Plone default view context @@plone):
<html tal:omit-tag="">
<body tal:omit-tag="">
<tal:block
define="viewname request/__view_context | nothing;
viewname python:viewname and '@@' + viewname or '@@plone';
view nocall:context/?viewname"
replace="structure provider:plone.rightcolumn" />
</body>
</html>
Et voilà, our portlets are showing up good and proper again.
]]>Some time ago I had to output some nicely formatted reports from a web application, to be usable offline by Windows users. Naturally, I used the aging but still reliable PyRTF module to generate RTF documents with headers, tables, and a consistent style.
As my application users are mostly Norwegians, however, I quickly discovered that the PyRTF module does not handle international characters (i.e. anything outside the ASCII codepoints), at all. There is no unicode support at all (it has been on the TODO list since forever), let alone converting unicode codepoints to whatever RTF uses to represent international characters.
Recently, a Stack Overflow question reminded me of how I solved this problem at the time, and clearly this question has come up before. Because some approaches I’ve seen can actually produce incorrect or overly verbose output (including pyrtf-ng), I wanted to explain and expand on my solution to provide a definitive answer to the problem, and also see how my original method faired in terms of speed.
Since PyRTF doesn’t filter the text you add to a document at all we can just encode unicode strings ourselves. Lucky for me, the Wikipidia entry on RTF has a fairly detailed section on how RTF handles characters outside the ASCII range. Together with the published RTF 1.9.1 specification (PDF) there is plenty of information on how to encode unicode codepoints to RTF control sequences.
There basically are two choices:
The \'hh control sequence; a backslash and single quote, followed by an 8-bit hexadecimal value. The value is interpreted as a code-point in a Windows codepage, limiting it’s use. You can assign different codepages to different fonts, but you still cannot use the full range of unicode in a paragraph.
The \uN? control sequence; backslash ‘u’ followed by a signed 16-bit integer value in decimal and a placeholder character (represented here by a question mark). The signed 16-bit integer number here is consistent with the RTF standard for control characters, a value between -32768 and 32767.
This control sequence can properly represent unicode, at least for the U+0000 through to U+FFFF codepoints. This sequence was introduced in the 1.5 revision of the RTF spec, in 1997, so it should be widely supported. The placeholder character is meant to be used by readers that do not yet support this escape sequence and should be an ASCII character closest to the unicode codepoint.
The \uN? format is the easiest to produce, especially if you ignore the replacement character (just set it to ‘?’ at all times, surely most RTF readers support the 1.5 RTF standard by now, it’s been out there for 15 years).
A quick search with Google showed me how pyrtf-ng encodes unicode points:
def writeUnicodeElement(self, element):
text = ''.join(['\u%s?' % str(ord(e)) for e in element])
self._write(text or '')
Unfortunately, the above snippet does a few things wrong: it uses a control code for every character in the unicode string, producing output that is at least 5 times as long as the input, and it doesn’t produce negative numbers for codepoints over \u7fff:
>>> example = u'CJK Ideograph: \u8123'
>>> ''.join(['\u%s?' % str(ord(e)) for e in example])
u'\\u67?\\u74?\\u75?\\u32?\\u73?\\u100?\\u101?\\u111?\\u103?\\u114?\\u97?\\u112?\\u104?\\u58?\\u32?\\u33059?'
A recent Stack Overflow answer improved on this by only encoding characters over \u007f (decimal 127) but it still iterates over every character in the string to do so:
>>> ''.join(['\u%s?' % str(ord(e)) if ord(e) > 127 else e for e in example])
u'CJK Ideograph: \\u33059?'
This outputs unicode because codepoints < 128 are left untouched; numbers are not properly converted to signed shorts either. Here is my variation that remedies these things, and dispenses with the str() call:
>>> ''.join(['\u%i?' % (ord(e) if e < u'\u8000' else ord(e) - 65536) if (e > '\x7f' or e < '\x20' or e in u'\\{}') else str(e) for e in example])
'CJK Ideograph: \\u-32477?'
This feels like rather a waste to me, and must be slow as well. I wanted to see how my own solution stacks up against the character-by-character, naive implementation.
While casting around for my own solution, I also looked into the Python codecs module to come up with ideas on how to do this more efficiently. Of course, the codecs provided by that module are all implemented in C, but the unicode_escape codec did produce output quite close to what I needed for RTF; codepoints between \u0020 and \u007f are left alone, the rest are encoded to one of the \xhh, \uhhhh or \Uhhhhhhhh 8, 16 or 32-bit escapes (with the exception of \t, \n and \r). Would there be any way to reuse this output?
Well, if you combine this with a bit of re.sub magic, you can in fact produce convincing RTF command sequences:
>>> import re
>>> import struct
>>> _charescape = re.compile(r'(?<!\\)\\(?:x([0-9a-fA-F]{2})|u([0-9a-fA-F]{4}))')
>>> def _replace_struct(match):
... match = match.groups()
... # Convert XX or XXXX hex string into 2 bytes
... codepoint = (match[0] and '00' + match[0] or match[1]).decode('hex')
... # Convert 2 bytes into a signed integer, insert into escape sequence
... return '\\u%i?' % struct.unpack('!h', codepoint)
...
>>> escaped = example.encode('unicode_escape')
>>> escaped
'CJK Ideograph: \\u8123'
>>> _charescape.sub(_replace_struct, escaped)
'CJK Ideograph: \\u-32477?'
Using the struct module gave me a quick means to re-interpret the hexadecimal notation as produced by the unicode_escape format as a signed short, but I did have to make sure there were 2 bytes at all times.
Of course, the above trick does not handle newlines, returns or tabs (\n, \r and \t respectively) correctly, nor does it escape existing backslashes yet, but I hoped back when that this proof of concept should operate several orders of a magnitude faster than the naive character-by-character method when dealing with mostly-ASCII input; most of the work is done in C by the codecs and re modules, after all.
So this time around I decided to time these:
>>> import timeit
>>> def test1(): ''.join(['\u%i?' % (ord(e) if e < u'\u8000' else ord(e) - 65536) if (e > '\x7f' or e < '\x20' or e in u'\\{}') else str(e) for e in testdocument])
...
>>> def test2(): _charescape.sub(_replace_struct, testdocument.encode('unicode_escape'))
...
>>> declaration = (
... u'Alle mennesker er f\xf8dt frie og med samme menneskeverd og menneskerettigheter. '
... u'De er utstyrt med fornuft og samvittighet og b\xf8r handle mot hverandre i brorskapets \xe5nd.'
... )
>>> testdocument = declaration * 100
>>> timeit.timeit('test1()', 'from __main__ import test1', number=500)
5.982733964920044
>>> timeit.timeit('test2()', 'from __main__ import test2', number=500)
1.4459600448608398
Cool, so my hybrid encode plus regular-expression based solution looks to be around 4 times as fast, at least when it comes to simple Norwegian text with a handful of latin-1 characters, my most common case. Note however that I am not handling the RTF escape characters properly, nor are the \n, \r and \t characters handled correctly.
But I am actually being too clever by half (read: pretty dumb really); why did I encode to unicode_escape in the first place? I was still in the process of fully understanding the issues and saw a shortcut. My regular expression isn’t particularly clever, I dabbled with the struct module to get my signed short values, and with all this hocus-pocus I lost sight of the goal: to escape certain classes of characters to RTF command codes.
But aren’t regular expressions quite good at finding those classes all by themselves? I may as well use a decent expression that selects what needs to be encoded directly:
>>> _charescape_direct = re.compile(u'([\x00-\x1f\\\\{}\x80-\uffff])')
>>> def _replace_direct(match):
... codepoint = ord(match.group(1))
... return '\\u%s?' % (codepoint if codepoint < 32768 else codepoint - 65536)
>>> _charescape_direct.sub(_replace_direct, example).encode('ascii')
'CJK Ideograph: \\u-32477?'
>>> def test3(): _charescape_direct.sub(_replace_direct, testdocument).encode('ascii')
...
>>> timeit.timeit('test3()', 'from __main__ import test3', number=500)
0.5356400012969971
Suddenly we have an 10 times speed increase! Not only that, I am now also properly escaping the three whitespace characers \n, \r and \t, and as an added bonus, the RTF special characters \, { and } are now also being escaped! I call this a result, and a lesson to learn.
We could also use a translation table to do my escaping for me. This is simply a dict that maps unicode codepoints to a replacement value. To create a static dict for all unicode values could be somewhat tricky, requiring either a custom __missing__ method or loading a generated structure on import.
Before digging into clever solutions to that, I should perhaps first test the speed of a simple translation table, one that only covers codepoints up to ‘\u00ff’, or latin-1:
>>> _table = {i: u"\\'{0:02x}".format(i) for i in xrange(0, 32)}
>>> _table.update({ord(c): u"\\'{0:02x}".format(ord(c)) for c in '\\{}'})
>>> _table.update({i: u"\\u{0}".format(i) for i in xrange(128, 256)})
>>> len(_table)
163
>>> timeit.timeit('testdocument.translate(_table).encode("ascii")', 'from __main__ import testdocument, _table', number=500)
2.66812801361084
Unfortunately, using .translate turns out to be slowing us down considerably. Reducing the table to just a few codepoints doesn’t help either:
>>> _basictable = {ord(c): u"\\'{0:02x}".format(ord(c)) for c in '\n\r\t\\{}'}
>>> len(_basictable)
6
>>> timeit.timeit('testdocument.translate(_basictable)', 'from __main__ import testdocument, _basictable', number=500)
2.0113179683685303
So it looks like I might want to avoid using .translate if at all possible.
So far, I’ve compared methods by testing them against some Norwegian text, typical of many European languages with a generous helping of ASCII characters.
To get a more complete picture, I need to test these methods against a worst-case scenario, a UTF-8 encoded test set from a great set of UTF-8 test documents:
>>> import urllib
>>> utf8_sequence = urllib.urlopen('https://raw.github.com/bits/UTF-8-Unicode-Test-Documents/master/UTF-8_sequence_unseparated/utf8_sequence_0-0xffff_assigned_printable_unseparated.txt').read().decode('utf-8')
>>> len(utf8_sequence)
58081
>>> testdocument = utf8_sequence
>>> timeit.timeit('test1()', 'from __main__ import test1', number=10)
0.7785000801086426
>>> timeit.timeit('test3()', 'from __main__ import test3', number=10)
0.8913929462432861
Interesting! So in the worse-case scenario, where the vast majority (99.8%) of the text requires encoding, the character-by-character method is actually a little faster again! But this also means that for most cases, where you insert shorter text snippets into an RTF document, and where a far larger percentage of characters do not need escaping, the regular expression method will beat the character-by-character method hands down.
So far I’ve focused only on characters within the BMP. You can apparently use a UTF-16 surrogate pair, at least according to Wikipedia for codepoints byond the BMP. However, the RTF specification itself is silent on this, and no endian-nes is documented anywhere that I can find. The Microsoft platform uses UTF-16-LE throughout, so perhaps RTF readers support little-endian surrogate pairs too.
However, I cannot at this time be bothered to extend my encoder to support such codepoints. On a UCS-2-compiled python there is a happy coincidence that codepoints beyond the BMP are treated mostly like UTF-16 surrogate pairs anyway, so they are sort-of supported by this method:
>>> beyond = u'\U00010196'
>>> _charescape_direct.sub(_replace_direct, beyond).encode('ascii')
'\\u-10240?\\u-8810?'
Note, however, that the first byte is -10240, or 0xd800 in unsigned hexadecimal, making this a big-endian encoded surrogate pair. Presumably on Windows that’ll encode the other way around.
On a UCS-4 platform the codepoint will be ignored by the regular expression and the .encode('ascii') call will raise a UnicodeEncodeError instead.
I am calling this ‘unsupported’ and a day. Suggestions for implementing this in a neat and performant way are welcome!
I am quite happy with the simple regular expression method, and prefer it over the character-by-character loop.
So I packaged up my regular expression method as a handy module on PyPI, complete with Python 2 and 3 support and a miniscule test suite; the source code is available on GitHub.
The module in fact registers a new codec, called rtfunicode, so after you import the package all you need do is use the new codec in the .encode() method:
>>> import rtfunicode
>>> declaration.encode('rtfunicode')
'Alle mennesker er f\\u248?dt frie og med samme menneskeverd og menneskerettigheter. De er utstyrt med fornuft og samvittighet og b\\u248?r handle mot hverandre i brorskapets \\u229?nd.'
Hopefully it comes in handy for others. Feedback is most welcome, as are patches!
]]>We’ve been experimenting with plone.app.relations to manage relationships between objects for a few years now. This package uses zc.relations to lay the links between content items in your site, which in turn relies on zope.app.intid to indirectly create those links. Basically, intids are pointers to the real objects and lets you handle the linking efficiently.
The relations machinery is not very forgiving if any intid has gone AWOL. Normally, the relations data structures are kept in sync through Zope events, but this doesn’t always work out. In our experience, you can end up with objects and their intids removed, but the relationships pointing to the now-gone intids still in place. When this happens, things break, and you get trackbacks ending in the dreaded KeyError: <long number> in getObject of zope/app/intid/__init__.py. The traceback line before that will be zc/relationship/index.py in the method resolveToken.
Now, the zc.relations package is very powerful and very, very flexible. This comes at a price, as it’s internal data structures are quite daunting to the uninitiated. If you have to repair these relations and all you have is the missing intid at one end of the relation, it’ll be a long hard slug through a maze of 3 or 4 different packages and opaque TreeSets.
Luckily, we already did the deep code dive for you. The following method, if passed an intid, will find any references to it in the relations data structure and remove these for you:
from plone.relations.interfaces import IComplexRelationshipContainer
from zope.app.intid.interfaces import IIntIds
def removeKeyErrorRelationship(iid):
"""Remove all relationships that point to a intid no
longer in the site
"""
intids = getUtility(IIntIds)
relationships = getUtility(IComplexRelationshipContainer,
name='relations')
relIndex = relationships.relationIndex
for direction in ('target', 'source'):
data = relIndex._name_TO_mapping[direction].get(iid)
if not data or data[0].value == 0:
continue # Empty set for this direction
for relid in list(data[1]):
keyref = intids.refs.get(relid)
if keyref is None:
# Not even the relationship exists anymore
relIndex._remove(relid, (iid,), direction)
else:
relation = keyref.object
try:
relation.__parent__.remove(relation)
except AttributeError:
# The relation object only exists in the intid utility;
# in this case __parent__ is None.
relIndex.unindex(relation)
relIndex.unindex_doc(relid) # be doubly sure
intids.unregister(keyref)
Note that this method assumes you already have the local site manager set up properly. This is a great little method to get rid of individual KeyError problems.
It would be better, if you could clear out all missing intids from the relations tool altogether, before they become a problem and things fall down. Luckily, there is! The following code will hunt down and remove all missing intids from the tool. Note that it’ll take a while (it’ll scan through two whole relations indexes), so you better sit back and relax while the work is done.
from plone.relations.interfaces import IComplexRelationshipContainer
from zope.app.intid.interfaces import IIntIds
from BTrees.IOBTree import difference
def clearAllMissingLinks():
"""Find and remove all missing intids in the
relations tool.
"""
intids = getUtility(IIntIds)
relationships = getUtility(IComplexRelationshipContainer,
name='relations')
relIndex = relationships.relationIndex
rtotal = itotal = 0
for direction in ('target', 'source'):
idx = relIndex._name_TO_mapping[direction]
for iid in difference(idx, intids.refs):
itotal += 1
for relid in list(idx[iid][1]):
keyref = intids.refs.get(relid)
if keyref is None:
# Not even the relationship exists anymore
relIndex._remove(relid, (iid,), direction)
else:
relation = keyref.object
try:
relation.__parent__.remove(relation)
except AttributeError:
# The relation object only exists in the intid utility;
# in this case __parent__ is None.
relIndex.unindex(relation)
relIndex.unindex_doc(relid) # be doubly sure
intids.unregister(keyref)
rtotal += 1
return itotal, rtotal
Note that this method returns the total number of intids identified, as well as the total number of relationships removed.
Instead of pumping out the water, we should of course patch the leak. We have yet to find it though, but if we do, we’ll make sure the affected packages receive the patch!
I’ve found that in practice some relationships only were still referenced by intid keyrefs and present in the relationships index, but no longer were present in the relationship utility itself. These have to be manually unindexed and removed; the code examples above have been updated to reflect this.
This article was originally published on jarn.com.
]]>A customer discovered that an important entire section of his site was missing and asked us to bring it back. This was in a heavily edited site, with loads of writes each day, but we quickly located the offending transaction: someone had deleted the object in question 9 days earlier.
Undo was no longer an option, though: too many things had changed, not least the catalog. Truncating the Data.fs (removing all transactions since, including the offending one) was not only undesirable, but impossible as the site stores the data in Oracle through RelStorage.
So, instead of permanently removing transactions, we used a handy little package to do some time traveling: zc.beforestorage.
zc.beforestorage does require a ZODB version 3.8 or 3.9; the customer installation is on Plone 3.0, so a newer ZODB3 egg was necessary for this operation. A small additional buildout configuration file (saved as beforestorage.cfg) helps out:
[buildout]
extends =
buildout.cfg
eggs +=
zc.beforestorage
ZODB3
zope.proxy
[versions]
ZODB3 = 3.8.1
zope.proxy = 3.4.2
[relstorage-patch]
recipe = plone.recipe.command
command =
cd ${buildout:eggs-directory}/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB
curl -s http://svn.zope.de/zope.org/relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch | patch -N -p0
cd ${buildout:directory}
update-command = ${relstorage-patch:command}
[instance]
zope-conf-additional +=
enable-product-installation False
The relstorage-patch section in the above code ensures that our ZODB3 egg is patched with the RelStorage additions, and the zope.proxy egg is needed because ZODB 3.8 requires a newer version. The enable-product-installation line is required because zc.beforestorage puts your ZODB in read-only mode (understandibly); the option tells Zope not to try and write product information to the ZODB.
Once buildout has been run with this configuration (with the -c switch), you’ll still need to edit the zope.conf file for your instance, usually in parts/instance/etc/zope.conf. You need to edit the <zodb_db main> section to wrap the storage in the beforestorage. Ours looked something like this:
<zodb_db main>
# Main database
cache-size 650000
%import zc.beforestorage
%import relstorage
<before>
before 2008-12-08T10:29:03
<relstorage>
<oracle>
dsn RELSTORAGE_DSN
password xxxxxxxxx
user xxxxxxxx
</oracle>
</relstorage>
</before>
mount-point /
</zodb_db>
Any line with the word ‘before’ in it is new. The timestamp we learned from the undo log, simply converted to UTC. Now, when you start the instance, you are in the past. You can’t alter this past (no killing of grandfathers), but you can read it. And lo and behold, the deleted object is back.
Now that we have found the lost object, we can recover it. We simply exported it; in the ZMI, choose the Export/Import button, and save the export on the server. Remove the zc.beforestorage configuration (just run buildout with your regular buildout file), restart, import the .zexp file, done!
Note that you’ll need to reindex the imported content and that any related data that lives outside of the object itself is gone. For example, its intid are gone and all relationships to it will have to be recreated etc. But you just saved your customers bacon, I’m sure they won’t mind a little manual work!
This article was originally published on jarn.com.
]]>By now you all should have installed last Tuesday’s Hotfix. If you haven’t yet, but are running Plone 2.5 or Plone 3.0 websites, you should do so yesterday, or at least as soon as humanly possible.
The Hotfix patches a serious security problem in the statusmessages and linkintegrity modules, where network-supplied data was interpreted as pickles. “Network-supplied” data in this case means both cookies and form data, and no authentication is required to exploit the holes.
The basic problem with the holes is that the Plone community was totally unaware of how dangerous the pickle module really is. Hanno Schlichting did file a report1 a few months ago stating that the code was potentially dangerous, but even he didn’t fully appreciate that pickles are a security hole only waiting for attacker input. The scary thing here is that the code in question was written by extremely capable and experienced developers, but none of them were aware of the fact that you cannot ever use pickles to load user-supplied data.
What is needed then, is education. This is my contribution.
So what is wrong with pickles? They are just a damn handy way to serialize arbitrary data into binary strings and back again, right?
Yes, they are that, but the pickle format used is also a simple stack language that allows the creation of arbitrary python structures, and execute them. This stack language allows you to import modules (the ‘c’ symbol), and apply arguments to callables (the ‘R’ symbol), thus causing code to be run. Combine this with the python built-in methods eval and compile and you have the perfect vehicle for an attacker to have the pickle loader routine execute arbitrary python code when loading a well-crafted pickle. Just image what an attacker could do with that to your Zope server. Do you think you’ll ever be sure you got all the backdoors out of your Data.fs?
So next time you need to preserve data across HTTP requests, please do not be tempted to use the pickle module to create strings for you. Rarely will you have anything more than a handful of simple datatypes to pass along anyway, so just invent a simple dataformat and use that instead. (No, using a subclass of the python implementation of pickle is not a simpler solution).
With statusmessages for example, each message consists of a message and a type string, both unicode. So we changed to a hand-rolled format using a 2 byte length header (11 bits of message length, 5 for the type) directly followed by the message and type strings (encoded to utf-8). When reading this from a cookie again later, the decoder simply has to read the lengths from the first 2 bytes, then read the right amount of characters to get the message and type back. A similar method was used to encode the linkintegrity data. Simple, effective, and impervious to attacks.
Congratulation!!
A.D.2111
All bases of CATS were destroyed.
It seems to be peaceful.
But it is incorrect. CATS is still alive.
ZIG-01 must fight against CATS again.
And down with them completely !
Good luck.
Zero Wing, 1989
This article was originally published on jarn.com.
the original link, http://dev.plone.org/plone/ticket/6943, is unfortunately now lost without trace. ↩