Skip to content. | Skip to navigation

Personal tools
Log in
Sections
You are here: Home Blogs Myroslav Opyr

Comments and impressions of what is going on around in the world of Plone Open Source CMS, written by Myroslav Opyr. Blog entries include analysis of open source technologies current state as well as overview of some interesting open source services and techniques.

Decoupled Python Refcounters

Memory page that is not tackled (is not written into but only data are read from) is a good computer citizen. If such page was already paged out to Swap, in case of Memory pressure it is just discarded. The computer bus between CPU cache and RAM will not be occupied by writes out. In case of such page being shared by multiple processes, it makes huge saving of RAM and without copy-on-write operations (as page is not written into) the use of such pages has no CPU/bus overhead, and improves utilization of CPU cache memory.

In Python every PyObject has refcounter in its header. Every "touch" of python object makes inc/dec of that structure member. Each such operation makes CPU caches dirty, occupies bus to RAM with flushes, and makes no sense in sharing the page between processes, as single incref will make the page to do copy-on-write operation.

What if refcounters are decoupled from PyObjects into their own arenas? To shift hotpoint from sparse refcounter fields of PyObjecs into tightly packed counter arrays (arenas) with PyObject having only largely immutable index into arena. The approach would have slightly larger RAM footprint (due to refcounter arenas), slightly slower incref/decref operation (another couple of pointer resolutions), and small overhead during construction/destroy operations of PyObject (as refcounter arena free/busy bitmap should be maintained).

What would such approach bring to and what it'll break in Python?

Document Actions

Zope at Google AppEngine

GAE exposed a challenge to run Zope in severe environment

When Google released GAE, a lot of folks in Zope world turned their sights to Google because Python, the heart of Zope was the language GAE was talking in then. Preliminary experiments showed that Zope and namely ZODB, persistence machinery behind Zope had little chances to survive at GAE. Folks succeeded into porting component architecture there, in running with non-ZODB storage, and in many other initiatives.

We set the goal high: Run Zope3 at GAE with ZODB and persistency. After 9 month of growth together with GAE, its SDK and service we've had basic zope3, worldcookery and grok' adder running at Google. We've learned a lot about GAE at its limits, were puzzled with a lot of obscure moments. Experienced unexpected and unreported kills. Rewrote launch and processing machinery several times.

There are many details to what was accomplished and how it was accomplished, you can see some from slides below. Zope3 is far from being production ready for GAE, but it works! We've even came with quick-n-dirty buildout that transforms Zope into GAE deliverable with single bin/buildout. Comments are welcome.

Topical discussion at http://chatterous.com/zope-gae.

Document Actions

Tendencies in Multi-core computing

Recently there were loud requests to remove to remove GIL in Python 3k.

They've produced healthy discussion upon the topic:

I was pointed to a good writeup upon Hardware/Software overview of tendencies in multi-core computing. It sheds some light upon progress made in the area touched by Guido in different languages/systems/platforms.

 

If we’re going to live in a world where CPUs are cheap and parallelism is the norm, then we need to think in those terms. If we need Perl/Python/Ruby/etc. programs to interact with parallelized libraries written in C/C++/Fortran, where’s the harm in spawning another process? Let the two halves of the program communicate over some IPC mechanism (sockets, or perhaps HTTP + REST). That model is well known, well tested, well-understood, widely deployed and has been shipping for decades. Plus, it is at least as language-agnostic as Parrot hopes to become. (+2 points if the solution uses JSON instead of XML.)

Fourth, there’s Patrick Logan, who rightly points out the issue simply isn’t about a multi-core future, but also a multi-node future. Some applications will run in parallel on a single machine, others will run across multiple nodes on a network, and still others will be a hybrid of both approaches. Running programs across a network of nodes is done today, with tools like MapReduce, Hadoop and their kin.

If you have a grid of dual-core machines today, and need to plan out how to best use the network of 64-core machines you will have a decade from now, here’s a very simple migration plan for you: run 32x as many processes on each node!

With that said, here is my recipe for taming the multi-core dust bunny:

  • Determine what kind of parallelism makes sense for you: none, flyweight, fine-grained or coarse grained.
  • Avoid troublesome low-level concurrency primitives wherever possible.
  • Use tools like GHC’s Nested Data Parallelism for flyweight concurrency (one program, lots of data, spread out over multiple CPUs on a single machine).
  • Use tools like GHC’s Software Transactional Memory for lightweight concurrency (many interoperating processes managing shared data on a single machine).
  • Use tools like MapReduce and friends for heavyweight concurrency (work spread out across multiple cooperating processes, running on one or many machines).
Document Actions

Open/close events of KSS Opener

How to apply custom reaction to open/close events of KSS Opener. How to disable default expand/collapse behaviour.

Besides defining ":opener-init" even that initializes opener, developer has ability to extend/override open/close events:

.term:opener-init{
evt-init-elementSelector: '.details';
}

.term:opener-open{
action-client: executeCommand;
executeCommand-commandName: replaceInnerHTML;
executeCommand-commandSelector: '#status';
executeCommand-html: 'expanded';
}

.term:opener-close{
action-client: executeCommand;
executeCommand-commandName: replaceInnerHTML;
executeCommand-commandSelector: '#status';
executeCommand-html: 'collapsed';
}

and HTML for the samle is:

<hr />
<div id="status">Current status</div>
<hr />

<div class="term">
History
<div class="details">
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Vestibulum
iaculis eros eu purus. Integer accumsan leo id lorem viverra vulputate.
Donec feugiat nunc molestie massa nonummy pulvinar. Proin porta pede
sit amet lectus. Duis leo urna, tempor non, condimentum condimentum,
commodo non, libero. Integer feugiat, pede at.
</div>
</div>

The resulting behaviour on page is:

The important thing is that on open and close you can have client-side or server-side actions. You can disable default expand/collapse behaviour, with preventDefault parameter:

.term:opener-open{
preventDefault: true;
}
Document Actions

Controlling Controller of KSS Opener

KSS Opener offers several configurable parameters that enables its real-world use. One of the most useful ones is ability to control where controller (small clickable handle) is injected in the page.

In the case you need your opener controller to be inserted into specific place, you can use pair of controllerPlacementSelector parameters:

dl:opener-init{
evt-init-elementSelector: '> dd';
evt-init-controllerPlacementSelector: '> dt';
}

if the above sample is applied to following HTML code:

<dl>
<dt>History</dt>
<dd>
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Vestibulum
iaculis eros eu purus. Integer accumsan leo id lorem viverra vulputate.
Donec feugiat nunc molestie massa nonummy pulvinar. Proin porta pede
sit amet lectus. Duis leo urna, tempor non, condimentum condimentum,
commodo non, libero. Integer feugiat, pede at.
</dd>
</dl>

And the result is:

The whole definition list has opener assigned. Controller (clickable handle) is inserted into <dt> element, not in root of list (that can confuse some browsers). This is why I avoided dl/dt/dd example in my introduction post.

Document Actions

Introduction of KSS Opener

KSS introduced revolutionary concept into Ajax/CSS development. However out-of-the-box feature set is limited, while offering quite good extensibility. Opener is KSS component that extends KSS selectors with the set of events to enable collapsible behaviour.

After installation product offers following event:

opener-selector:opener-init {
evt-init-elementSelector: element-selector;
}

All elements that matches opener-selector will get controller that will offer expand/collapse functionality with mouse click.

Real-life example is:

.term:opener-init {
evt-init-elementSelector: '.details';
}

and having HTML code:

<div class="term">
History
<div class="details">
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Vestibulum
iaculis eros eu purus. Integer accumsan leo id lorem viverra vulputate.
Donec feugiat nunc molestie massa nonummy pulvinar. Proin porta pede
sit amet lectus. Duis leo urna, tempor non, condimentum condimentum,
commodo non, libero. Integer feugiat, pede at.
</div>
</div>

 

will produce into something like:

Document Actions

History of KSS Opener

In times of Plone1 there was linear (produced with list of html elements that had different class to represent embedding) Navigation Tree in Plone. We've done several attempts to rewrite it into nested html structures, then to improve tree generator, then to make it dynamic.

In Plone1 the navigation tree, nested structure by its nature and semantics, was rendered into ugly list of <a> tags that had just "navlevelX" class that made it possible for designers to style by some extent.

Then we had proper <ul><li> nested structure that was huge step forward.

Then we had icons eliminated, that is another big step (not perfect yet, complex hover/active styles still makes life of designer hard).

And now we have Plone-3.0 knocking our doors. The most prominent feature everybody will be talking about is Ajax in Plone. Some will be mentioning KSS that made thing happen for complex CMS.

We had some efforts to address each of the items above, but none of them has matured enough to find its way into Plone code-base. And now we are going to address Ajax aspect of navigation tree.

Document Actions
Blogger: Myroslav Opyr

myroslav.jpg

Leave Testimonial

go here

Our RSS Feeds

Subscribe to our RSS feeds:

rss2-icon.png Quintagroup Blog
rss2-icon.png Python Blog
rss2-icon.png Plone Products
rss2-icon.png CMS.info Blog

Plone PDF brochure

Tag Cloud