Zope

Carlos is continuing work, with some help from the community, on the ZODB book.  Currently you can see daily updates at http://zodb.readthedocs.org.  The new API section, http://zodb.readthedocs.org/en/latest/api.html lists the entire ... [More] zodb/transactions API on one page -- thanks Dag Odenhall. Now if we could get richer descriptions on the method's.  We are looking for volunteers.  Some suggested topics:Conflict resolution BTree usageZEO aka "Client Storage"Demostorage usage and real world examplesPickleCacheAll work is on github, https://github.com/cguardia/ZODB-Documentation [Less]

Interested in the latest news about the upcoming Plone Conference 2011 in San Francisco?  Visit ploneconf.org . The Conference website, put together by a team of volunteers, provides the most recent information about the conference, as well as the ... [More] ability to sign up and pay for the event.  Head there soon to make sure you can take advantage of their earlybird discount.  Only a limited number  of tickets are available at the earlybird rate, and the conference has been capped at 400 attendees - so act quickly. Make sure you bookmark the site, as it will be the hub for submitting proposals for talks and training at the event - as well as the place for the event schedule once the choice has been made on speakers and classes to be offered. [Less]

Need a sarcastic response to a support interaction as an open source project maintainer? Look no further!

How to get back to square one with a Plone buildout If all else fails, reboot. Or, in the base of a buildout, delete any generated files and start again. The commands below should remove everything and allow you to run a fresh buildout. Run this from ... [More] the root of the buildout directory, and don't worry if you get a few "file not found" warnings. $ rm -rf .installed.cfg .mr.developer.cfg parts/ eggs/ develop-eggs/ $ find . -name '*.pyc' -or -name '*.egg-info' | xargs rm -rf $ bin/buildout -n Finally, if you are using a shared buildout directory and a previous buildout had network trouble, it's possible that you'll have a partially downloaded .zip or .tar.gz or .egg file that is corrupt and confuses buildout. If so, you can remove the offending file from the dist/ directory in your download cache, e.g.: $ rm ~/.buildout/downloads/dist/foo.bar.zip or, if you want to clean up all downloaded source archives: $ rm ~/.buildout/downloads/dist/* This should be safe, since any egg that's actually used will have been turned into a .egg file/directory anyway. Check your ~/.buildout/default.cfg file if you've forgotten where your download cache is located. [Less]

Yeah, Django is awesome. You have to know that I use Django every days at work. But... there's some points that I really dislike. I guess I'm the best french django opponent. Now I can show this post when people ask me why. Django is monolithic ... [More] Everybody know that. No discussion is needed. Django has a real community Some Django's third party products are great. But all contribution made for Django are only for django. Django has it own middleware specification even if the pep 333 exists. That's sucks. What if I want to use all the great stuff with my own wsgi framework ? Next point: Django use svn. What ? This old VCS ? Why I can't contribute by forking on github or bitbucket ? Is that a real modern community ? Django templating can't contain some logics Oh yeah, that's a great philosophy. But I'm a mature developer and I want to use a simple modulo in a template without having to write a 20 lines template tag. I promise that I don't put so much logics in my template. You can use Jinja! Yes, I can. But then I need to forget all the great stuff provided by third party products. So, in fact, I can't. Webob is great but Django's Request/Response is better Web0b is a 200% well tested library. It just work. Why this is not used by Django ? Is it so hard ? Some people tried to include that library in Django but... it just fail. See this page. No longer maintained... I guess the author switch to another framework. Packaging Is that really exist ? No, It doesn't matter. All my python code should go in app/ then I can't give you can checkout my code in your apps/ folder. no problem. After a few years passed with Tarek "el packager" Ziade I became sick when I see the code generated by Django's templates. Django has view class !! That's the good news of the latest release. Awesome ! That's just made me remember Zope 2. I guess that this post explain this better than I can. EOP End Of Post. I guess I've pointed all things I dislike in Django. Or maybe it's just a work in progress... Django still great. But it can be better without so much effort. Use WebOb... Add a setup.py to project/app templates... Use a Pyramid like renderer to allow more than one template engine... Use github... Please, just do it, Django people! [Less]

Yeah, Django is awesome. You have to know that I use Django every days at work. But... there's some points that I really dislike. I guess I'm the best french django opponent. Now I can show this post when people ask me why. Django is monolithic ... [More] Everybody know that. No discussion is needed. Django has a real community Some Django's third party products are great. But all contribution made for Django are only for django. Django has it own middleware specification even if the pep 333 exists. That's sucks. What if I want to use all the great stuff with my own wsgi framework ? Next point: Django use svn. What ? This old VCS ? Why I can't contribute by forking on github or bitbucket ? Is that a real modern community ? Django templating can't contain some logics Oh yeah, that's a great philosophy. But I'm a mature developer and I want to use a simple modulo in a template without having to write a 20 lines template tag. I promise that I don't put so much logics in my template. You can use Jinja! Yes, I can. But then I need to forget all the great stuff provided by third party products. So, in fact, I can't. Webob is great but Django's Request/Response is better Web0b is a 200% well tested library. It just work. Why this is not used by Django ? Is it so hard ? Some people tried to include that library in Django but... it just fail. See this page. No longer maintained... I guess the author switch to another framework. Packaging Is that really exist ? No, It doesn't matter. All my python code should go in app/ then you can checkout my code in your apps/ folder. no problem. After a few years passed with Tarek "el packager" Ziade I became sick when I see the code generated by Django's templates. Django has view class !! That's the good news of the latest release. Awesome ! That's just made me remember Zope 2. I guess that this post explain this better than I can. EOP End Of Post. I guess I've pointed all things I dislike in Django. Or maybe it's just a work in progress... Django still great. But it can be better without so much effort. Use WebOb... Add a setup.py to project/app templates... Use a Pyramid like renderer to allow more than one template engine... Use github... Please, just do it, Django people! [Less]

A couple of smallish projects I had lined up for late June/July got delayed to August/September, so I’m currently free for small projects (days-weeks). I am always looking for big projects (weeks-months), and Colliberty can also take on huge projects ... [More] (2-3 developers for months) and project manage them. We do both direct consulting and subcontracting, and everything is done remotely (unless you have loads of money). Contact me at [email protected] or [email protected]. Filed under: plone, python [Less]

Whither is archaic English for "where are we going", "what comes next"? So: what does the future hold for django? About releases: there's about 10 months between releases. The good news is that we're getting closer and closer to the ... [More] actual planned release date every time. 1.3 was only a few weeks late. Backwards compatibility is high on django's list. We're proud of it. Sometimes things changes, but with a limited amount of effort you can get everything working yet. Another major thing to be proud of: security. No CSRF problems. No XSS. No database injections. Everything is real safe. What's going to be in django 1.4? For certain (as they're already committed into trunk): Class-based form wizards. Signed cookies. Configurable admin filters. Possible (because it depends on work that still needs doing, note also that this list is not fixed as Russell cooked it up himself): App refactor. (Jannis mentioned the following two links in twitter when I asked about this: https://github.com/jezdez/django/tree/app-loading and https://code.djangoproject.com/browser/django/branches/soc2010/app-loading ). More class-based views. Lazy foreign keys. ... which means configurable User models! Probably not (as it needs more work and 1.4 features ought to be virtually finished right now): Anything from gsoc 2011: Schema migration. Compiled templates. Composite keys. Template based forms. NoSQL/Query refactor (gsoc 2010). Well, what's beyond 1.4? Look at what we're missing. Well, there's nothing that's fundamentally missing. There was a list from a 2004 flickr keynote. Most of those shortcomings have actually been adressed. Trends to watch There are 5 interesting trends that django might have to take a look at. But note that django itself influenced the way people looked at the web. You had php and zope and both were completely different. Django was a new paradigm (just like rails and turbogears). Way smaller than the big huge zope. Completely different mechanism from php. Better packaging tools. When other frameworks are mentioned in this context, django is often mentioned as monolithic (and if they're particularly nasty, they'll use the term "not invented here"). But now the packaging landscape starts to become quite good. We have the packaging tools now. We have to use it. We could split things up a bit more. Perhaps more fine-grained release cycles. Perhaps you could swap in a different template mechanism. Deployment tools. PHP script uploading is easy. Not so with python. There's lots of work happening, but we're not there yet as a broader python web community. Look at java: you can create a .war file and you can deploy it. Where is our .war file? Services like http://ep.io spring up, but are your deployments portable between those services? Not at the moment. Time for some standardisation? Should we push that as a community? Microframeworks. How on earth can an april fools joke like Flask get actual traction? Turn into a popular framework? Django is lots and lots smaller than zope, but these new ones are even smaller. What is small? What is micro? Could we adopt some? Can we become more attractive? We should think about this. Project merges. In nature, explosions of variations sometimes re-converge again. Many recent python web frameworks are merging. BFG + pylons for instance, into pyramid. Should we as django look at that too? Start using common bits and pieces? Swap out bits that are available now as generic components that weren't available when django started? Concurrency. Why is node.js getting so much press? Hopefully not because javascript is such an nice language to write in. Django is not in a fantastic position to do node-js like things. How does django fit into this concurrent picture? There are things in our stack that lend itself to more concurrency, like database querying. Time for creative thinkwork. Whither or wither? Whither means "which way are we going". Wither means "dying on the vine". Do we adapt? Where are we going? How do we keep capturing all developers' imagination? Zope was successful but it is withering now. It isn't capturing developers' interest anymore. Watch out for inertia. We have backwards compatibility, but this means we're a bit big. And it means that most work is maintenance and corner cases that need proper fixing. Not many high-stepping high-visibility heroic new things. Django is stable now! And that means a lot of work. So we need volunteers. Django needs more. It is good that they added more core contributors last year. Again he asks us to work with the core developers, not against them. Ask friendly. Python 3 The elephant in the room. Now is the time to start being serious about this. (Note by me: read Armin Ronacher's recent talk at PyGrunn as background material.) It also means that we have to start looking at django 2. Just some raw brainstorming by Russell: Python 3? (Note: Jannis pointed me at https://bitbucket.org/loewis/django-3k/ for some previous work in this area). Getting rid of the settings singleton? Based on pyramid? More modular? These aren't finished discussions! This is something we all have a vested interested in. The community should participate A LOT in this discussion. Django is yours, so participate! [Less]

Participants: Andrew Godwin, Andy McKay, Jesper Noehr, Eric Florenzano. What are the common mistakes you see. Things that take a long time ("sending an email") that lock up a process. Using the filesystem for caching. Testing locally and ... [More] pushing it live and see it fall over. Using an external API can also take a long time. Work with timeouts to work around it. Scaling wise, what should you think about before scaling becomes an actual issue? And what should you definitely leave until the moment comes you need to scale? First things first: use a queue like celery, even on small sites. Get used to such a queue and have it in place, that'll help a lot with performance later. Make sure you've got your database schema is mostly OK. It doesn't have to be perfect right away, but at least mostly OK. Expect something to change and assume you'll have to swap something out later to improve the performance. One important aspect of scaling is measurement and profiling. What are the best practices and good tools for doing that in production? Bitbucket has a middleware that switches on with a special query string and that starts up the python c profiler and gives them data on the request. The debug toolbar is a great help in development. For realtime stats graphite and statsd are an option. Or munin or kakti for real-time generic server information graphs. Logging. Always set up logging. Look at the logfiles and figure out what happened. Opennms, pingdom, munin, nagios, django-kong were mentioned as monitoring tools. Puppet vs Chef vs Whatever for provisioning servers in a Django stack. Fight! Puppet is good. Chef is good. Puppet is alright. (So: not much of a fight :-) ) Django ORM: how much of an issue is this going to be when I want to scale? It is much less an issue than it used to be. You'll only get to know the hotpoints for YOUR application when you run into them. When you optimize beforehand, the points will be different than those you'll really hit. And then there are ways to solve it. Caching, asynchronous, less joins, splitting things, etc. You can denormalize, too. Simple: check your indexes. Do you have the right ones? Are you missing ones? Also changing your actual database server configuration default values can make a lot of difference. Spend two days figuring out all the options. And check postgres for rediculously low default memory values. Incremental roll-outs help with detecting problems. When all your 15 new instances suddenly die, you know you need to change something. Considering that using a caching proxy, like for instance Varnish, is commonly used for improving performance/scalability, are there any options out there for Django which handle cache invalidation in a good manner that you know of? Use etags. Most caching is dependent fully on your individual app. So something generic is virtually impossible. Varnish gives you lots of control. You can invalidate pages from your python code. So set up a couple of proper database triggers. Is Django fast enough? Should more attention be on speed and benchmark tests? Yes and yes. It is fast enough, but we should watch it. Django is fast enough. If you want to scale, scale over multiple boxes instead of building out one single box. But: watch out that django doesn't get any slower! Code deployment to web workers: there are lots of different ways, can we get the groups thoughts on the best practices? By hand. Pip. But it is a bit slow. Now they use github (with a local git mirror for their sites). Fabric. Simple bash script that ssh-s to the server and that updates everything. HAProxy helps in getting a server offline and getting it transparently back up after the update. If you were starting a new project today, which Python VMs would you consider? Probably cpython as an ops person would probably not allow us to run pypy. But I'm watching pypy and it looks good. What's the worst scalability failure story you've ever heard? Running postgres with 32MB of memory (a default setting...). A sysadmin that, to prove his valid point, pulled the electricity plug out of the live machine. He won. Returning a string instead of an iterator in a wsgi script that was getting lots of hits. One character at a time... What do you use to find slow sql queries? Django debug toolbar. Another trick is to evaluate querysets early to better see what's going on (as querysets are lazily evaluated). Use mysql/postgres's configuration option to log slow queries. How do you mimick a large load and can you simulate it? Use apachebench, but keep in mind that that won't be a "perfect" worst-case load. The other answers were mostly "we can't do that". Incremental roll-outs help. Key question: can you respond quickly? Can you deploy quickly. How to handle database rollbacks when you rollback a release? Most either don't use south or they don't do rollbacks. A migration can only ADD columns or tables. They're never removed. Never. Addition-only. This way the old code can talk just fine to the new database structure. Suggested reading: always ship trunk. Which wsgi runner do you use? Mod_wsgi is not out of date or slow in any way, it works just fine. Gunicorn is awesome. Especially the built-in asynchronous mode and eventlet can help a lot if you use it. What's your best experience regarding scalability? A php site. 0.5MB traffic to 100 MB of traffic within in a month. It teached him a lot. A plone site 8 years ago. Plone was two request-a-second at that time. They had squid in front. It was the oxfam site that was used for post-tsunami donations. He was Real Happy with squid that day. Some multimedia website. From 0 to 8 million users in one year. Learning on the job! How to deal with backfilling data? After adding tables, you sometimes have to fill them with default data. How to do it without killing your server? Use celery and use a management command to slowly push small batches unto your task queue. [Less]

I have recently changed jobs (hence the lack of posts recently), switching from 3 years of dedicated open source Python development, mainly in the Zopes and Plone, to Microsoft ASP.NET. I thought that a post is some of my observations on the ... [More] transition would be interesting.ASP.NET has a very different ethos to any of the Python frameworks I have used in the past. It is far more frontend orientated, with most of the functionality tied in some way to the HTML page. The typical way to construct a page in ASP is to embed (pre-built) ASP html controls into a page’s markup (using XML). Examples of the controls are textbox, label, image, table etc. There are python frameworks that do this, it’s not that unusual, but ASP takes it to a whole new level.One of the most useful functions of a control is that it is capable of automatically maintaining its own state throughout multiple POSTS and reloads. Fill in a text box and on page reload you text will still be there without you having to code anything.Control can also disseminate datasources, such as SQL, Arrays or even web services. A table control can be linked to a SQL datasource which will then render as a HTML table of data without writing a single line of actual code.The power of these controls far surpasses those available in python frameworks, especially if you start buying commercial controls. A table control can by default render a HTML table capable of allowing end user to filter, group, sort and export the data. Nice ajax updates are also relatively simply. Link the table control to an ajax manager control and any functions that would previously have been a POST get transformed to ajax. As a nice example you can generate graphs from a datasource control using the charting controls in about 2 minutes.So where is the down side, there has to be one right? Yes there is. The main problem with ASP.NET is actually where Python shines: in the elegant control of information from source to page. ASP may well be able to do very clever things quickly and simply on the front end, but controlling the data through a full lifecycle can be a pain as soon if you want the least bit of customisation.How do you check for specific errors that occur when fetching data on one of those fancy automated datasources? - You have to subscribe to its events. What if I want to change how and when the data is fetched? – You have to hope the control exposes the functionality you want and documents it. In comparison Python frameworks almost universally provide a very consistent model for the organisation of data, from your SQLAlchemy model, to your queries to your View and data marshalling up to the template. It can be done nicely is ASP but it takes more time and design to get it correct.Python’s organisational architecture is far superior to ASP’s and only getting better with clean and simple design patters (decorators), WSGI, Clean object models, Views, Routes etc. ASP’s out of the box front end elements are equally as superior to anything Python frameworks has to offer, the likes of Deform are a nice clean, Pythonic start but don’t get anywhere near to power of their ASP equivalents. [Less]