re-motion

Fabian had the news first, but here it is: We wanted to take re-motion to CodePlex already, but LGPL v3 is not a valid license there. We modified the license headers of re-motion from LGPL v3 to "LGPL v2.1 or later”, so we should be cool with ... [More] CodePlex. Here’s the link: relinq.codeplex.com All the stuff, including the Subversion repository, is still here on re-motion.org, and CodePlex has only a few links and downloads. That’s a first step, and we believe that some people will learn about re-linq only through its presence on CodePlex. Later, we plan to decouple re-linq from the rest of re-motion, since there is not a lot of shared code: re-linq uses almost nothing from re-motion, and it should therefore be a separate project, with a separate release cycle. re-store (re-motion’s ORM) uses re-linq and will continue to do so. In upcoming releases, re-store will be treated just like any other downstream project from re-linq’s perspective. [Less]

As Fabian recently posted, re-linq just got its body parts reassembled. This is good news for everyone who wants to create a capable LINQ provider. Fabian has the details, but let me phrase it for mere mortals like myself. Creating LINQ providers ... [More] used to be a pain. It looked easy from a distance, since LINQ is similar to most query languages in many ways (including SQL, NHibernate's HQL, Entity Framework's Entity SQL or even the querying part of XQuery). How hard can it be to transform a LINQ abstract syntax tree (AST) to any of those? The problem is, LINQ won't give you such a tree, but something entirely different that is really hard to translate to anything. The main trick of re-linq is that it gives you exactly what you'd have expected in the first place: an AST that resembles the LINQ statement you wrote. Plus some infrastructure to make your transformations (including visitors and a few stock transformations you're likely going to need) and other stuff (like creating additional queries for eager fetching). Now finally creating a LINQ provider is as simple as you would think. You don't know what the hack transparent identifiers are and how they make your live harder? Now you don't have to. We believe it's perfectly possible to create a very powerful LINQ provider in a few weeks. If you want to support every conceivable LINQ query, it's going to be a bit more, depending on how different your target language is from LINQ. But still, there's a lot in re-linq that helps you achieve even that. Fabian explains how to use re-linq here, and Markus Giegl has a nice sample on CodeProject that shows how to do this. We'll keep you updated. For the brave, here's our latest build. This probably matters most for the NHibernate team. They just released a LINQ provider that supports basic query scenarios, but they plan to switch to a new one that uses their new HQL AST model. Using re-linq, they will hopefully be able to quickly create a provider that surpasses the old one's capabilities by far. Steve Strong is already working on it. - Stefan [Less]

As Fabian recently posted, re-linq just got its body parts reassembled. This is good news for everyone who wants to create a capable LINQ provider. Fabian has the details, but let me phrase it for mere mortals like myself. Creating LINQ providers used to be a pain. It looked easy from a distance, since LINQ […]

As I explained in my last post, the idea of re-linq is to transform ugly IQueryable-based ASTs into nice query expression ASTs (from ... select). A discussion with Frans Bouma, distinguished LINQ tamer, brought up a point we discussed for quite a ... [More] while internally: does this approach limit our ability to express every conceivable LINQ query? And if not, will it still provide value, or will this result in ASTs that are just as hard to use as IQueryable ASTs, only different? Frans brings up the following example: from c in md.Customer.Where(x=>x.Country=="Germany") join o in md.... select c.Orders; Initially we were thinking that re-linq would just lead to a 80+% solution, because using Queriable's extension methods just lets you do things that cannot be expressed directly using query expressions. (That was good enough for us anyway, since all we wanted to do back then was to get serious - but not necessarily complete - LINQ support for re-store, our own ORM. So we just went ahead.) When we thought about turning re-linq into a much more generic beast, we revisited this assumption and discovered that for any direct use of Queryable there is not only a way to express the same using query expressions (from ... select notation). It is also easier to process these expressions, because when we get down to it, using a full LINQ provider is only useful if we transform the query to a powerful enough query language, like SQL, HQL or Entity SQL. And those languages usually support sub-queries, but none of them supports anything even remotely similar than direct invocations on Where(), SelectMany() etc. For Frans' example, that would be: from c in (from x in md.Customer where x.Country=="Germany" select x) join o in md.... select c.Orders; The transformation result might not be shorter, but it's much easier to process and transform because it is structurally similar to the output model, you can use existing transformations in a modular way for sub-queries, and everything strange (like transparent identifiers) has already been dealt with at that point. A more intelligent transformation could transform the whole thing to just from c in md.Customer join o in md.... where x.Country=="Germany" select c.Orders; These queries are semantically identical (i.e., there should only be a difference if you execute these statements imperatively using LINQ to objects). This is a very interesting transformation that does a nice optimization. For any simple SQL-emitting back-end, this would also lead to much cleaner SQL queries without the back-end even being aware of this particular optimization). At the end of the day, this is a matter of how many special cases you want to support. LINQ to SQL is known to go out of its way to produce the simplest SQL possible even for frightening LINQ queries. But some optimizations might be even easier to achieve using re-linq, because the patterns are more easily recognizable. Either way, moving this kind of optimizations into a shared OSS project sounds more promising to me than solving it for every single provider. It doesn't only reduce the overall effort, it also means that all optimizations that are done on the query model itself can be shared between providers. Everyone who thinks that re-linq cannot do some trick that they absolutely need is welcome to contribute to re-linq. (For instance, we here at rubicon are not particularly interested in optimizing LINQ joins, because joins are usually implicit in ORMs, and a good DB like SQL server would recognize the pattern anyway and come up with identical execution plans for both queries. But if someone else builds this optimization (and it won't cost us an arm and a leg in terms of performance), we will gladly use it.) Now we're not there yet, and I cannot promise that we're not going to discover situations that are more difficult to solve than we anticipated, but we're confident that this is not a dead-end approach that will just fail for more complex scenarios. The design behind re-linq's transformations is not trivial, and we are quite confident that it is a solid ground to build on. I'm sure Fabian will find some time to go into internals when he's back at work. - Stefan [Less]

As I explained in my last post, the idea of re-linq is to transform ugly IQueryable-based ASTs into nice query expression ASTs (from … select). A discussion with Frans Bouma, distinguished LINQ tamer, brought up a point we discussed for quite a while internally: does this approach limit our ability to express every conceivable LINQ […]

I'm pretty sure that everyone who has ever tried to create a serious LINQ provider has suffered through the same stages: Fascination: The way C# 3's new features were combined to create query comprehensions are cool. Lambdas, anonymous types ... [More] , extension methods, and all the rest are really useful language features on their own, and the query expression syntax (from ... select) is really just a very thin layer on top of it (although they somehow managed to get Monads into the game). Excitement: Via IQueryable and Expression<Func<...>>, we even get ASTs for the queries we write, and we can process them to generate SQL! Hope: LINQ query expressions look much like SQL, so there must be a simple way to transform one into the other. Frustration: The Queryable class makes a big mess out of those nice-looking query expressions. As we look at more complex queries and read the C# specs, we realize that the transformation from query expressions into method invocations follows some strange rules. They are very necessary for creating expressions that can be compiled and executed in C#, but unfortunately, we get the same bloated expressions for our ASTs. Now transformation to SQL doesn't look so simple anymore. Transparent identifiers anyone? Despair: System.Core does not contain any useful code to deal with those ASTs. LINQ to SQL does, but it doesn't expose it (unfortunately, because LINQ to SQL is one oft the few LINQ providers that can deal with anything you throw at it). Capitulation: We settle with a simple LINQ provider that understands just a primitive subset of LINQ (and throws exceptions for everything else), or we find that other LINQ providers might be good enough. OK, now the last point is an exaggeration. Not everyone surrenders to the complexity of LINQ, or we would not have any useful LINQ providers. But few people actually get past this point, just have a look at Frans Bouma's 15-part series Developing Linq to LLBLGen Pro. And honestly, it's a real PITA that everyone would have to do this from scratch. If we look at the nice syntax that query expressions have, it's really a pity. Why can't we have ASTs that resemble this beautiful syntax? After all, they are so much more like anything we'd want to transform them to (except for compiled code). SQL, XQuery, NHibernate's HQL, Entity Framework's Entity SQL, you name it. Starting last year, we created a LINQ infrastructure that supports our own O/R mapper (re-store), and we started from exactly this point: To reduce complexity, we'd transform LINQ ASTs back into ASTs that resemble the original query expression syntax. (In some cases this is not the original syntax, since the extension methods of Queryable can also be called directly. But in compiled code this makes no difference, so we transform everything into query expressions.) The result is re-linq. If you are thinking about writing a LINQ provider that supports more than just from/where/select, I'd like to encourage you to take a close look at re-link. Also, do not hesitate to contact us about missing features or chances to participate. (Unfortunately, providing public access to our issue tracker and a mailing list are still on our to-do list, but supply will follow demand!) Fabian has written a short paper about re-linq (including open issues), and we're going to publish more information in this blog soon. Stay tuned. - Stefan [Less]

I’m pretty sure that everyone who has ever tried to create a serious LINQ provider has suffered through the same stages: Fascination: The way C# 3’s new features were combined to create query comprehensions are cool. Lambdas, anonymous types, extension methods, and all the rest are really useful language features on their own, and the […]

Hi all, it's Fabian again. In case you're interested in learning more about the details of mixins, I've started a series on my own blog. For now, it consists of three articles: It provides a mixin mechanism What's a re-motion mixin ... [More] , anyway? What can we do for you? (Features of re-motion mixins) The series is bound to grow, so take a look if you're interested. Fabian [Less]

OK, we promised. Here on this blog, and to customers and partners too. Sure, we will release it. Just a few more weeks. Days! Well, this year, for sure. Of course, when you start promising stuff, life makes sure you'll be too busy to keep them. But ... [More] this year, by all means, was not a bold promise. So here I am, on  New Year's Eve, copying the last few license files to our repository: https://svn.re-motion.org/svn/Remotion/trunk/ Q: What took you so long? A: Other stuff got in the way. Plus, this is a large code base, with different licenses for different components. First, we had some funny ideas how we could get away with a single source file license header for multiple licenses, just to make our own lives easier. After dropping this idea, it turned out that we still had to restructure our repository, or nobody would understand which license applies to which part. Q: What do you mean, multiple licenses? You told us it would be LGPL, didn't you? A: The re-motion core framework is licensed under the GNU Lesser General Public License (LGPL) v3. This includes mixins and a lot of stuff that we have not blogged about yet, like our O/R mapper (re-store) and our ASP.NET libraries for data binding (re-bind) and control flow (re-call). This includes everything in https://svn.re-motion.org/svn/Remotion/trunk/Remotion/ There are two components that we're releasing under the GNU Affero General Public License (AGPL) v3: An access control facility (re-strict) and a document management system (re-vision). This includes everything in https://svn.re-motion.org/svn/Remotion/trunk/SecurityManager/ and https://svn.re-motion.org/svn/Remotion/trunk/Dms/ Q: When are you going to provide some documentation about all of those components? A: Next year. Q: Wait: does that mean New Year's Eve again? A: Oh very funny, but fair enough. No. We have a large amount of documentation for re-strict re-store, re-call and re-bind already. Fabian is going to blog a lot about mixins (he promised!). Documentation for re-strict is next on our list. re-vision will take some time, we only have the client-side components checked in yet, the server-side stuff is still in the realm of an individual project's repository, as is the upcoming human workflow library (guess what: re-flow). Q: What are your next steps? A: I believe the most obvious feature missing from re-motion right now is a nice little community site. We will have to overhaul our development process to, so that outside contributors have a chance to catch up with what we're thinking and planning. Q: I cannot access your repository. A: Make sure you don't forget the 's' in https. Enough for today, see you next year! I'm going to see friends who have a little house with a very nice view over Vienna. Let's see how the city celebrates our release! ;-) Happy New Year! - Stefan Edit: It's re-store that we already have documentation for, not (yet) re-strict [Less]

Ayende Rahien started a nice thread via his blog entry Aspects of Domain Design. He shows how a rule could be implemented as an aspect (in the AOP sense) that modifies a method. That was an invitation to discussions about whether this should really ... [More] be done via aspects, and the invitation was well received. In the comments, I remarked that this sounds much more like multi-dimensional separation of concerns to me. After all, AOP as a paradigm focuses on cross-cutting concerns (like "logging, transactions, security, and caching" - Ayende), but not quite as much on individual types and their semantics. The notion of multi-dimensional SoC seems to be buried quite deep in the common wisdom of our profession, although there are quite nice implementations, and people can often see the advantages. Still, its status is that of alchemy, only less known. Some people use AOP frameworks or languages for multi-dimensional SoC, and technically, dedicated frameworks for multi-dimensional SoC have a lot in common with AOP frameworks. But personally, I would not like to do much of either using a toolkit made for the other (read more here). Either way, it's quite hard to find guidance about how to use any technique for the goal of multi-dimensional SoC. (Ivar Jacobson's Aspect-Oriented Software Development with Use Cases being a rare exception.) So I thought I'd just start writing about some of our experiences at rubicon with our own mixin implementation. I always believed that dynamic rule mechanisms should be made explicit where they are to be expected. We created our mixin mechanism to enable 3rd parties to create arbitrary extensions and modifications (notwithstanding the "virtual"-requirement) without modifying our source code or waiting for explicit extension points. Of course, in a large enough development organization, you have to treat other teams as 3rd parties too, because it becomes just unmanageable to create extension points whenever anybody needs them. For the requesting party that would be pretty tough too, because they would have to live off the trunk and wait for every modification to be designed, implemented, tested and delivered before they can start working. Using mixins proved to be a panacea for many of those problems. At the beginning, there were several reservations about overusing them, and in fact I'm still not sure how happy I am with the hundreds of mixins that emerged in some projects. But I did eventually give in to the notion that in some cases, explicit extension points would make the system just more fragmented and ad-hoc than using one well-understood paradigm for all of them. It's still something that needs to be decided on a case-by-case basis, and I'm not sure if Ayende's example (Account.Withdraw) is a very good one to make that point, at least not without any context. There's another example I like for using mixins: parallel class hierarchies. Here's the problem: I have an OO domain and a generic UI layer that uses data-binding on steroids. This UI can display icons for objects, but of course it cannot go ask the domain objects. We don't want UI code in our domain classes after all. So we have an icon service that the UI can ask for icons. Pseudo code: interface IIconService { string GetIcon (DomainObject obj); } Now most domain objects just return classname.gif, so this service is easy to implement. But wait, here's a special rule: Male and female persons use different icons. Now we could have an if-statement in GetIcon, but that's not very OOP. How do we use polymorphism without tainting the domain? We need another class tree, somehow attached to the original one. Here's how this is done using mixins: namespace Domain { class DomainObject { } // layer super type class Person : DomainObject { } } namespace UI { public interface IDomainObjectMixin { string GetIcon (); } [Extends(typeof(DomainObject))] class DomainObjectMixin<TDomainObject> : Mixin<TDomainObject>, IDomainObjectMixin where TDomainObject : DomainObject { virtual string GetIcon () { return This.GetType().Name + "gif"; } } [Extends(typeof(Person))] class PersonMixin<TPerson> : DomainObjectMixin<TPerson> where TPerson: Person { override string GetIcon() { return (This.Gender == Gender.Male) ? "male.gif" : "female.gif"; } } } Now you can cast any DomainObject to IDomainObjectMixin and ask for its icon. The PersonMixin class polymorphically provides its own implementation, while all other types derive the implementation of DomainObjectMixin. class MyIconService : IIconService { public string GetIcon (DomainObject obj) { return ((IDomainObjectMixin)obj).GetIcon(); } } I think this leads to a nice separation of layers. Sure, you have to grasp mixins, but you need to understand this concept once, as opposed to understanding a variety of ad hoc solutions. (The use of generics and where-clauses in parallel class hierarchies is a bit ugly, but it follows a simple rule that you can just learn if you don't want to dig deep and understand why it's required.) Kind of reminds me of Greenspun's Tenth Rule, except that a more obvious conclusion from that rule would be to use another language, I guess, but a well-designed framework is still much better than "an ad hoc, informally-specified, bug-ridden, slow implementation" in each project. There's another argument I hear a lot that I'd like to address. It goes like this: In OOP, composition is usually preferable to subclassing. That's not quite true. Subclassing is a very convenient way of gaining a certain set of features (like polymorphism and implementation inheritance), and it would be more than a bit tiresome to do OOP without it. But the more complex a problem gets, the less likely it is that we can solve it just by subclassing. For many problems, single inheritance just won't do, and multiple inheritance brings its own set of problems. But let me get this straight: yes, we're trying to make mixins as simple to use as subclassing, so they are easier to learn, and you can apply some of your valuable OOP experience. But the implementation of mixins is basically composition. (There is some sub class generation, but hey, somewhere the composition gluing has to be done.) So the bottom line is, mixins are just classes with a set of composition and delegation rules attached to them. That's a weird way of thinking about mixins, but the most correct one if you're reasoning about them in the context of the inheritance vs. composition debate. - Stefan Edit: missing reference for Account.Withdraw Edit: replaced Type Of (T) notation by Type<T> (Left over from an attempt to post this sample to a blog that thinks <T> is an HTML tag.) [Less]