Beagle

My coworker, napping:A particularly beautiful breed.

But we in it shall be remember'd: LugRadio Live USA 2008 is the 12 and 13th of April in hilly San Francisco.I will be there, speaking on Android, our complete, free, and open mobile platform. The talk will be a hit, a very palpable hit.

I am in Chicago, a little late reading the Sunday Times, but this article on beating the market caught my attention:Investors collectively spend around $100 billion a year trying to beat the stock market. That’s the finding of a rigorous effort to ... [More] measure the total costs of Americans’ efforts to surpass the returns they would have received by simply holding a stock index fund. The huge price tag helps explain why beating a buy-and-hold strategy is so difficult.In his new study, Professor French tried to make his estimate of investment costs as comprehensive as possible. He took into account the fees and expenses of domestic equity mutual funds (both open- and closed-end, including exchange-traded funds), the investment management costs paid by institutions (both public and private), the fees paid to hedge funds, and the transactions costs paid by all traders (including commissions and bid-asked spreads). If a fund or institution was only partly allocated to the domestic equity market, he counted only that portion in computing its investment costs.Professor French then deducted what domestic equity investors collectively would have paid if they instead had simply bought and held an index fund benchmarked to the overall stock market, like the Vanguard Total Stock Market Index fund, whose retail version currently has an annual expense ratio of 0.19 percent.The difference between those amounts, Professor French says, is what investors as a group pay to try to beat the market.The study's conclusion:What are the investment implications of his findings? One is that a typical investor can increase his annual return by just shifting to an index fund and eliminating the expenses involved in trying to beat the market. Professor French emphasizes that this typical investor is an average of everyone aiming to outperform the market—including the supposedly best and brightest who run hedge funds.The bottom line is this: The best course for the average investor is to buy and hold an index fund for the long term. Even if you think you have compelling reasons to believe a particular trade could beat the market, the odds are still probably against you.I go further: The odds are, most definitively, against you. You might beat the market this year, but you won't the next. Repeat after me: You cannot beat the market. Put your money in an ETF—or a small basket of them—and leave it alone.There are theories that decree this—equity prices reflect all known information and are an unbiased and collective valuation and thus you can only outperform the market through luck—but you do not have to subscribe to them, as mere arithmetic can make the case:Active investing yields average returns. Proof. Let M be the entire market. By definition, M's return is the market's total return minus net costs. Let P be a subset of M such that x is in P if x is pursuing a passive strategy. That is, P is tracking the total market. Then P is also earning the market's total return, minus costs (which are very low). Now, let A be a subset of M such that x is in A if x is pursuing an active (that is, managed) strategy. Since M=P A and both M and P are earning average returns, A is also earning average returns, minus costs (which are large). QED.To be sure, you could now argue that A is composed of two subsets, those who consistently and substantially outperform the market and those who consistently and substantially underperform the market. I think its clear that, in any given period, some funds are in one subset and some are in the other but, over the long-run, no one fund is consistently in either and you get reversion toward the mean.As an aside, one reason M and thus P can earn the returns they do is because of A's active strategies that close arbitrage opportunities, set prices, and otherwise make the market more efficient. In other words, it is A's moves that make the market efficient. Is that worth $100 billion? Definitely. [Less]

If the news reports and blogs are to be believed, this is the age of Semantic Something. First people wanted to search web, then file contents, and then search emails and other user data. Everybody was talking about desktop search; along came Beagle ... [More] , Spotlight, Google Desktop Search, Kat, MetaTracker, Pinot, Strigi etc. While desktop search at its core is nothing but a crawler which reads different file formats and stores them in a searchable database, searching is the most trivial and IMO, boring application built on Beagle's infrastructure.These days the focus seems to have shifted to Semantic Desktop and Semantic Web. Most blog comments and mailing list posts about Semantic-Fu have a hint of it being vapourware. Its not totally their fault either; the ideas are around for a long time and people are working on it for many many years. But there is no glittering gold in sight. Only recently some interesting Semantic Web ideas have started taking shape. Semantic Desktop is a slightly different game but it should not be far behind. After taking about 40 developer years, Beagle is just about ready to take desktop search beyond simple file content search. Historians might want to take note of the dashboard project and how beagle came into being as a necessary requirement for that truly beyond-desktop-search application.The core idea behind Semantic Desktop, upto my understanding, revolves around the buzzword jack-of-most-trades RDF. And for the impatient kind, here is a rude shock - RDF is not useful for human beings. Even further, it is not even meant for you, me and us; storing every conceivable data in the RDF format is not going to make our life any easier right away.RDF or Resource Description Framework is a generic way to describe anything, to be accurate any description of anything. It is a fairly elaborate yet structured format; very easy for programs to analyze that information but extremely redundant to human eyes. Notwithstanding what the AI experts are claiming about the future of AI, human mind can work without immediate deductive reasoning and in fact does that a lot of time. It recognizes familiar words without reading the alphabets one at a time, it deduces the color by merely glancing at it, it conjures up strange connections; its a wonder that will be hard to completely characterize by any set of rules. At least at the current stage, algorithms have to be told the facts and the relations between them for them to do any kind of processing with its data. These are the things that we just know when we see something and is thus the reason why storing the description of something in an RDF format is not going to gain me anything immediately. On the other hand, this is also the reason why applications should be fed data in an RDF format to allow it unhindered access to the semantics of the data.If that felt hand wavy, try to think about the difference between the semantics of a data and its syntax. An array could be used to represent a linked list, a queue, a stack, a tree or an heap - the latter are the different semantics of the representations, the array is one of the many syntactic representations of one of the latter concepts. A bunch of pairs could be stored in a database table; the table is a syntactic representation of the data which has the semantics of a bunch of name, phone-number pairs. It is hard to work with the semantics of an idea, in a sense it is something up in the air; on the other hand storing some data in a suitable working form could fail to capture some concept about the data. Also, once stored in a particular form it is easy to miss the bigger picture; thus limiting the scope of what we could do with that data.Saying all that, for the time being think of the RDF format as a bunch of objects and facts where each object is related to some number of facts. The semantics of related could differ based on the context, and RDF is powerful enough to describe even that semantics and a whole bunch of other facts about the facts. With beagle pulling data from nooks and corners of a user's desktop and providing a service which allows applications to search this data, it is a shame if we cannot exploit the relationships in this data for a better mankind... err... dolphins... err... us.Consider all the emails I have. Now I know that there some emails that are part of discussion threads. Beagle does not. With the beauty of N3 (a close cousin of Semantic-Fu and RDF), I can write this extra information as a set of rule (the single '.' represents end of one rule). I am using emails msgid to track emails in a thread. I could not help but notice the similarity of these rules with prolog or other logic programming languages./* an email with subject 'foobar' is in its own thread */{ ?email :title 'foobar' . ?email :msgid ?msg . } => { ?msg :inthread ?msg } ./* if any email refers to some email in thread, then this email is also in other email's thread */{ ?ref :inthread ?parent . ?email1 :reference ?ref . ?email1 :msgid ?msg .} => {?msg :inthread ?parent} .Using the RDFAdapter of the beagle-rdf branch, I can use this to get all the emails in the thread with foobar in its subject. Note that I am able to write my set of rules only when I see this data as actual emails and not a bunch of lucene documents with fields. The latter carry no meaning. Further note that, I can also use the BeagleClient API to perform field specific queries to obtain the same results. The difference is that the process of using BeagleClient will require me to think about the relationships from scratch and then figure out the right sequence of queries. Instead I could store all the relationship among the emails in the email-index in an RDF format (and also related information not stored in the index e.g. saying a list of email addresses are all mine and should be treated as for one person). Then, whenever I want to extract some information, I can write the question (again in an RDF format) and let the RDF-Magic figure out the how to execute this question against that data given this set of inference rules. Isn't it cool ?If I missed it earlier, this kind of data-mining operations are not for my everyday use (here my refers to usual computer users) and is not for everybody. Still it is can sometimes come in handy. Imagine the possibilities if you can write the relationships between a file in an mp3 playlist (playlist filter), its download link and how your arrived at that page (webhistory indexing), the email you sent with that file as an attachment in a zip file (email attachment and archive filter), its ratings and usage statistics in Amarok (amarok querydriver) and of course the actual file on the harddisk (user home directory indexing).Warning: The RDF Adapter in beagle uses the sophisticated SemWeb library which allows anyone to perform graph operations (selecting subgraphs, walking on graphs, pruning nodes and edges etc.) on the RDF graph of the data. Unlike most RDF stores for desktop data, beagle is not optimized for RDF operations and could take quite a bit of time and heat up the CPU. It took me about 4 seconds to find all threads with the word beagle among 500 emails (my actual email index has about 20K emails! I refuse to imagine what will happen if I run it on the full index). If you are interested, checkout the rdf branch and take a look at the test SemWebClient.cs. [Less]

So the more I work with Django the more I long for a solid development environment to work in. I use Wingware for much of my python development, with its rockin debugger and code completion, its more than I could ask for. Until the curse of the Java ... [More] class.  This quarter I’m taking a Java projects course,  most of the class uses Eclipse but a few use Netbeans. My problem is, I got spoiled so fast by the incredible templates support, content suggestions, quick fixes and always dead on code completion. Going back to Wing feels like a halfway-there IDE. I know that pythons interpreted nature makes source completion much more difficult, now I would argue that with an interpreter, you could actually step through the code to some extent. However, I respect that dynamic objects are never gonna be easy to support. My beef is with the lack of support for super-popular frameworks (this goes for everybody!) Ruby on Rails has literally dozens of solid IDEs and a few that are just spectacular (see Aptana, or Netbeans). Why can’t I get even basic highlighting support for my Django templates? Why can’t I get any completion options on Models except my own? Its just frustrating, Django is still a pleasure to develop in, even with just Gedit and a terminal, but is it really out of the question to consider providing a big pretty environment for those of us that like that? I did dig up this and this. I guess its a step in the right direction, but its almost embarrassing next to the Rails environments. [Less]

We released version m5-rc15 of the Android SDK. See also Apps for Android.Per usual, I shall twitter the election results from tonight's four state battle royal.And, finally, this presidential contest Google Maps mashup is cute and fun.

From Marc Andreessen, the founder of Netscape: We asked him directly, how concerned should we be that you haven’t had meaningful experience as an executive — as a manager and leader of people? He said, watch how I run my campaign — you’ll see my ... [More] leadership skills in action. Well, as any political expert will tell you, it turns out that the Obama campaign has been one of the best organized and executed presidential campaigns in memory. Even Obama’s opponents concede that his campaign has been disciplined, methodical, and effective across the full spectrum of activities required to win — and with a minimum of the negative campaigning and attack ads that normally characterize a race like this, and with almost no staff turnover. By almost any measure, the Obama campaign has simply out-executed both the Clinton and McCain campaigns. We then asked, well, what about foreign policy — should we be concerned that you just don’t have much experience there? He said — and I’m going to paraphrase a little here: think about who I am — my father was Kenyan; I have close relatives in a small rural village in Kenya to this day; and I spent several years of my childhood living in Jakarta, Indonesia. Think about what it’s going to mean in many parts of the world — parts of the world that we really care about — when I show up as the President of the United States. I’ll be fundamentally changing the world’s perception of what the United States is all about. Ohio, Texas, Rhode Island and Vermont vote tomorrow, and many polls show a tight race. I’ll be following Robert’s coverage on Twitter while I am at the Bruins game. Should be fun! [Less]

So I just finalized my registration for PyCon and booked my flight! I can’t even begin to express my excitement! If anyone else plans on attending, I made a wiki page for you to add your name to! See you there!

The Times offers this handy graph about that raise you're not getting:This depicts year-on-year changes in hourly and weekly real wages. Real, as opposed to nominal, figures are adjusted for inflation. Deflating the earnings allows us to compare ... [More] historic prices in today's dollars—apples to apples, instead of apples to apples covered in rising oil prices.From the article:Since last October take-home pay has been contracting for about 80 percent of the American workforce, including factory workers and non-managers in service businesses.Although the author allows that the "blame lies with a combination of slower wage growth and higher inflation," the article does not address the data driving the fall—exactly 1% (contra to the graph) or 18 cents (in January 2008 dollars)—which is high inflation to the tune of 4%. Nominal wages grew by 3.7% year-on-year; if we were experiencing inflation more in line with the norm—say, 2007's average of 2.85%—real growth would have been positive. Instead, inflation, particularly over the last three months, has been sufficiently high to cancel out the nominal growth.Four percent isn't 1974 (11%) or Zimbabwe (66,212%), but there are troubling parallels with the former. Are we looking at a sustained period of both inflation and depressed growth—the dreaded stagflation? My money is on the recession but not the inflation, but there are a lot of unknowns. Zimbabwe's problems are easier: Long-term inflation is always and everywhere a monetary phenomenon, so lay off the printing press.To the rescue, Chairman Bernanke today signals he is open to more rate cuts. Err, wait. That won't help. Ah, but look over here: The FOMC finally acknowledges via its semiannual congressional report that it has an informal inflation target of 1.5 to 2.0% core over three years and slightly less over the long-run. That should aid in restoring the Fed's inflation hawk credentials. Now is decidedly not a fun time to be Fed Chairman.For the curious, the data backing this graph and my post is from the BLS report Real Earnings in January 2008. Earnings were deflated using the CPI-W, which includes food and energy. [Less]

So I attempted to migrate my blog from Wordpress to Movable Type Open Source. I’ll go into specifics later, but suffice to say that it was a miserable failure. Problems with planets, constant admin interface crashes, extremely spamable, and most of ... [More] all, just unusable. As a result of my reversion back to Wordpress, I have lost a few posts, I will try to migrate them back, but its possible I’ll just do a major ‘things that we have missed’ post in a bit. Anyway, more updates coming soon. [Less]