Forums : Suggestions for Ohloh 2.0

Hey everyone,

We just open sourced Launchpad, which makes me very happy.

I'd love for my Launchpad contributions to appear on my profile, but I think this means Ohloh needs to upgrade its Bazaar.

When I try to point Ohloh at Launchpad's Bazaar branch (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-stable), I get an error message saying it cannot get the revno. But...

$ bzr revno http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-stable
8307

Launchpad has been open sourced in Bazaar's new 2a format, supported by bzr 1.16.1 and bzr 1.17. I think if Ohloh upgraded its internal Bazaar to one of these versions, I'd be able to point it at the Launchpad tree.

Thanks,
jml

Jonathan Lange over 15 years ago

Hi Jonathan,

I agree, it looks like Ohloh is going to have to upgrade its version of Bazaar.

I don't think this can happen immediately -- We will have to free up some time for testing and deployment. Perhaps we can get this done next week.

Thanks,
Robin

Robin Luckey over 15 years ago

BTW congrats on opening up Launchpad, that's great news. :-)

Robin Luckey over 15 years ago

Thanks Robin, both for your congratulations & the swift response.

I can't wait for it :)

Jonathan Lange over 15 years ago

Hi Robin,

Any word on the Bazaar upgrade?

Thanks,
jml

Jonathan Lange over 15 years ago

Hi Jonathan,

We have made the code changes required so that we are compatible with bzr 1.17. It's working on development machines.

Now we need to actually deploy bzr 1.17 onto our server cluster. I'm going to take a stab at this today, and I'll let you know how it turns out.

Robin Luckey over 15 years ago

Ohloh is now running bzr 1.17 across the board.

The tests are all looking good. Let me know if you discover any troubles!

Thanks,
Robin

Robin Luckey over 15 years ago

Thanks Robin!

I've added an enlistment for Launchpad and it appears to be working -- it's currently downloading source code history.

I'll let you know if I hit any problems.

jml

Jonathan Lange over 15 years ago

It's currently failing with:

Step 2 of 3: Importing source code into database (Failed about 5 hours ago.)

The url is:
http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-stable

Jonathan Lange over 15 years ago

Hi Jonathan,

The move to bzr 1.17 has not been smooth.

In Launchpad's case, we are trying to get bzr to peform this enormous task:

bzr log --long --show-id --forward --include-merges -r 1.. -v

Each attempt takes upwards of an hour before it fails. The failures occur in various, non-repeatable ways.

Sometime bzr complains that some data is missing (I don't believe it), and we see an error like this one:

bzr: ERROR: The file id x_Stuart_Bishop__Sat_SepW24_14:41:02_2005_18800.0 is not present in the tree .

Other times, we are seeing general exceptions, with bzr complaining that it has encountered an internal error with an exception stack.

We're sort of between a rock and a hard place. It seems like bzr 1.17 is unstable, with unrepeatable failures, yet older versions of the bzr are not compatible with new repositories.

There are some new Ohloh bugs that turned up over the weekend. bzr 1.17 is different enough from older versions, that our code is breaking in some subtle ways. I'm going to work through some bug fixes on some smaller projects, and perhaps those fixes will help the situation on larger repositories like Launchpad.

However, our initial problem is just a basic inability to fetch the log without error, and I don't know yet how to fix this on our end.

Robin Luckey over 15 years ago

That's pretty bad news.

Any time bzr raises a stack trace, it's a bug in bzr. I'd greatly appreciate it if you could file bugs when you get those, or even email me with the stack traces.

The missing file id might actually be missing. Launchpad was an alpha tester of the 2a format, and has a repository that's considered unusual by Bazaar standards. I'll chase this up with bzr.

Working through bugs on smaller projects sounds like a perfect plan. :)

Jonathan Lange over 15 years ago

Forgot to say, I really appreciate your help with all of this.

Jonathan Lange over 15 years ago

Hi,

Have you managed to make any progress in this front?
Can we help out?

Martin Albisetti over 15 years ago

Looks like it's importing OK now, but really slowly. My back-of-the-envelope calculations put Step 2's completion on the 13th of this month.

Is this really just the command you've pasted above? I'm curious as to whether you've discovered a particularly badly behaving code path in Bazaar, or whether it's something on the Launchpad server.

Thanks!
jml

Jonathan Lange over 15 years ago

Well, this is something of a pleasant surprise -- it looks like an alternate repository URL was supplied, and our download from this URL completed successfully.

We were also able to extract the log from the repository.

Now we're in the arduous step of examining every commit to determine the file changes that occurred. For Bazaar repositories, this takes a very, very long time. It might take literally months.

I'm not sure why Ohloh's performance is so terrible with Bazaar -- we use the same strategy we're using with other source control systems. We seem to be OK with smaller repositories, but once the repository reaches a certain size, things really get out of hand.

I haven't had time to really investigate the source of the poor Bazaar performance, so I'm not sure if it's something stupid on Ohloh's side, or if Bazaar just isn't well suited to Ohloh's typical strategy, and we should be trying a different method.

I'll be keeping an eye on it. As long as it keeps moving forward (even slowly) I'll leave it be.

Thanks,
Robin

Robin Luckey over 15 years ago

I like pleasant surprises :)

If you're interested in speeding up the processing, I recommend that you post a mail to Bazaar's mailing list , or ask on #bzr on Freenode.

You would not believe how much thought they've put into the question of Bazaar performance. :)

Thanks for the help,
jml

Jonathan Lange over 15 years ago

Otherwise, if you can give us some insight on how you are using bzr (bzrlib? command line? what operations?) maybe we can help you tune it.

Martin Albisetti over 15 years ago

It looks like the Launchpad code import finished today, but the line counts look a bit fishy:

https://www.ohloh.net/p/launchpad/analyses/latest

In particular, -3,291 lines of SQL and -2,406 lines of HTML. Not sure how those numbers would get into the stats.

James Henstridge over 15 years ago