Forums : Ohloh General Discussion

Hi there,

today I was randomly looking through the Kudos rankings. I figured that looking at people with high Kudos might reveal some interesting new projects etc. Doing so I observed something that to me (after reading the How is Kudos rank calculated description) makes no sense. Essentially, I would have expected the following to hold: If a person gets Kudos from others, and participates a lot in many high-profile projects, (s)he will have a higher Kudos rank than somebody who got no Kudos from others, and contributes less to fewer, lower-ranked projects.

However, that doesn't seem to be the case. I could use my own profile for comparison now (which, to be honest, was my initial reason to write this post -- yes, I am a Karma whore :-), but that's not necessary. To give an example, consider these two persons:

• 4267 (rank 494)
• 4197 (rank 588)
• 183 (rank 595)

I don't understand how Mr 4267 can be 100 ranks higher than Mr. 183). That seems to make no sense -- but maybe I am just missing something?

The most extreme example, however, is this guy:
4513 (rank 400)
who has done nothing (well nothing visible on Ohloh at the time of this writing), except for getting Kudos from somebody on Kudos level 1. What?!?

Would be nice if somebody could try to explain these particular examples, and maybe the Kudos ranking description could be clarified a bit. :-)

Ed: linkified post.

Max Horn almost 18 years ago

hi Max,

I'll let Robin give a more thorough response (he wrote the code ;-). However I wanted to thank you for the thoughtful post.

We launched kudo ranks suspecting we'd have to revisit the algorithm after some initial seeding period - which I think we've now reached. I invite others to give us your opinions/questions on how you'd kudo ranks to change.

Jason Allen almost 18 years ago

Hi Jason,

thanks for the reply, I now look forward to maybe hearing from Robin. I know how difficult it is to get a ranking formula balanced (SourceForge.net took years for it, and the original formula was totally flawed). Esp. when you use an iterative definition as you do, or as Google's pagerank uses, it's a hell of a time to get those tuned, and in the end you'll always get some data which is surprising. There's a reason this kind of algorithms comprises an interesting research field of its own in comp.sci. and math ;-). So good luck on this one.

(Oh and thanks for linkifying my post.)

Max Horn almost 18 years ago

Hi Max,

Your questions have prompted a lot of discussion around here, and answering your questions created a long post.

Rather than give exact mathematical answers (which I am happy to do later if you are still interested), I'll explain the underlying principles and how they lead to the results you see.

First, I checked for bugs, and I've confirmed that everything is working as designed (although perhaps not as expected!)

Currently, the main shortcoming with KudoRank is sparse data. We're running a ranking algorithm on nearly 50,000 participants with only about 2000 people actively contributing data. The sparse data means that a large number of people have very similar scores. While the difference between a KudoRank 10 and a KudoRank 9 is pretty large, the differences are tiny after that. For the particular accounts you've highlighted the underlying scores are so close that it's a virtual tie, and a difference of several decimal points determines the ordering.

As time goes on, problems from sparse data are diminishing, and primarily I want to exercise patience and let the data sort itself out. However, I think we have at least one design flaw which causes nearly everything you found -- which is:

On the KudoRank wiki page, Rule 3 declares, Giving away more Kudos dilutes your opinion. The intent of this rule is to prevent one person from stuffing the ballot box with thousands of Kudos. However, we actually have the opposite problem: a person who gives only a single Kudo has a very powerful impact on the results, as compared to a typical user who might hand out several dozen Kudos.

We use a simple 1/n rule: if you hand out 2 Kudos, you pass along 1/2 of your influence to each recipient. If you hand out 10 Kudos, each recipient gets 1/10.

This creates the undesirable scenario you found. Mr. 4513 manages to score very highly with only a single Kudo from someone with lowly KudoRank 1. Meanwhile, Mr. 4197 has received Kudos from several people, including someone with KudoRank 10 and someone with KudoRank 9, and yet does not manage to rank as highly. Surely this can't be right!

The explanation is that Mr. 4513 is sole recipient of all Kudo influence from account 4360, getting 100% of the available boost. However, Mr. 4197 is receiving relatively small slices of influence from his Kudo senders. For example, account 1194 has given Kudos to 29 people, dividing his influece so many ways that even though he has KudoRank 10, his net influence on any one kudo recipient turns out to be smaller than a single, undivided contribution from someone with KudoRank 1.

I think this is broken, and it isn't something that will correct itself as we collect more data. Some possible fixes:

1. Cap the 1/n rule, giving n some minimum value. For instance, you can never give more than 10% of your influence to any individual. This would prevent one KudoRank 1 person from launching another KudoRank 1 person all the way up to KudoRank 9 with a single action.

2. Remove the 1/n rule completely, which lets you give your full Kudo influence to as many people as you choose. This means that, as expected, a Kudo from someone with KudoRank 10 would be worth much more than a Kudo from someone with KudoRank 1, no matter how many other Kudos those senders might be passing around. I like this option because it simplifies, rather than complicates, the system.

Finally, understanding Mr. 4267 comes down to understanding how we integrate stacks in the calculation. When you stack a project, that's equivalent to giving a single Kudo to that project, to be shared by all of its contributors. The Kudo is split up proprotionately among the contributors based on their number of commits. If the project has only a single contributor, that contributor gets the whole Kudo, and it's mathematically equivalent to giving that contributor a regular Kudo directly.

This means that Mr. 4267 has received 'nearly' the equivalent of 5 Kudos, because WiX has been stacked 5 times and it is 'nearly' a solo project. How does this put him 100 ranks higher than Mr. 183? Well, that's thanks to the 1/n rule, which dilutes the kudos given to Mr. 4267 and Mr. 183 such that 4267 comes out on top -- but just barely. Their scores are actually quite close.

OK, this post is getting quite long, so I'll stop for breath and invite feedback. I've come away with the conclusion that the 1/n rule needs to be addressed in some way, although some thought needs to go into the exact change.

Let me know if I've answered your questions well enough, and I'm very interested to hear your thoughts.

Thanks,
Robin

Robin Luckey almost 18 years ago

Hi Robin,
thanks for the info, that answers most of my questions. Being a mathematician I am always interested in mathematical details ;-). I dealt a bit with the theory of fair voting systems in the past, which has quite some relations to the problem at hand. In particular I was reminded a bit of Arrow's theorem.

For the 1/n rule, let me point out another possibility (I am not trying to promote it, just pointing out another possibility): You could use a non-linear scaling, e.g. make the influence of a Kudos proportional to 1/log(n+1) (base 2 logarithm used here). This way giving a single kudos counts fully, giving 16 counts 1/4th, etc. And then of course you could combine this with a capping scheme, too... And other formulas are possible as well, of course. In fact, 1/sqrt(n) might be a good one, too. This would in fact be in line with some results in election theory which imply that the total influence of a group among several groups (think of the population of one state out of a federation of states, like in the US or in the European Union) grows proportional to the square root of its size, instead of growing linearly as one would naively assume. But I didn't do anything to properly research the connection to this result from election theory, it's more a gut feeling, so don't take it too seriously (but it might be worth trying this out anyway).

The alternative of not doing any scaling of the kudos a person gives (i.e. each counts full) is appealing because of its simplicity and clarity. Your voice counts fully, no matter what you do. However, this is not without problems, too, as televoting or webpolls show (or, for that matter, democracies): A few dedicated persons willing to make sure there voice is heard can spoil the system. In our case by spending kudos on lots of persons. Even kudos bombing would be possible: Create lots of fake accounts, give kudos to each other, and then use this accumulated Kudos power to boost other people (why on earth you would want to do that is beyond me, but my lack of imagination won't stop anybody ;-).

However, this is just randomly playing with the parameters, without real understanding of what's going on. It might be worth to do a bit research and look at the literature whether there are some applicable research results and/or discussions of similar problems...

As for the project kudos: This also seems bad, but is likewise probably not easy to fix (and possibly even impossible to fully fix, see ). It seems queer that a single contributor to a low profile project would get more kudos than a high profile contributor on a project with many other contributors. But I do understand now what's going on there. This is indeed a difficult problem. Wonderful, I love difficult problems, esp. those which aren't mine ;-).

Cheers,
Max

Max Horn almost 18 years ago

Very interesting discussion, thanks for sharing these thoughts.
I'm also very interested in popularity/reputation metrics, but I'm wondering whether you guys have considered the effects of making the specs of the kudo algorithm (partly or completely) public. Most projects I know of keep their ranking algorithm strictly secret to prevent spam (whereas some reverse engineering is tolerable).

dartar almost 18 years ago

Um, dumb question (or clarification rather)

When you stack a project, that's equivalent to giving a single Kudo to that project, to be shared by all of its contributors.

Okie, hypothetical user Joe Bloggs has a kudos rank of 10.
Joe has been really impressed by 5 people, and has given each of them kudos, giving each 1/5th of his influence.
Joe then hits the project search and stacks 5 projects he couldn't live without.

Now the clarification.... Which of the following is correct?

• Joe is now giving 1/10th of his influence to each of the 5 people and the 5 projects.
• Joe is now giving 1/5th of his influence to each of the people, and each project stacked receives 1/5th of his influence, regardless of how many projects have been stacked.
• As last option, but 1/6th (5 people and a spare person for the projects)
• Joe is giving 1/5th influence to the 5 people, and a separately calculated 1/5th to each of the 5 projects

And so on through a few other possible permutations.

The way I see it, Ohloh users fall in to two categories ... those that go round stacking every project they use (which is how I interpret the original intention of the feature), and those who just stack the ones they can't live without or consider their favourites (I fall in to this category).

If the number of projects you've stacked is used in calculating the amount of influence given to people, then the latter group have a massively more effective influence since the former group might stack 50 projects while the latter group may only stack 5. This ends up allowing people to cheat the system and get better kudos by keeping their stacks to a minimum.

If, on the other hand, projects receive a fixed amount of influence (whether that be based on kudos given, kudos rank, or arbitarily fixed), then the lack of voting value as opposed to giving kudos opens the possibility of mass project creation/stacking to boost kudos, and also tends to result in the reasonably predictable result that certain projects and people are almost guaranteed to swamp the top of the list (although that isn't a bad thing if that was the aim of the behaviour).

Um ... I've forgotten where I was going with this ... I blame being tired... I'll shut up now :)

Daniel / Nazca ... almost 18 years ago

Hi Daniel,

I think your major concern is that our current system might discourage people from creating large stacks. And you're right that we'd like users to stack all the projects they use -- this gives us the richest data, allowing us to find better suggestions for you and people with more similar stacks.

Proposal: Allow you to weight the value you place on the items in your stack. Perhaps this is just ratings, but make it easy to rate all your projects right from your stack window. This suggestion was sent to us by a user and I thought it was a good idea at the time. But now with your comments, I think it might be useful in itself but also might encourage more people to stack more of their projects.

What do you think? Would you put more projects on your stack if they were sorted, weighted, etc., by rating or importance and we made it easy for you to do the rating?

Andy Verprauskus almost 18 years ago

IMHO, kudos given to a project should be (at least automatically) weighted proportionally to the size (weight) of the project.
Weight of a project should, in its turn, be proportional to its activity (commits per time), quality (documentation), code size, famousness, ...
For your last question, yes, I'd appreciate if I could weight my own stack. I use some open source softwares because there aren't another thing better, but I don't think they are as good as they could be.

DotMG almost 18 years ago

Another potential problem I just noticed today (as I experienced it myself): Get a couple of your friends to sign up, say three. Then, each of you gives kudos to each other. Don't take too many. Now each of you got kudos from three people, so you get a nice boost in the current system (due to the current sparse data situation). Hence you get a nice kudos boost. Which increases the worth of your kudos, which means that your buddies get a bigger boost, which means you get an even bigger boost in the end.

I think it's important to balance the system against this kind of use. This is, in this situation the diminishing effect giving kudos to three people should be balanced against the boost these people get, and the back-boost you get from them in turn.

How much does the kudos rank factor into the kudos I give, anyway? Is the kudos of a rank 9 person 9 times higher than that of a rank 1 person? Hm.

Max Horn almost 18 years ago

Maybe this has already been covered (or implicitly assumed in the mathematical background), but it sounds from the preceding discussion like the value of kudo in general does not change from person to person. The value granted can be diluted by granting multiple kudos but that value is the same for people of different ranks. It seems to me that the opinion of a high ranking developer matters much more than the opinion of a low-ranking developer, such that a high ranking developer giving out a modest amount of kudos should not devalue their opinion so low that it is below even the lowest ranked developer. Is it just a matter of scale - can the existing system be fixed (or at least ameliorated) by simply increasing the value of kudo proportional to ranking? I think there may also be another phenomenon in effect - I would hazard a guess that higher ranked developers are also more popular and familiar with other higher-ranked developers, and therefore are more predisposed to actually grant kudos than a low-ranked, infrequent, developer just doesn't happen to really know anybody else.

aaron almost 18 years ago

@dartar: I would prefer full disclosure of the algorithm, because this will lead to better community trust of the system. However, you're right, it leads to folks trying to exploit weaknesses. As it stands, I'd rather be open, and fix problems as they occur.

@Daniel / Nazca: As the system works today, if you give out 5 kudos and 5 stacks, you'd be giving 1/10th of your influence to each. I don't like this. I'd rather that you were giving a fixed amount of influence to all 5 of your kudos, and another (probably smaller) fixed amount of influence to each of your stacked projects. I don't want to penalize people for participating and stacking lots of projects.

@Max: Yes, a small group can log-roll eachother up the scale -- You can become a 10 with a just a couple of bogus accounts put together in a chain. I need to give it some thought, and it seems counter-intuitive, but I suspect that removing the 1/n scaling factor would actually make this harder to do because it will elevate the scores of real people. I need time with paper and pencil on this one.

@aaron: Yes, kudos from higher-ranked people are more valuable. Beneath the 1-10 numbers we publish on the site, there is a hidden floating point score, and this score is what actually gets propagated around when we do the kudo calculations. When someone kudos you, you inherit a fraction of their score. So yes, a kudo from a higher-ranked person does give you more 'points' than a kudo from a lower-ranked person. I do think it will be interesting to see how highly ranked people hand out their kudos, but for now our population pool is too small to guess.

I'm going to run a couple of experiments with a few different KudoScore algorithms, and perhaps we'll publish under a new algorithm later this week. Keep up the input. This is an interesting discussion.

Arrow's Theorem was fascinating and humbling.... thanks for the link!

Robin Luckey almost 18 years ago

Andy:

I think your major concern is that our current system might discourage people from creating large stacks. And you're right that we'd like users to stack all the projects they use -- this gives us the richest data, allowing us to find better suggestions for you and people with more similar stacks.

My concern is more that the data it is producing is effectively weighted heavily away from the people using the system as intended.

Example, mon amigo w00t has 5 items stacked, and is giving kudos to 6 people ... I'd guess he's created his stack on the basis that it contains projects he's directly involved iAlson or (in the case of PHP) relies heavily on. As a result, the influence he's giving is pretty heavy.
Compare this with mr webs who has stacked more projects than I wish to count. Since he doesn't seem to have bothered kudos w****ing, he has none in and none out ... meaning he is having an infinitesimally small influence on them.

The developer user who influences people and projects s/he is directly involved in is bound to end up with ridiculous values.
The end user that stacks all the projects s/he actually uses (theoretically increasing the project population in the process) has next to no influence on the results.

Personally I don't see this as a problem for the projects by stack counts (which is an interesting and fairly accurate metric that relates as closely to actual user counts as does running /list on freenode).
Unfortunately the kudos ranks for users end up completely out of wack.

Proposal: Allow you to weight the value you place on the items in your stack. Perhaps this is just ratings, but make it easy to rate all your projects right from your stack window. This suggestion was sent to us by a user and I thought it was a good idea at the time. But now with your comments, I think it might be useful in itself but also might encourage more people to stack more of their projects.

So I could stack, say, PHP and Putty ... but since I can't really live without PHP but I'm phasing out my use of putty (bye bye windows so bye bye putty), I can give PHP 90% of the kudos and Putty 10%?

I'd probably actually stack more projects if we had separated stacks. Then I could build a stack for my operating base (Gentoo, PHP, Apache and Subversion maybe), one for the tools I use (Firefox, Vim, webdev toolbar, etc), one for my more recreational use (inkscape and gimp spring to mind) .. and one for projects I'm actually involved in coding.

Might be interesting to see an overal list of projects sorted by ranking ... personal stacks ... maybe it would make sense to have some sort of personal importance rating? Then we could make it so it runs distro, webserver, scripting language?

Would probably make sense to have personal importance seperated from our rankings of projects...

DotMG:

IMHO, kudos given to a project should be (at least automatically) weighted proportionally to the size (weight) of the project. Weight of a project should, in its turn, be proportional to its activity (commits per time), quality (documentation), code size, famousness, ...

That sounds painfully complicated. Plus weighting by activity isn't paticularly fair on smaller projects. If you stack KDE or a distro, it'll probably gobble up all your influence.
Quality and famousness aren't paticularly good choices for metrics, since they are largely personal opinions.

Max Horn:

I think it's important to balance the system against this kind of use. This is, in this situation the diminishing effect giving kudos to three people should be balanced against the boost these people get, and the back-boost you get from them in turn.

Logically speaking, the rate at which the kudos rank increases will slow down and eventually start decreasing again. It's something of a parabolic curve? Math terms kinda need my head working lol.
As the users join, the projects list gets more populated, and the actual commiting and kudos receiving person count increases, people already on the kudos ranks will tend to loose places unless they gain more kudos.

Robin:

As the system works today, if you give out 5 kudos and 5 stacks, you'd be giving 1/10th of your influence to each. I don't like this. I'd rather that you were giving a fixed amount of influence to all 5 of your kudos, and another (probably smaller) fixed amount of influence to each of your stacked projects. I don't want to penalize people for participating and stacking lots of projects.

Yeah, it would make much more sense to have a separate kudos pools for user to user kudos and user to project kudos, especially when you combine that with the aforementioned project kudos weighting.
Seems kinda counter intuitive having the most people with the least activity on the site having the greatest influence on ranks.

Yes, a small group can log-roll eachother up the scale -- You can become a 10 with a just a couple of bogus accounts put together in a chain. I need to give it some thought, and it seems counter-intuitive, but I suspect that removing the 1/n scaling factor would actually make this harder to do because it will elevate the scores of real people. I need time with paper and pencil on this one.

Sadly I suspect this will give you a headache.
Cheating the system with loops can't really be blocked as loops, because all kudos giving eventually produces a looped set of nodes.
Looking for loops below a certain size doesn't help because that means you need to arbitrarily pick a size at which a loop should be considered fair ... but then again, I'm sure there are an awful lot of reciprocal kudos links in the database ... person A gets kudos from person B, so he gives it back.
You could possibly look for closed loops, where a group of people have created a web of kudos gives that doesn't cause kudos to leave that group, by way of project stack or direct kudos grant, but that still runs the risk of penalising groups of people that don't have much interaction outside their communities.
You possibly place a tax on kudos loops shorter than, say, 3 or 4 nodes... but that'll probably hit everyone in the database in some way, so the value of it is debatable.

I think valid accounts bouncing each other up the kudos ranks is something anyone will struggle to find a good way of dealing with, since there isn't a decent way of identifying those users.
Targeting bogus accounts might be an easier way to remove some overinflated kudos from the ecosystem heh.

It's kinda like trying to rig an automated system to find an lock multis on the mmorpgs that don't allow people to have more than one account. Any user smart enough to have created multiple accounts that are not obviously the same (different email addies, access via proxy's, etc etc) is going to be able to maintain gaming patterns that are not sufficiently different from natural clan behaviours to be targetable by any sort of automated tool.
scratches his head

Daniel / Nazca ... almost 18 years ago

Actually, I don't think it's impossible to get around loop abuse, i.e. people pimping up each other by forming a complete graph. The problem here is that so far the Kudo ranking has been developed from the guts, and not by applying rigorous mathematical arguments. This is prone to break / not work, as sometimes common sense just fails to find the right solution, or even comes up with solution that seem correct but actually can be shown to be worse than most other algorithms. Which is why I proposed that one should at least try to look at the literature, I'd not be surprised if there weren't already some papers etc. dealing with similar problems.

As for the stacked-project-kudos, it makes for an example were one could challenge the whole approach. Right now, you have taken an user centric view point -- you check how many projects got kudos from a person, and then assign to each some value. But IMO a project centric view is also possible, and actually is the one I assumed in use (until I learned in this thread that I was wrong), and which lead me to my wrong assumptions in the first post on this thread. Namely, if I look at a project page, and see that 500 people stacked it, I think Wow, lots of people use it and would expect that it gets a big boost from this, compared to a project with say 10 people who stacked it. And for this I do not care at all what the Kudos rank of the people who stacked it is.

So if 100 anybodies use project X, while project Y is only used by the the top 4 (according to Kudos rank) developers -- then I definitely would rank X higher. Probably with 10 vs. 4, too. I guess below that I could accept a certain higher influence of the high profile devs, but I wouldn't complain if all users were treated equally on this. At most, I'd say a Rank 10 dev should get maybe 20-50% more influence, I'd say (from my guts, too, I admit :-).

Going further, assume project A and B are stacked by 10 people, but the 10 stackers of A each stacke 100 projects, while those 10 stackers of B only stacked B and nothing else. Then I would expect that A and B get the exact same boost. I simply see no justification to do it differently... :-)

Just my two cents, and again, I didn't sit down to work out on a rigorous theory behind this, so I don't claim to be right or anything, just stating my thoughts here.

Max Horn almost 18 years ago

I love all the brainstorming, but it seems like there's still a lot of basic misunderstanding about what's going on with stacks and kudos. I'll do my best to answer in under 10,000 words.

The KudoRank algorithm is not from the guts. It is a very simple system inspired by the Google PageRank algorithm, and you can read as much math on the web about that as you care to read. Although they're not identical, understanding PageRank will help you understand KudoRank.

There is no looping exploit. Kudos are a fixed sum game -- you can only gain kudos if you take them from someone else. If two people kudo eachother, they form a symmetrical system where each takes as much as he gives, and they end up with the same score they began with. Likewise for any isolated island of people: the total kudos in island is constant, and no amount of inter-kudoing can grow the total. Contrary to there being a looping exploit, the more well-connected or looped a graph is, the more the kudos of the individuals will converge to the average.

There is an exploit to reach the top of the rankings, and it requires that all of the kudos in an isolated group be handed up to a single person. If bogus accounts A, B, C, and D all kudo eachother like this...

A -> B -> C -> D

...or like this...

A -> D
B -> D
C -> D

...then account D will end up with almost 4 times the average number of kudos, and will be ranked near the top of the Ohloh charts. I can think of two solutions to this problem:

1. We limit the amount of kudos that can be propagated by a single link from one person to another. It doesn't make this kind of exploit impossible, but it does require a lot more bogus accounts to reach the top of the charts. Early experiments today make this look like a very effective patch.

2. If someone uncovers such an abuse and posts about it in these forums, and we can easily blacklist the kudos from the bogus accounts.

The 1/n factor when passing along kudos (where n is the count of recipients) is the requirement that 'conserves' kudos in the system, and is what makes kudos a zero-sum game. The immediate problem is that if n is 1 (or other small number), you're pushing a lot of kudos to a single recipient. Capping this contribution to a small value is my current plan. While it breaks conservation, it makes the total kudos strictly less than otherwise, so it doesn't harm the system.

Kudos are person-centric, and don't have anything at all to do with ranking projects. Projects don't get a boost from kudos.

Kudos and stacks are two separate features. The KudoRank algorithm itself doesn't know anything about stacks, and stacks definitely don't know anything about kudos.

However, our kudo data is sparse, and our stack data is rich, so we decided to leverage the stack data to improve the quality of the KudoRanks. Also, we feel that being a key developer on a project with a lot of stacks ought to be worth something in the KudoRanks.

Therefore we created a mapping that converts stacks into kudos right before we recompute the rankings. So yes, stacks affect KudoRank, but this is not innate to the KudoRank algorithm itself. By actually converting stacks into kudos for purposes of the calculations, we keep the KudoRank algorithm very pure. Am I veering too far into abstract tangents?

I'll close with a question. Consider two hypothetical developers:

• Developer A is a lead developer on a popular project with 100 stacks, but he has not received any direct kudos.

• Developer B is not a contributor to any popular project (or the code for his project is not on Ohloh), but he has received 10 direct kudos from several important developers.

Which of these developers do you think should have a higher KudoRank? How many stacks vs. how many kudos makes two developers equal?

Robin Luckey almost 18 years ago

Hi Robin,

thanks once more for the clarifications and explanations, esp. the link to PageRank, and also about the zero sum principle. That makes a whole lot sense.

As for projects not receiving Kudos: Aye, you are right of course (no surprise, you are undoubtedly the expert ;) they don't. And I very badly expressed myself there. My point was more like this: If I stack a project, I assign a certain amount of kudos to each developer of the project. Another way to view this is that I assign some kudos to the project, which is then split between the developers of the project. So, while technically speaking I assign Kudos to the project members, not the project itself, from my point of view I didn't do any such thing, I gave Kudos to the project (I stacked it, thus marking it as useful to me). If I now look at the profile of a developer of that project, I do not see the indirect Kudos; I only see the direct kudos, and the size of the stacks. Hence if there are two people without any direct kudos, but each is member of a (different) project which has been stacked 100 times -- then I'd expect them to have equal Kudos rank. And am irritated if they diverge by several hundred ranks.

So, as a conclusion, I guess what I really want to say is this: Naively spoken I would prefer if stacking a project would assign a fixed amount of kudos to the project resp. it's team members, treating all stackers equal. However, in light of your words on zero sum games, this might not be possible. Still, it would solve two problems: The can't understand indirect kudos when looking at person overview page, and the discourage people from making huge stacks problem.

As for you developer A vs B question: Tricky one, really. In the end it boils down to defining arbitrary weights, resp. a philosophical one: Which color is bigger? green or yellow? well you just can't compare colors that way. Neither can you really compare stacking vs. Kudos, as the relation is different for everybody. It will always require a somewhat arbitrary decision to weld together the two concepts and make them comparable by force ...

If being stacked by 100 people is well above average, then I'd feel this should give a quite big kudos boost. But not as much as direct kudos from 10 important devs -- direct kudos is the true kudos, it means yes, *this* guy did great work. While stacking a project means Hey, this project is great, and only by indirection implies the devs (or at least some of them) can't be totally wrong either, given that they developed this great app.

I think this is also one of the core difficulties with KudosRank: It mixes two different concepts and tries to fit them under a single umbrella. More, if you also try to factor in the activity level of a project and its member by considering the commits counts...

Cheers,
Max

P.S.: Any chance you guys could allow for a little bit wider columns in this forum? It just doesn't seem to be made for people like Robin and me who prefer writing too long posts ;-)

Max Horn almost 18 years ago

slightly wider columns - done.

Jason Allen almost 18 years ago

Actually, I don't think it's impossible to get around loop abuse, i.e. people pimping up each other by forming a complete graph.

The problem is doing it in a way that doesn't impact legitimate users giving legitimate kudos.

So if 100 anybodies use project X, while project Y is only used by the the top 4 (according to Kudos rank) developers -- then I definitely would rank X higher.

If the top 4 people on this site shared a common project in their stack, I'd definitely be interested in looking at it to see what makes it special.

Going further, assume project A and B are stacked by 10 people, but the 10 stackers of A each stacke 100 projects, while those 10 stackers of B only stacked B and nothing else. Then I would expect that A and B get the exact same boost. I simply see no justification to do it differently... :-)

There is an argument that says that since the people stacking project A are stacking so many projects, each project in the stack has less importance in their eyes than the over all.
Similarly, if you give kudos to 100 people, there is a logical leap that says that those 100 people have less kudos in your eyes than if you felt only 10 deserved your kudos.
Think of it as you having £100 to give away ... if you split it between 100 people, then those 100 people probably have less need of it in your eyes than if you split it between just 2 people.

Yes that argument is arguably flawed, but that's beside the point. The more critical issue is that ohloh is trying to create opposite trends for user kudos and project kudos, but treating them both with the same system.
Specifically, the former is aimed at generating a trend of giving kudos to those we feel most deserve it, while the latter is aimed at giving kudos to every project we use.

If two people kudo eachother, they form a symmetrical system where each takes as much as he gives, and they end up with the same score they began with.

Not quite .... logically, A -> B -> C -> D -> A will cause all 4 accounts to have far more kudos than they started with, because an account with no kudos given to it must still have a non zero amount of kudos to give, and giving kudos doesn't decrease the amount you personally have. Thus each time kudos scores are recalculated, the actual kudos value of each account should work out about the same but increase relative to the general population, which each increase being of a decreasing amount.

Kudos are person-centric, and don't have anything at all to do with ranking projects. Projects don't get a boost from kudos.

Kudos and stacks are two separate features. The KudoRank algorithm itself doesn't know anything about stacks, and stacks definitely don't know anything about kudos.

Uh, wait a sec, I thought that having 5 kudos gives and 5 projects stacked resulted in each being given 1/10th of your influence ... that's not very separate.

Also, we feel that being a key developer on a project with a lot of stacks ought to be worth something in the KudoRanks.

Indeed, but stacking a lot of projects shouldn't decrease the value of your opinion on another person.

If being stacked by 100 people is well above average, then I'd feel this should give a quite big kudos boost. But not as much as direct kudos from 10 important devs -- direct kudos is the true kudos, it means yes, this guy did great work. While stacking a project means Hey, this project is great, and only by indirection implies the devs (or at least some of them) can't be totally wrong either, given that they developed this great app.

Interesting thought ... you could derive kudos from projects based on the number of people stacking that project relative to other projects... although I think I prefer the separate pool of kudos and the the weighted distribution discussed above as a solution.

Oh and yay for bigger columns :D

Daniel / Nazca ... almost 18 years ago

OK, I just noticed a new wonderful example which highlights some problem with the KudosRank:

Take a look at Mr. 3409. He is ranked 565th at the time I am writing this. He neither received nor gave any direct Kudos. However, he performed 98% of the work on the single project he is member of. This project was stacked by four people -- one Mr. 3409 himself (he's a 9er), and three level 1 guys.

Compare this to e.g. my buddy Mr. 803, right now only at 1344, he's on level 8. Yet he got direct Kudos from three people, and some stack Kudos. Just weird.

I assume the system handles the case where Mr X is in project Y and stacks it, so that he doesn't give himself Kudos that way?

Does the system

Max Horn almost 18 years ago

Hi Max,

You cannot give Kudos to yourself by any means. We got at least that much right. :-)

The key point to keep in mind here is that Mr. 3409 and Mr. 803 are on isolated islands. They have no Kudo senders in common, and no stacks in common. So although we know that 3409 and 803 are near the top of their respective groups of kudoers, we have no way to know how these two groups as a whole rank relative to eachother.

All we can tell from the data we have is that Mr. 3409 is very important to three people -- he is the sole author of an application stacked by three people, and those three people have stacked little else.

Meanwhile, not to insult Mr. 803 in any way, but he has received kudos from 3 people who have been handing out a relatively large number of kudos, and his kudo score input from stacks is not significant.

This is not much data to work with, and with no direct link between the two groups of people, I think the system doesn't do a half-bad job.

That being said, I am still planning to roll out the limit on how many Kudos you can get from a single person. And while this will make it difficult to for 3409 become a 9 on the strength of just 4 stacks, it will also make it more difficult for 803 to score highly based on 3 kudos and weak stacking. I've started inching this adjustment in already, and it will be fully rolled in by the end of this week.

The main point is that the two groups of people need some kind of common link before their relative comparisons become trustworthy. We need ever more data.

Robin

Robin Luckey almost 18 years ago

just to make sure, kudo isn't inversely related to number of commits is it? because this looks wierd to me: http://www.ohloh.net/p/bbdkp/contributors

sajaki about 16 years ago

Hi sajaki,

These kudo rankings are weird because you are the only source of kudos in this list.

Kudos are assigned to the team when Ohloh members click the I Use This button to add the project to their stack. So far, you are the only person to do this on Ohloh.

Because our system doesn't let you give kudos to yourself, everyone on the team is getting kudos but you.

In order to get some kudos to yourself, you'll have to find some other Ohloh members to add this project to their stacks.

Robin Luckey about 16 years ago

I had dinner with Scott Collison about a year ago. Knowing I was going to meet Scott, I spent some time browsing around ohloh. When we finally met I gave him my expert advice on kudo ranking:

Scott, it doesn't matter what algorithm you use as long as Linus comes out on top.

(Linus was then ranked #5, and I felt sorry for the programmers whom ohloh had deemed superior.)

daveshields about 16 years ago

Looking at sajaki's project (2/3 comments above) it would seem to me that a fairer system (sorry I'm no mathamatician) would keep the kudo ratio and commit ratio equal where possible. What i mean is if it's not giving him kudos when he stacks, it should not give the others either.

I think that would certainly even out the kudos somewhat.

Or even limit a contributer's stacking kudos to an arbitrary lower maximum (eg: 0.01) for himself. and if his max is 0.01 then someone contributing double the amount of commits on the same project would get 0.02 and someone commiting 100x what that cotributer is commiting would get 1 kudo

(?)

It seems odd to me that if one person who is a contributer happens to stack their own project (don't we all use our own projects?) it automatically lessens their own karma in relation to other contributers

Clive Crous about 16 years ago

Ah, reviving the dead (threads)...

This is an unclaimed contributor, with 3 commits to ubuntu, and kudos by two rank 1 people -- and he gets rank 10, and inf act is currently #2 in the overall Kudos list: https://www.ohloh.net/p/ubuntu/contributors/16086800264953

Kudos for achieving such a high rank with so little effort ... ;)

Now, Ubuntu is of course a huge project, and probably stacked by tons of people. Yet this is a developer with just 3 commits (and on a tiny subproject of ubuntu for that matter). But even taking everything on this thread into account, I find it hard to accept that he is ranked higher than all those level 9 folks... e.g. the rank #65 guy who is one of the authors of GNU tar, and received kudos from two other level 9 and one further level 7 user...

Yeah, yeah, maybe they are in different disconnected subsets of the kudos graph, maybe ubuntu is stacked a million times more than ubuntu, maybe those guys have spread their kudos 20-fold each... Still, it's hard to understand emotionally. :)

Max Horn over 13 years ago