I worked with Ferrucci in BW. The internals of the employee rating system that preceded him were hilarious. In short:
Everyone is encouraged to rate anyone else in a variety of categories, as often as possible. Every rating is public. You know who is rating you, and how they did it. Those ratings are put together to get a score in every category, and can be seen by anyone. It's your Baseball Card.
The problem is that not everyone is equally 'credible' in their rating. If I am bad at underwater basketweaving, my opinions on the matter are useless. But if I become good, suddenly my opinion is very important. You can imagine how, as one accumulates ratings, the system becomes unstable: My high credibility makes someone else have bad credibility, which changes the rankings again. How many iterations do we run before we consider the results stable? Maybe there's two groups of people that massively disagree in a topic. One will be high credibility, and the other bad, and that determines final scores. Maybe the opinion of one other random employee just changes everyone else's scores massively.
So the first thing is that the way we know an iteration is good involves whether certain key people are rated highly or not, because anything that, say, said that Ray is bad at critical thinking is obviously faulty. So ultimately winners and losers on anything contentious are determined by fiat.
So then we have someone who is highly rated, and is 100% aware of who is rating them badly. Do you really think it's safe to do that? I don't. Therefore, if you don't have very significant clout, your rating of people should be simple: Go look at that person's baseball card, and rate something very similar in that category. Anything else is asking for trouble. You are supposed to be honest... but honestly, its better to just agree with those that are credible.
So what you get is a system where, if you are smart, you mostly agree with the bosses... but not too much, as to show that you are an independent thinker. But you better disagree in places that don't matter too much.
If there's anything surprising, is that more people involved into the initiative stayed on board that long, because it's clear that the stated goals and the actual goals are completely detached from each other. It's unsurprising that it's not a great place to work, although the pay is very good.
> My high credibility makes someone else have bad credibility, which changes the rankings again. How many iterations do we run before we consider the results stable?
Tangential to your point but this is actually a solved problem and you only have to run once. You might even recognize the solution [1] [2].
Why only once? When we did pagerank in class we chose a precision factor and then iterated pagerank until the delta values for the matrices were less than the precision factor
More generally factor graph optimization / msg passing, if you don't need the constraint that it must be an eigenvector/eigenvalue operation. The number of iterations is bounded by graph width but is locally constant, if memory serves.
I'm surprised by your comment. It's a basic knowledge of getting feedback in UX research or any social research that social pressure influences your response. That's why most feedback surveys are anonymous. Even in politics, there is an old theory called the Spiral of Silence on how people shut down their contrarian opinions in public.
It looks like Ray's fixation on "radical transparency" ignored the basics of human nature.
Unfortunately, no one will trust that a work survey is anonymous. They sent it to an email address that includes my name, with a unique link. And it's often possible to just infer who it is.
I like my employer very much but still do not trust that their frequent employee surveys are anonymous. But since managers get grief for low response rates, I do the surveys, but skip each of the questions.
We have Six Seasons already and the movie is apparently in production now(Although the strike has delayed things a bit and may cause downstream trouble).
Another aspect of this kind of system is that it favours visible over invisible jobs. The people working their asses off in the technology sewer might literally keep everything running, but because they are:
A) underfunded and understaffed
B) have no time and energy for niceties
all those in more comfy positions might be inclined to vote against them, while they probably don't vote at all because they got no time for that shit.
As you said, if you let a pidgeon vote on how good your chess moves were you will get pidgeon shit on a chessboard. A costly form of bureaucratic performance art, sans the art.
I worked at a place where publicly praising folks via company wide emails was encouraged. It was a painful experience. Almost immediately mutual admiration societies started up. It was like a competition between these small groups to send these nauseating emails every Friday. Mostly it was sales, HR, and the executive team sending these emails.
Most folks just didn't participate and it faded away pretty fast.
This is classic example of if you give people a metric (that is meant to be a condensation of things that truly matter) based on which they are evaluated it, they will just start optimizing for and gaming that metric instead of focusing on things that truly matter.
It would be so cool if someone could apply game theory for mortals that could digest these systems and show how to make them more "fair". It may be an impossible task but it seems worthy of exploration.
> It would be so cool if someone could apply game theory for mortals that could digest these systems and show how to make them more "fair". It may be an impossible task but it seems worthy of exploration.
I consider this to be a hard, but clearly not an impossible task. What I rather consider to be nigh impossible is to get a good definition on what we actually want to achieve by creating these systems. There is where in my opinion the true difficulties lie.
Just to give some shallow examples:
Do we want to make this purpose be quite fixed (say, for decades) (and create a system around this purpose)? Then it will be quite hard to change the ship's directions if because of some event the economic environment changes a lot.
On the other hand: do you want to make the system's purpose very flexible, so that the system can react to such circumstances? You can bet that this will be gamed towards the political whims of the people inside the system.
On the other hand: if you want to get a glimpse at system ideas that did "work out", look at long-existing religions, or entrepreneurial dynasties with a history of centuries. It should be obvious that these examples are far off from a "democratic", "particapative" spirit (which perhaps should be a lesson for anybody designing such systems).
My goal would be to "maximize prosperity for all". It could be better qualified and defined, but making sure honest hard-working people don't get screwed over and cheaters don't prosper.
I think the goal should be the goal, as the circumstances will continue to evolve. That is, the process should serve the people rather than the other way around.
Sorry, I'm just an old naive idealist and would like to cling to hope that we can move society forward in a positive direction.
I have met or value relationships with plenty STEM BW alums, but they tend to speak about it as a funky place, seemed to color their views about buy side jobs and suggest that you know what you’re getting into with a job there.
Seems anecdotally that it’s used as a stepping stone into leadership or used to bank savings.
Several tech leaders I know are out of BW.
An SRE I knew there was at ~$600 TC and got a retention offer at ~$900 when they left to do a passion job.
Everyone is encouraged to rate anyone else in a variety of categories, as often as possible. Every rating is public. You know who is rating you, and how they did it. Those ratings are put together to get a score in every category, and can be seen by anyone. It's your Baseball Card.
The problem is that not everyone is equally 'credible' in their rating. If I am bad at underwater basketweaving, my opinions on the matter are useless. But if I become good, suddenly my opinion is very important. You can imagine how, as one accumulates ratings, the system becomes unstable: My high credibility makes someone else have bad credibility, which changes the rankings again. How many iterations do we run before we consider the results stable? Maybe there's two groups of people that massively disagree in a topic. One will be high credibility, and the other bad, and that determines final scores. Maybe the opinion of one other random employee just changes everyone else's scores massively.
So the first thing is that the way we know an iteration is good involves whether certain key people are rated highly or not, because anything that, say, said that Ray is bad at critical thinking is obviously faulty. So ultimately winners and losers on anything contentious are determined by fiat.
So then we have someone who is highly rated, and is 100% aware of who is rating them badly. Do you really think it's safe to do that? I don't. Therefore, if you don't have very significant clout, your rating of people should be simple: Go look at that person's baseball card, and rate something very similar in that category. Anything else is asking for trouble. You are supposed to be honest... but honestly, its better to just agree with those that are credible.
So what you get is a system where, if you are smart, you mostly agree with the bosses... but not too much, as to show that you are an independent thinker. But you better disagree in places that don't matter too much.
If there's anything surprising, is that more people involved into the initiative stayed on board that long, because it's clear that the stated goals and the actual goals are completely detached from each other. It's unsurprising that it's not a great place to work, although the pay is very good.