My take: multi turn evals are hard because to do it really correctly you have to simulate a user. This is not yet modeled well enough for multi turn to work as well as it could.
The flip side of this is that for some tasks (especially in ml/ai), doing it manually at least a few times gives you a sense of what is correct and a better sense of detail.
For example, spending the time to label a few examples yourself instead of just blindly sending it out to labeling.
(Not always the case, but another thing to keep in mind besides total time saved and value of learning)
I think the methods here are highly questionable, and appear to be based on self report from a small amount of employees in Denmark 1 year ago.
The overall rate of participation in the labor work force is falling. I expect this trend to continue as AI makes the economy more and more dynamic and sets a higher and higher bar for participation.
Overall GDP is rising while labor participation rate is falling. This clearly points to more productivity with fewer people participating. At this point one of the main factors is clearly technological advancement, and within that I believe if you were to make a survey of CEOS and ask what technological change has allowed them to get more done with fewer people, the resounding consensus would definitely be AI
I think that “I’m not technical” is often an excuse for throwing work at other people and frankly can be a form of learned helplessness. Nowadays, there is less and less reason to ask other people to write one off scripts/queries, you can ask AI for help and learn how to do that.
Since this is HN some disclaimers
-no that’s not always what’s happening, when “not technical” is thrown around
-no it’s not always appropriate to use AI instead of asking an expert
It may be a good thing to throw scripts off to someone else. Division of labor is a good thing. You cannot possibly learn everything to a good (not even high) standard. Even if you could, no lawyer would have themselves as a client - when a lawyer needs legal advice they go to a different lawyer because they want that different perspective: this is often a good perspective for other subjects as well.
The question is what you will/should learn for your limited time alive. Society needs well educated (I include things "street smarts" and apprenticeship in educated here) people in many different subjects. Some subjects are important enough everyone needs to learn them (reading, writing, arithmetic). Some subjects are nearly useless but fun (tinplate film photography) and so worth knowing.
Things like basic computer skills are raising to the level where the majority of people today need them. However I'm not sure that scripting is itself quite at that level. (though it is important enough that a significant minority should have them)
I’m talking about a general trend I see in use of this term, not that it’s always a bad thing to say “I’m not technical so someone else should write the script”
I agree with everything you said!
Both things are happening in the world: people using this terminology to throw work at others needlessly, and people doing good division of labor.
It seems like if they in fact distilled then what we have found is that you can create a worse copy of the model for ~5m dollars in compute by training on its outputs.
“In my humble opinion, these companies would not allocate a second of compute to lightweight models if they thought there was a straightforward way to achieve the next leap in reasoning capabilities.”
The rumour/reasoning I’ve heard is that most advances are being made on synthetic data experiments happening after post-training. It’s a lot easier and faster to iterate on these with smaller models.
Eventually a lot of these learnings/setups/synthetic data generation pipelines will be applied to larger models but it’s very unwieldy to experiment with the best approach using the largest model you could possibly train. You just get way fewer experiments per day done.
The models bigger labs are playing with seem to be converging to about what is small enough for a researcher to run an experiment overnight.
> You just get way fewer experiments per day done.
Smaller/simpler/weird/different models can be an incredible advantage due to iteration speed. I think this is the biggest meta problem in AI development. If you can try a large range of hyper parameters, fitness function implementations, etc. in a few hours, you will eventually wipe the floor with the parties forced to wait days, weeks and months for their results each time.
The bitter lesson certainly applies and favors those with a lot of compute and data, but if your algorithms fundamentally suck or are approaching a dead end, none of that compute or information will matter.
> This has not at all been my experience. When forced to do layoffs in a large company, executives tend to look at performance reviews
Speaking specifically to Facebook and Instagram, I know of more than one team where the manager wasn't consulted when someone higher up (I think they were advised by BCG) chose whom to cut.
The kicker? They frequently cut the highest paid. That obviously removed with a bias towards seniority. But it also meant that managers woke up to find their best-bonussed people gone while their worst performers--the cheapest on paper--remained.
Happened to me. Ranked near highest in company, promoted, pay bump two months before a layoff targeting US staff. They kept engineers in Australia who made 50-60% less. Before severance ran out, I landed a job that paid 50% more than before. Layoffs don’t indicate that much about the people laid off; they say a lot more about management.
I have found at the companies I have worked for it is two things for a layoff: cutting a product/change in strategy and how much you are being paid. Layoffs for performance tend to be more one off than a massive cut.
What I am most surprised about is how many really good performers get cut for a product or strategy change. A company will be cutting a high performer while searching for a high performer at the same time like it is taboo to move people within the company.
IME, it's based off of "performance", not performance. So what happens is someone 4 levels above you on the org chart has a google sheet of all their underlings, their most recent grade - erm I mean performance review result - and their total comp.
They sort by total comp, then go down the list and figure out a reason to let that person go. Was there literally anything in your performance review summary they can ding you for? Yes? Phew, that was easy. No? Well, keep looking. "Strategic mismatch for skillset" "too junior, want senior" "too senior, want junior" "role eliminated due to headcount allocated to team being reduced" etc.
So, in the layoff I was privy to, somehow everyone who still had the large lucrative 4-year stock-denominated grants was suddenly gone, and the people who had the newer cash-denominated grants were still there. Meanwhile, several cheaper employees who were perennially underperforming were retained.
Honestly it really soured me on equity grants. It's a game of "heads I win (my company didn't grow and i got to pay you peanuts), tails you lose (my company grew and now i can just fire you and re-hire someone with a cheaper grant so you can't vest those now-very-lucrative appreciated shares)".
One of my former employers did a big layoff last year -- lots of folks who I remembered as being "important" (long time employees, large contributions, lots of domain knowledge) were let go. Seemed pretty dumb, but the wreckage stumbles forward anyway.
> Seemed pretty dumb, but the wreckage stumbles forward anyway.
the momentum in the org will continue its course, but without those long time employees who have the deep domain knowledge built up over the years, there's no way to steer nor alter course. So it's luck that a company continues on, because this momentum can't do anything else but continue on the current course.
I think it depends on the size of the company and who ends up being involved in the decision amking and when.
From what I have seen, the amount that someone is making can be a contributing factor. Maybe your performance reviews but that doesn't always paint a full picture of your actual performance since those are often kinda black and white.
In every layoff I have been a part of (wether or not I was personally affected) the managers found out the morning of and were not consulted before it happened. In more than a few cases someone critical was let go.
Counterexample 1: company had two products, one java one c++ based. Founders decided to focus on one product to extend runway. Everyone on 2nd product team laid off, regardless of perf reviews.
Counterexample 2: layoff needed to boost short term profit metrics for potential sale of company. Since focus is cost cutting, not long term viability, expensive folks targeted (ie senior high performers)
Counterexample 3 (union shop, yes rare I know but gov and academia often have union IT workers. Layoffs are purely seniority based (as in most recently joined union = first to be laid off)
Last layoff I was in, my boss was in the middle of writing my promotion paperwork. The people cut were mainly people with seniority ($$$). The company then sent my contact info to scammers in a probably well-intentioned attempt at placement help.
I've been at two small companies that had layoffs, and who was cut/retained was 100% based on what was best for the company (except some visa holders were retained). Which is what it should be.
What you said is one of the reasons a person can be laid off. Another is when the product, project, division, etc. is shut down and all personnel are laid off regardless of performance. Sure, they'll be given an opportunity to find a new role within the company but there are always far too few openings for more than a few to be retained.
In my experience performance reviews don't accurately differentiate the high performers vs the low performers. No one gets below average unless you are really struggling. The only real currency is if PMs want you on their team or not.
My experience is that a list of employees ordered by (compensation/contribution) is created, sorted highest to lowest. Those nearer the top of the list are most likely to be let go, those lower on the list are most likely to stay. And, contribution != performance. Profitability (margin) and revenue growth of the product and team affect contribution as much as individual performance.
I don't think this is necessarily true for some larger companies. Sometimes, entire divisions are axed. It's more trouble than its worth (to the higher ups) to pick out the few good ones when the company is in turmoil. In startups, I agree, it's never "nothing to do with performance."
We can catch things early, it shouldn’t be limited to only for smokers.