How to Run a Calibration Meeting at a 500-Person Startup

At 50 people, calibration is one conversation between the founders. At 5,000, it is a programme with its own team. At around 500, it is the awkward middle: too many people for one room, not enough People team capacity for a heavyweight process, and enough managers that a “4” already means different things in different departments. This guide sets out a calibration structure that works at that size, step by step.

It builds on the general guide to running a calibration meeting and the quarterly process in the performance management framework. Timings below are the playbook’s own guidance and practical starting points, not industry benchmarks.

Why 500 people changes calibration

Three things happen as a startup approaches 500 employees. First, there are now 50 to 80 people managers, many promoted in the last two years and rating people for the first time. Second, departments have developed their own norms: engineering may rate tough, sales generous, operations in the middle. Third, leadership no longer knows everyone, so it cannot sanity-check ratings from personal knowledge.

The result is the problem Kahneman and colleagues call “noise” in Harvard Business Review: people applying the same guidelines reach different judgements on similar cases. Research summarised in HBR’s “Reinventing Performance Management” found that a large share of the variance in ratings reflects the rater rather than the person rated. Calibration is the step that catches this before ratings drive pay and promotion.

McKinsey’s 2024 article “In the spotlight: Performance management that puts people first” describes committees as a way to bring diverse perspectives and mitigate individual managers’ biases, and suggests combining manager judgement with committee review. A tiered calibration does exactly that.

The structure: two tiers, not one giant room

Do not try to calibrate 500 people in one meeting. Split it into two tiers.

Tier	Who is calibrated	Who attends	Length
Tier 1: department sessions	Everyone in the department, typically 40 to 120 people	Department head, the managers whose reports are discussed, a People partner as facilitator	About 90 minutes to 2 hours per session
Tier 2: leadership session	Top and bottom ratings, promotion nominations, cross-department outliers	Executive team, Head of People, People partners	About 2 hours

For a 500-person company that usually means five to eight department sessions, plus one leadership session. Large departments (often engineering) can split by sub-function so no session discusses more than about 80 people. Session lengths are illustrative; the goal is that the room has time to discuss outliers, not every individual.

Who does what

Facilitator (People partner). Owns the pre-read, keeps time, asks for evidence and logs every change. Does not argue for or against individual ratings.
Department head. Owns the outcome for the department and makes the call when managers disagree.
Managers. Present evidence for their top and bottom ratings and accept challenge from peers who have worked with their reports.
Head of People. Chairs Tier 2, checks consistency across departments and owns the final sign-off process.

Before the meeting: preparation that makes or breaks it

Publish the bar. Every manager should have the written talent bar by seniority and the scorecards before they rate. Calibrating against an unwritten standard turns into a debate.
Close ratings before the session. Set a firm deadline for manager scorecards, a few working days before calibration, so the facilitator can prepare.
Require evidence notes. Each rating needs a one-line evidence note per scorecard area. No note, no rating.
Build the pre-read. For each department: the distribution by manager, a list of people rated at the top and bottom, anyone whose grade moved by more than one level since last cycle, and missing scorecards. Send it at least a day ahead.
Run a quick pattern check. Paste each manager’s ratings (numbers only) into the ratings spotter to flag possible leniency, severity or central tendency. Use the flags to decide where to spend time.

The department session agenda

Time	Item	Owner
0–10 min	Ground rules: we discuss evidence, not personalities; the bar is the reference; changes are logged	Facilitator
10–25 min	Distribution review by manager: who is rating high, low or flat?	Facilitator
25–60 min	Top ratings: each manager evidences their A-player ratings against the bar	Managers
60–85 min	Bottom ratings and underperformance: evidence, prior conversations, next step	Managers, People partner
85–105 min	Outliers and jumps: anyone whose grade moved sharply, plus missing or thin evidence	Facilitator
105–120 min	Read back changes, confirm promotion nominations for Tier 2	Department head

Notice what is missing: walking through everyone rated in the middle. At 500 people that is most of the company, and the evidence for “meets the bar” is rarely contested. Spend the time where decisions are made.

The three checks, at scale

The playbook’s calibration process checks three things. At 500 people each needs a concrete rule.

Generous graders. If one manager has rated half their team as A-players, ask them to name their strongest performers and evidence each one. The playbook’s guideline weights (15–25% A-players, 65–85% above bar, 0–10% underperformers) are a prompt for questions, not a quota; McKinsey describes companies moving from forced distribution to distribution guidance for the same reason.
Missing or thin evidence. Any rating without an evidence note goes back to the manager before the session, not during it.
Outliers and sudden jumps. Anyone who moved more than one grade since last cycle gets a short discussion: is the change genuine, or a grading mistake?

The leadership session

Tier 2 does not re-run department calibration. It looks across departments. Bring a single view of the distribution by department, the list of promotion nominations, everyone rated at the top and bottom, and any rating the department session could not agree. The executive team’s job is to check that a top rating in engineering represents the same standard as one in sales, approve promotions, and spot teams that need more attention. The playbook allows about a week at most for calculation, calibration and leadership review together.

What to record

For every change: original rating, new rating, reason, the evidence discussed, and who approved it. Keep this log with the review record. It protects managers when they explain a result, gives employees a fair answer if they ask why, and shows next cycle which managers consistently rate high or low.

After calibration

Tell managers the final results before employees hear them, and brief them on any rating that changed so they can explain it. Results are then discussed in each employee’s next 1:1, with a clear next step: promotion path, development focus or improvement plan. Book a 30-minute retrospective with the People team to note what slowed the cycle.

Common mistakes at this size

One huge session. A room of 40 managers discussing 500 people produces fatigue, not fairness.
Calibrating without the bar. If the standard is not written down, the most senior or most persuasive person sets it in the room.
Discussing everyone. Focus on the top, the bottom and the outliers.
Letting changes go unrecorded. Undocumented changes are hard to defend and impossible to learn from.
Skipping the cross-department view. Without Tier 2, each department is internally consistent but the company is not.

How much time this takes

Six department sessions of two hours with eight people, plus a two-hour leadership session with ten, is 96 + 20 = 116 person-hours per cycle (illustrative figures). Put your own numbers into the review cost calculator to see what calibration costs your company each year, and where pre-reads and tighter agendas would save the most. For a ready-to-copy agenda and log format, use the calibration meeting template for mid-market tech companies.

Sources: Kahneman et al., Noise, HBR (2016); Buckingham & Goodall, Reinventing Performance Management, HBR (2015); McKinsey, In the spotlight (2024). Session structure and timings are illustrative playbook guidance.

Use with this guide

Free tool: Performance review bias checker
Method: Performance management framework
All tools: free performance management tools

How to run a calibration meeting for a 500-person startup