Your pilot group is not representative

Pilot groups get assembled from volunteers, and volunteers come from the teams that are curious about IT changes. That is usually one department, on hardware bought in the same procurement round, running a build that was imaged at the same time.

You have not tested ten users. You have tested one configuration, ten times, and you now have high confidence in a result that does not generalise.

What actually varies

The dimensions that produce different migration outcomes are rarely the ones we recruit along:

• Office version and update channel. The difference between a current channel install and a machine two years behind is where most client side surprises live.
• Profile age. A profile carried through three machine refreshes carries accumulated configuration that a rebuilt one does not. These are the machines that break.
• PST attachments. Users with large local archives behave completely differently during a profile rebuild, and they are disproportionately the long tenured staff nobody put in the pilot.
• Device join state. Domain joined, hybrid joined, Entra joined, and workplace joined machines take different paths through authentication. A pilot from one team is usually all one state.
• Working location. On the corporate network, the SCP lookup resolves. From home over VPN, sometimes. From home without VPN, not at all. Three different failure profiles.

How I pick now

I stopped asking for volunteers and started selecting against those dimensions. Pull the inventory, sort by Office build, profile age and join state, and take one machine from each cluster. Then add the two users everyone in support already knows by name, because their machines are unusual and that is exactly why they are worth testing.

Ten users chosen this way find more than fifty volunteers from one floor.

The one you always miss

The person on extended leave. They come back three weeks after the cutover to a machine that was off for the entire migration, that has not had a policy refresh, and whose credentials expired while it was in a drawer. Nothing in the pilot covers this because the pilot ran on machines that were switched on.

It is not a big population, but it is a guaranteed one, and knowing it is coming turns a strange escalation into an expected item on the list.