Peer learning and refereeing

Nicholas Bagley over at the Incidental Economist passes along a great article about the value of non-academic peer review and its impact on learning:

Beginning in 2014, we designed a randomized, controlled trial to test the effectiveness of peer review with the food safety staff of King County, where Seattle is located. Half of the inspection staff was randomly assigned to engage in peer review. For sixteen weeks, these inspectors spent one day per week with a randomly selected fellow inspector, taking turns conducting inspections and independently scoring health code violations. We then used information from these peer inspections to identify and train for violations that cause the most confusion.

The results were remarkable. We discovered that, when observing identical conditions in restaurants, health inspectors disagreed nearly 60 percent of the time. Inspectors differed in their assessments of risk magnitude and in interpretations and applications of the health code to particular circumstances, resulting in varying citations for the same condition.

He is very excited to see peer review and peer coaching in the medical realm as a means of transforming the practice of medicine from folk ways informed by science to a more systemic practice. There are significant inconsistencies in treatment methods for the same example patient depending on where the doctor was trained, where he currently practices and what equipment is available (and who has an ownership stake in all of the various billable opportunities).

To some extent, peer review has always been a part of medicine—think here of M&M conferences. And it’s starting to get more attention. As Ho and Elias note, Atul Gawande wrote an important New Yorker article in 2011 about surgical coaching. Here at the University of Michigan, Justin Dimick has landed an NIH grant to investigate surgical coaching more generally.

My first thought was on soccer refereeing.

I’m at the point in my soccer career where I am spending a decent amount of time every year going to clinics, reviewing game film, watching highlighted clips from the MLS and international leagues to see what USSF wants us to do. I also get several formal assessments a year from either USSF or NISOA (the college referee umbrella group). These formal moments of training are valuable as they provide me with a framework of what I should be doing. (As a reminder I am not and never have been on the MLS track — those guys and girls are doing this 20 hours a week or more)

However it is not the best training I receive each year.

The best training is the long car rides home after East Nowhere State beat Palookaville College. This is where extremely strong post-game analysis occur.

Those rides are with two or three other referees who I trust know what they are talking about and with whom I have a long standing relationship with.

We can break down game situations. We can talk about body language. We can talk about positioning. We can honestly point out that I got to the goal line late as the assistant referee so there was unnecessary uncertainty on a corner kick/goal kick decision. I can tell a peer that I think he makes great decisions but that he needs to assertively own his decision. Right now he passively indicates direction or a serious foul or a trifling control foul with minimal je ne sais quoi authority. He is 5’4″ and will get run over unless he owns the field. We can talk about an orange card situation where I went yellow while my AR-1 thought it was a solid red and my AR-2 thought it was a legitimate yellow.

On a ride back from a regional identification tournament, the three of us talked about a single decision for three hundred miles and then another decision for seventy five more miles. These conversations can be extremely detailed and brutally frank moments of self-assessment and peer assessment. I have been told my people that I greatly respect that I truly fucked up a game (I agree with that assessment for that particular game, my head was not there at the start and the teams did something I was not ready for). I have told peers that a decision style that they use would get people hurt and would end careers. It can be a brutal car ride after a bad game and even after a good game, flaws will be identified, decisions points questioned. Very little smoke is blown up peoples’ asses.

These discussion work for two reasons. First, the conversation stays within the car. It is not reported back to assessors, it is not reported back to assignors (although sometimes they are in the car as well as they were running the middle) and it is not reported back to coaches. The conversation’s repercussions end the moment the car pool ends.

Secondly it is between peers. I know that if Amy is making a point about my body language contradicting my foul selection it is because she picked something up in her 2,900 games worth of experience. If John is arguing that a scenario where I went double ass-chewing when it should have been double yellow, it is because he had a similar experience in the MLS and got called out by a PRO assessor. If Ben is telling me I got too wide, I know I too wide and placing too much on him from to call. If I tell Andrew that he really needs to watch the hand check fouls because it was driving the game into the gutter, he trusts my judgement. When I told Al to own his decisions, it was because I had worked with him for seven years and had seen how it detracted from his games.

We’re peers. We have credibility with each other and we know that we are not trying to game each other for some type of advantage. If they make me a better assistant referee, it improves their chances of having a good game in the middle. If I make them a better referee on a tiny facet, it makes my experience as an AR better. I know that Chuck has been caught in a situation where there are seven touches within the 6 in two seconds or less and he groped towards figuring out an offside decision so the advice he gives on how he handled it makes everyone better.

This experience of peer coaching is extremely valuable and it should be transferable to most highly skilled domains as long as the incentive structure for the coaching is well thought out so that the act of receiving and learning from coaching is non-punitive, this seems like a good idea to me.