Aggregation Theory and Education

Five years ago, Marc Andressen took to the pages of the Wall Street Journal to describe how software was “eating the world”:

“More and more major businesses and industries are being run on software and delivered as online services—from movies to agriculture to national defense. Many of the winners are Silicon Valley-style entrepreneurial technology companies that are invading and overturning established industry structures. Over the next 10 years, I expect many more industries to be disrupted by software, with new world-beating Silicon Valley companies doing the disruption in more cases than not.”

It’s not hard to come up with a list of examples of this prediction coming true, from taxis (Uber) to hotels (AirBnB), but software-driven disruption has yet to significantly change K-12 education. Existing online education offerings, such as Coursera and Khan Academy, show promise, but few would argue that their impact on K-12 education to date could be described as anything close to “disruptive.” Jay P. Greene explains why all-online education has failed to gain traction:

Online courses appear to be less effective in getting the average student to learn and I suspect the problem is that teaching online is less able to create social communities and authentic relationships that are necessary to motivate students. Having a human being in front of students who would be disappointed if students did not learn the material seems important and something that online instruction has not been able to simulate. Students appear to be better motivated to learn when they have an in-person, authentic relationship with a teacher and when they try to please that teacher by working hard to learn. Digital instruction or a human being on the other side of the internet may not be able to create that same relationship and motivation.

The question remains: if K-12 education is going to be disrupted by software, what would it look like? Is there a way to combine the scalability of software with the power of strong interpersonal relationships?

Delving deeper into the disruption of other industries might give us an answer. Ben Thompson’s Aggregation Theory provides a great framework to understand how the Internet is changing entire sectors of our economy:

“The value chain for any given consumer market is divided into three parts: suppliers, distributors, and consumers/users. The best way to make outsize profits in any of these markets is to either gain a horizontal monopoly in one of the three parts or to integrate two of the parts such that you have a competitive advantage in delivering a vertical solution. In the pre-Internet era the latter depended on controlling distribution.

For example, printed newspapers were the primary means of delivering content to consumers in a given geographic region, so newspapers integrated backwards into content creation (i.e. supplier) and earned outsized profits through the delivery of advertising. A similar dynamic existed in all kinds of industries, such as book publishers (distribution capabilities integrated with control of authors), video (broadcast availability integrated with purchasing content), taxis (dispatch capabilities integrated with medallions and car ownership), hotels (brand trust integrated with vacant rooms), and more. Note how the distributors in all of these industries integrated backwards into supply: there have always been far more users/consumers than suppliers, which means that in a world where transactions are costly owning the supplier relationship provides significantly more leverage.

The fundamental disruption of the Internet has been to turn this dynamic on its head. First, the Internet has made distribution (of digital goods) free, neutralizing the advantage that pre-Internet distributors leveraged to integrate with suppliers. Secondly, the Internet has made transaction costs zero, making it viable for a distributor to integrate forward with end users/consumers at scale.”

Applying Aggregation Theory, K-12 schools can be viewed as an industry that integrates the supply and distribution of educational content. Like newspapers, this integrated product is limited to a certain geographic area. It also helps us to understand that most popular education reforms only seek to improve certain aspects of this relationship instead of fundamentally changing it. For example, charter schools may introduce more competition for students (customers), but they still (generally) operate with the same model of integrated supply and distribution of educational content to students.

Some tech companies are trying to flip the equation by integrating the delivery of educational content more directly with the student experience, making the actual content more of a modular component. Apple, Google, and Facebook (in collaboration with Summit Charter Schools) each offer products that allow teachers to provide a digital classroom experience for their students while also maintaining in-person relationships. This may help them avoid the pitfalls of the all-online education efforts identified by Green, but can any of them truly change the integration of content and delivery in K-12 education?

Of the three large tech companies, I think the Facebook/Summit approach has the most potential to deliver Aggregation Theory-style disruption. The online classroom products of Apple and Google are mostly abstractions of a traditional classroom in the cloud: teachers assign work to each student, then students submit it by a given due date. All of this happens online, but everyone in the class is still on the same timeline.

The Facebook/Summit collaboration, named Basecamp, takes a more Netflix-ian approach to delivering content. Here’s how it’s described on the Basecamp website:

“Students work through playlists of content at their own pace and take assessments on demand. They also work with teachers to set short-term and long-term goals and connect these back to their daily actions.”

Under the Facebook/Summit approach, time is no longer a constraint for students that have already mastered certain content, nor is it a limitation for students that may need more time to fully grasp a concept before moving on to the next lesson.

This represents a fundamental shift in the relationship between students,how they learn, and the role of educators in supporting students through that process. The Basecamp platform is still in its infancy and has much room to develop, particularly to address the needs of students with ß or limited English proficiency. However, we should still recognize the potential this approach could have in changing how K-12 schools operate.


Why American Schools Are Even More Unequal Than We Thought

Using data to inform our conversations about public school performance is a good idea, but too often, the measures we use are reduced to imprecise terms like “proficiency,” which can carry several different meanings when describing a local, state, or national assessment1.

As Susan Dynarski notes in The Upshot, this is also a common problem with the most-frequently used proxy for “poverty” in education, Free/Reduced Price Lunch (FRPL) eligibility:

“Nearly half of students nationwide are eligible for a subsidized meal in school. Children whose families earn less than 185 percent of the poverty threshold are eligible for a reduced-price lunch, while those below 130 percent get a free lunch. For a family of four, the cutoffs are $32,000 for a free lunch and $45,000 for a reduced-price one. By way of comparison, median household income in the United States was about $54,000 in 2014.

Eligibility for subsidized school meals is clearly a blunt indicator of economic status. But that is the measure that policy makers, educators and researchers rely on when they gauge gaps in academic achievement in schools, districts and states.”

In practice, this means that when we refer to FRPL students as “economically disadvantaged,” we’re really painting with a broad brush. Thankfully, Dynarski and her co-author, Katherine Michelmore, devised a way to use current FRPL data to produce a more precise picture of student economic disadvantage: instead of looking at FRPL-eligibility in the current school year, we can use longitudinal datasets to look at how many years a student has been FRPL-eligible.

The concept is simple. If you were comparing two fifth grade students, student A and student B. If student A has been FRPL-eligible for a year and student B has been FRPL-eligible for five years, it’s clear that student B has a greater economic disadvantage than student A.

Dynarski continues:

No one ever actively decided that eligibility for subsidized meals was the best way to measure students’ economic disadvantage. The metric was widely available and became by default the standard way to distinguish between poorer and richer children. But it was always an imprecise measure, and we can do better at little cost.

We’ve already seen researchers stand up to advocate for better ways to quantify “proficiency” - I hope we see a similar movement by researchers to advocate for better measures of poverty. Supporting Dynarski’s approach would be a good (and cost-efficient!) step in that direction.

  1. This is a particular problem when NAEP results are released. Just remember, friends don’t let friends engage in misNAEPery 


Book Notes - The Checklist Manifesto

Books have always been important to me. My grandmother was a librarian and as she babysat me, I was able to spend hundreds, if not thousands, of hours searching, consuming, and falling in love with books. That passion continues today, but I find that I have less time to devote to extended, deep periods of reading.

I’m going to follow CGP Grey’s lead and start publicly sharing notes on books I’ve read. Here’s his explanation of why he started:

I’m going to attempt, for a little while anyway, to make public some of my notes from some of the books that I’ve read. This is partly because people are forever asking what I’m reading, but it’s mostly as a way to try and encourage myself to read both more deeply and more frequently – a target I have been trying, and failing, to hit for all my adult life.

All projects and hobbies of mine eventually die from lack of attention if they cannot serve multiple purposes. So it is my hope that these notes will add even more reason to engage more frequently with long-form writing.

There isn’t anything close to a public clamor for my reading list, but I do hope that this practice helps me with his second goal - increasing my incentives to read books with more depth and more frequency. Additionally, I hope that formalizing this process will help me draw better connections between the books I read and what it could mean for education and/or individual productivity.

With that said, I’m going to apply Grey’s template to the last book I finished, The Checklist Manifesto:

The Big Idea

Professionals of all varieties - programmers, doctors, teachers, pilots, etc. - perform cognitively demanding tasks on regular basis in high-stakes settings. They apply the skills they’ve honed through training, their understanding of their field, and their general experience to execute complex tasks. But no matter how competent or prepared these professionals are, they still make mistakes.

“In a complex environment, experts are up against two main difficulties. The first is the fallibility of human memory and attention, especially when it comes to mundane, routine matters that are easily overlooked under the strain of more pressing events.”

“A further difficulty, just as insidious, is that people can lull themselves into skipping steps even when they remember them. In complex processes, after all, certain steps don’t always matter. Perhaps the elevator controls on airplanes are usually unlocked and a check is pointless most of the time. Perhaps measuring all four vital signs uncovers a worrisome issue in only one out of fifty patients. “This has never been a problem before,” people say. Until one day it is.”

There’s nothing wrong with confidence in your ability to practice your profession, but it does breed a degree hubris that even when seemingly benign, can lead to significant errors. Gwande provides several examples of this in the medical field, where skipping seemingly obvious steps can lead to someone having their right knee opened up even if they’re in for an ACL repair on their left knee.

It’s not hard to see how this could translate to the classroom. A teacher might typically use exit tickets to gauge how much students understood the concepts taught in each lesson. It’s easy to imagine a scenario where the teacher assumes a lesson went well and forgoes this step. This may only happen once in a while, but it could be the case that while the lesson seemed to go well, students actually misunderstood the concept that was taught. It’s a much lower-stakes situation than a botched surgery, but even this small disruption could alter the success and pacing of future lessons.

The concept of minimizing mistakes in routine processes is also applicable to data analysis. Each data analysis project involves follows the broad format of importing, cleaning, transforming, and reporting data. If small steps along the way aren’t double-checked, a tiny error can lead to significant problems with the results.

Gwande stresses that using checklists to minimize error isn’t meant to dumb down the work of professionals. Instead, he argues that:

“It is common to misconceive how checklists function in complex lines of work. They are not comprehensive how-to guides, whether for building a skyscraper or getting a plane out of trouble. They are quick and simple tools aimed to buttress the skills of expert professionals. And by remaining swift and usable and resolutely modest, they are saving thousands upon thousands of lives.”

Key Passages

When the B-17 bomber was developed in the mid-1930’s, it was the most complex aircraft ever built. It had four engines when nearly every plane had one or maybe two engines. This, along with the plane’s other features, made the task of piloting it incredibly complex. The first time the plane was tested, it crashed shortly after takeoff not due to any mechanical issue, but to pilot error. The new design was simply too complex for one person to manage on their own.

Development on the plane continued and another test flight was scheduled. This time, the pilots developed what is now a staple of aviation: a checklist. It was concise, but it ultimately helped flight crews man the B-17 without incident from that point forward.

I found this to be a terrific example of intellectual humility by the test pilots. They were extremely well-trained, but they were willing to admit to themselves that while they were experts, their brains weren’t reliable enough to ensure perfect execution every single time.

“Substantial parts of what software designers, financial managers, firefighters, police officers, lawyers, and most certainly clinicians do are now too complex for them to carry out reliably from memory alone. Multiple fields, in other words, have become too much airplane for one person to fly.”

Educators and data analysts alike find themselves with “too much airplane” right now. We need to have the humility to:

  1. Admit that these are jobs we can’t do alone.
  2. Develop tools to help overcome the problem of “too much airplane.”

Actionable Items

“You want people to make sure to get the stupid stuff right. Yet you also want to leave room for craft and judgment and the ability to respond to unexpected difficulties that arise along the way”

Checklists must strike an important balance. They need to cover enough critical points in a process to minimize the “stupid stuff” that could happen, but they also need the flexibility and brevity to be useful in a real-world setting. Gwande explains the difference between a good checklist and a bad one:

“Bad checklists are vague and imprecise. They are too long; they are hard to use; they are impractical. They are made by desk jockeys with no awareness of the situations in which they are to be deployed. They treat the people using the tools as dumb and try to spell out every single step. They turn people’s brains off rather than turn them on.

Good checklists, on the other hand, are precise. They are efficient, to the point, and easy to use even in the most difficult situations. They do not try to spell out everything—a checklist cannot fly a plane. Instead, they provide reminders of only the most critical and important steps—the ones that even the highly skilled professionals using them could miss. Good checklists are, above all, practical.”

After developing a good, concise checklist comes the most critical step: implementation. As Gwande first developed his surgical checklist, he practiced using it with his surgical team to figure out what worked, what didn’t, and how they could improve it before using it in an actual surgical setting.

Once the checklist was tested, it needed to gain adoption with other surgeons and their teams. Gwande started introducing it to chief surgeons - the idea was that if a hospital’s leadership was able to embrace the change, others would follow. This is exactly what happened, and the incidence of surgical errors began to drop. But then something else began to happen:

“Spot surveys of random staff members coming out of surgery after the checklist was in effect did indeed report a significant increase in the level of communication. There was also a notable correlation between teamwork scores and results for patients—the greater the improvement in teamwork, the greater the drop in complications.”

The checklist process actually helped keep surgical teams more aware of what each person was doing by establishing a template for regular communication between everyone in the operating room. It’s an example of the hidden benefit of checklists:

“Just ticking boxes is not the ultimate goal here. Embracing a culture of teamwork and discipline is.”

Setting up checklists can not only improve processes, it can help build stronger teams. For educators and analysts alike, that’s a goal worthy of pursuit.


Tinkering

The most recent episode of Mad Dogs & Englishmen1 starts off with a discussion of buying and assembling and AR-15 from scratch. It’s an interesting process, but the conversation took an even more interesting turn as they focused on the decline of a tinkering ethos in our society.

The machines and devices we use are often too complex to repair, disassemble, or customize. If you open the hood of your car, you won’t be able to see or even access many of the most critical parts - you’ll most likely see a series of injection-molded plastic meant to not only protect but to discourage amateur tinkering. It is more likely that you’ll be able to tinker with your car by software (via OBD-II compatible devices like Automatic) than by mechanical manipulation.

Understanding how to tinker/hack software is an increasingly valuable skill, but we should also recognize and cultivate the ability to take apart, assemble, fix, and customize physical objects. The Maker Movement is a great example of keeping mechanical tinkering alive by encouraging kids and adults alike to get their hands dirty by blending both digital and physical craftsmanship.

As more of our lives become digitized, I hope that we continue to find ways to maintain some form of physical tinkering.

  1. Mad Dogs and Englishmen is a podcast featuring National Review writers Kevin Williamson and Charles C.W. Cooke 


Intellectuals are Freaks

Michael Lind on the weakness of intellectuals’ approach to inequality:

The views of intellectuals about social reform tend to be warped by professional and personal biases, as well. In the U.S. the default prescription for inequality and other social problems among professors, pundits, and policy wonks alike tends to be: More education! Successful intellectuals get where they are by being good at taking tests and by going to good schools. It is only natural for them to generalize from their own highly atypical life experiences and propose that society would be better off if everyone went to college — natural, but still stupid and lazy. Most of the jobs in advanced economies — a majority of them in the service sector — do not require higher education beyond a little vocational training. Notwithstanding automation, for the foreseeable future janitors will vastly outnumber professors, and if the wages of janitors are too low then other methods — unionization, the restriction of low-wage immigration, a higher minimum wage — make much more sense than enabling janitors to acquire BAs, much less MAs and Ph.Ds.

I’m a member of the very intellectual class Lind describes. I share his concern that our “intellectual class” is too isolated from the experiences of the majority of our fellow citizens and that this can lead to ideas/policies that would work great for Belmont but fail to address the needs of Fishtown.

But does that mean the “intellectual class” solution of “more education” is the wrong answer?

Every year, fewer jobs are available to those with only a high school degree. If we want to expand opportunity for working class Americans, it means ensuring a greater portion earn a BA - it also means that we need more people completing career/technical education and earning 2-year degrees.

Lind argues for short-term protectionist solutions to address low wages: more unionization, less immigration, and higher minimum wages. These may sound attractive, but they are ultimately stopgap measures that don’t address the systemic challenge in our economy, illustrated in the graphic below:

Why did wages for the Western working class stagnate while the Asian middle class grew just as much as the income of the most affluent? As Yuval Levin explains in The Fractured Republic:

“In the 1950s, the nation produced more than half of the world’s total manufactured goods—a mind-boggling, if obviously transitory, advantage.

The equality of the postwar era was a unique moment in American history, subsidized in large part by the devastating impact of WWII on competing economies. As other nations rebuilt and technology facilitated the rise of globalization, the wage premium for American labor withered away. Protectionist policies can only attempt to insulate against this trend - they don’t address the fundamental challenge that work historically performed by high school graduates in the US can now increasingly be performed more cheaply abroad or by robots.

The long-term solution is obvious: more education (both career and college oriented) to help more people access highly-skilled jobs with high wages, but we also need to consider shorter-term solutions to help make this transition less disruptive without resorting to protectionism. This will certainly involve a re-thinking of our social welfare system, either by re-imagining A Better Way to run our current system, or replacing it with something more radical, like a Universal Basic Income.

While I disagree with Lind’s proposed solution to inequality, I agree that more members of the “intellectual class” need to get outside of our bubbles to better understand the lives of working class Americans. Our nation is currently a fractured republic and coming apart - those of us in the “intellectual class” need to actively work to reverse the isolation we experience and cultivate more relationships and empathy for people of all backgrounds in our society.