TOPcast Episode 14: The Problem with Quality: Consensus vs. Connoisseurship

TOM		
From the 2016 OLC Innovate Conference in New Orleans, Louisiana, I’m Tom Cavanagh.	
			
KELVIN		And I’m Kelvin Thompson.	
			
TOM		And you are listening to TOPcast. So we probably sound a little different. Don’t we, Kelvin?	
			
KELVIN		Yeah. We sound really—I don’t know—hipper and more mellow than normal.	
			
TOM		Yeah.	
			
KELVIN		Why is that?	
			
TOM		We are currently sitting out on the sidewalk in the Garden District of New Orleans during the OLC Innovate Conference at a coffee shop that you [Kelvin] have discovered.	
			
KELVIN		Yeah. The Mojo Coffee House. Locally roasted, batch-craft-roasted coffees. We walked seventeen miles to get here.	
			
TOM		Felt like it in the New Orleans humidity.	
			
KELVIN		That’s right. What do you think of this place?	
			
TOM		It was good. Yeah, we should probably snap a picture before we leave.	
			
KELVIN		That’s right. Oh, I meant to do that. I did snap one of the back of [Tom’s] head in front of the sign earlier. It was lovely.	
			
TOM		Nothing better than the back of my head. It’s my best side. Get my good side.	
			
KELVIN		So we always talk about our coffee. I’m having one that was recommended by the barista from their stock here. It’s a Kenya Peaberry, and it was done on a pour-over method. It was quite, quite tasty, I thought. But you [Tom] opted for something different.	
			
TOM		I opted for something different, indeed. So we are in New Orleans, and I thought, “Well, when in Rome.”	
			
KELVIN		I thought we were in New Orleans!	
			
TOM		
Yeah, I decided to get a café au lait, and it was quite good.	
			
KELVIN		Is that what Nero drank?	
			
TOM		Yeah, that’s what Nero—or Constantine—drank.	
			
KELVIN		Something in Rome.	
			
TOM		And so, it was good. And so, thematically, these tie to our topic of the day. Don’t they?	
KELVIN		Do tell, Tom. How do they tie to the topic of the day?	
			
TOM		So, if those of you who have been regular listeners recall, we talked a few months ago about the iron triangle of quality, cost, and access. We said we would occasionally revisit those broad categories as time went on. Today, welcome to the TOPcast episode: Quality. So you [Kelvin] got a particular type of coffee that was prepared in a particular way to insure quality, I would assume. I got a café au lait—which you might say maybe isn’t necessarily quality, but I thought was quite good. I was going with a definition of quality that is more along the lines of “Does the product or service meet expectations?” And so I’m in New Orleans, I got the expected drink, and it definitely met expectations. To me, I had a quality experience.	
			
KELVIN		That’s good. I like that. We have touched on quality a little bit as we laid a foundation for the iron triangle. I think we’ve talked in previous episodes about the fact that there is a tension between those three points of the triangle: the affordability, the access, and the quality. We’ve used coffee before. Off-the-shelf coffee is maybe not perceived generally as good as some of the boutique, artisanal-roasting kind of coffee, so I think there is inherently this sense that you’re coming into this devoted place where they do something on site. Well of course! Of course they’re going to do it better. It must be quality. But how do you really determine quality? Is there some universal standard?	
			
TOM		Well, you know—going back to our first podcast—it is one of those boundary objects that you say the word quality and two different people might have something different in their heads about what quality is. Maybe you [Kelvin] love country music and consider that quality music. Maybe I love experimental free-form jazz, and I think that’s quality. But maybe we each hate the other kind and don’t think it’s quality music. But they might be groundbreaking quality music. It’s just not particularly meeting our expectations. Maybe that’s the definition of quality. But then I do think that there probably are some objective standards of quality that somehow live alongside these individual expectations. So, in our field, something like Quality Matters or the Quality Scorecard from OLC—these might be considered objective standards of quality.	
			
KELVIN		No, that’s good. Also SUNY put out the OSCQR. I love the name OSCQR. The Online Course Quality Rubric?	
			
TOM		Alex Pickett would be proud of you.	
			
KELVIN		Well, I butchered some acronym last time. Center for Online Teaching Excellence in a prior episode, but she [Alex] forgave me and loved me anyway so that’s good. Yeah, I think that’s right. In our field, for sure, lots of rubrics. In the state of Florida, we’re going through some iron triangle exercises for a number of reasons. We’ve been having state-wide discussions about quality, and so there’s been some discussion of adopting some sort of a standard that could be of use to all our state universities and state colleges. It’s interesting to get people to agree on a finite list of standards for “high quality” online education.	
TOM		That is a good question. It’s something that we as a state are working our way through. And now, maybe I’ll circle back to this idea of expectations. If you are paying for a premium, private education—online or otherwise—your expectations for what you’re going to receive might be a lot higher. And if they’re not met, you might perceive that as not being quality for the money. It’s more of a value question, I guess. But, if you are at a community college—which maybe isn’t a fair thing to say because some of the great work is happening at community colleges—for some people, because you’re paying less, you’re willing maybe to accept a different standard of quality.	
			
KELVIN		That’s interesting.	
			
TOM		I don’t know. I just sort of posit that as a question. I’m not sure how I feel about it because, like I said, I know plenty of people in community colleges that do fantastic work. Well, a story that just came out this week, some students from George Washington University have sued the university because they felt their Master’s degrees weren’t quality. They didn’t get the same education that the folks in the classroom got and that was what they were expecting. And again, this just broke in the news, so we don’t know what will happen with this case. But I do think that’s a bit of a clarion call for all of us who work in online education to take a look at our programs and make sure that we are delivering on the promise of quality that we make to students. I’m not saying GW didn’t. I honestly don’t know the details of that case, but obviously some students feel like they didn’t get what they were paying for.	
			
KELVIN		
And I guess there’s countervailing sets of expectations, right? It’s like the seemingly age-old debate around end-of-term evaluation. There’s a whole body of literature around that. Some people find value in students’ feedback on the course and the instructor’s teaching, and others roll their eyes saying, “Well, how is that student qualified to decide whether this was a good experience or not? They don’t know what they don’t know.”	
			
TOM		That’s right.	
			
KELVIN		So, countervailing expectations. Whenever I’ve taught online, I’ve had one student every semester who in my weekly feedback forms say, “Well, it’s an online course. I don’t expect it to be any more connected or any more interactive than it is.” There’s sort of this a priori assumption that it’s going to be less somehow so it’s just dismissed. 	
			
TOM		I hope that’s changed.	
			
KELVIN		I hope so. But it’s not a lot. It’s usually just one student. I don’t get the same student every semester, so I think it’s a funny thing. But it’s expectations of the student versus expectations of the faculty versus expectations of other authorities (Board of Trustees or the public or state government or whoever the stakeholders are for the institution). I think all of those expectations should be held in balance somehow.	
			
TOM		Yeah. And kind of back to the iron triangle: it’s really hard to impact quality at a large scale without affecting cost or access. We could make it high quality by just making sure only a certain kind of student has access to it.	
			
KELVIN		Really highly motivated, engaged, and bright students.	
			
TOM		4.0 students who can afford a high tuition and come from the best socioeconomic backgrounds. But that’s not why we’re here. That is the challenge of the iron triangle. How do you increase quality and not impact cost or access? Or at least even reduce the other two?	
			
KELVIN		So to get a little bit practical for our online education field, here are some things I find perennially surface in these kinds of conversations about quality: interaction between and among students and faculty. That’s a thing that’s fostered and carried out. What are other things?	
			
TOM		Well you could almost run down the [Quality Matters] rubric and kind of say, “Yeah, those are all things that kind of indicate quality in a course.” There are other rubrics that probably do a good job or just as good a job.	
			
KELVIN		
CSU Chico rubric comes up a lot.	
			
TOM		That’s one that I’ve used before. I thought the old Blackboard Greenhouse Award rubric was pretty good. That or some combination of them or something that maybe includes something concept-specific for a particular institution.	
			
KELVIN		Can I go on a rant for just a second? I promise it won’t be long.	
			
TOM		(laughter) Why not?	
			
KELVIN		I think you’ve heard me rant on this before. It’s just a personal pet peeve. I’ve spent some time in this area of rubrics and evaluative standards. Almost universally, these kind of efforts tend to be about course design, about designing for something to happen rather than the actual carrying out of the thing. So the actual teaching part–the lived experience–is often ignored or sidestepped. I think that’s a short-sighted thing. I know as we’ve been talking in our efforts in Florida, we’ve been trying to keep those things in balance. We try to deal with the actual teaching part and not just the design part.	
			
TOM		Yes. I’ve had moments where when I’ve been teaching an online class, I’ve had students say things like, “Wow you sure do respond quick! That’s great you got right back to me!” The implication being not all faculty—I’m not saying what institution this is—did that. That is a delivery issue, not a design issue.	
			
KELVIN		That’s right. So those two things—delivery and design, to use those terms—are pretty closely related, but they’re not exactly the same thing. You could probably have a course that’s extremely well-designed but poorly executed.	
TOM		Right.	
			
KELVIN		You could have one that is a little lacking in the design part but an exceptional instructor could probably make up some difference there.	
			
TOM		Yeah I think a really good instructor can compensate just by being hyper-engaged.	
			
KELVIN		That’s a thing. You hope for good efforts in both categories.	
			
TOM		You would hope.	
			
KELVIN		But this whole thing is kind of challenging. We all are looking to define it. I think about our colleague Chuck Dziuban from UCF who likes to quote Robert Pirsig on the topic of quality from Zen and the Art of Motorcycle Maintenance. I 
don’t know the quote, but it comes up as a theme in the book, which I read years and years ago. I think the gist of it at essence is everyone wants to think, “Oh yeah. Quality’s a thing.” But what is quality? You try to define it, and you wake up from the dream or the soap bubble bursts or something. It’s gone suddenly. You can’t pin it down. It’s elusive.	
			
TOM		So let me give a non-online learning anecdote. We’ve all seen movies that are a huge hit that you say to yourself, “Why is that a hit? That movie was inane or stupid. Man, a whole lot of people seem to love it so they think it’s quality.” We had a fundraiser for part of our annual giving campaign at UCF where we had a chili competition.	
			
KELVIN		I remember this.	
			
TOM		Yes. You probably know where I’m going. It was a competition, and everybody got to vote on the best chili that was made by various members of the staff. I’ll call him out by name. Dr. Bill Phillips, one of our beloved colleagues—one of our lead Instructional Designers—made chili. For the record, I ended up voting for it, so I liked it. Bill did not win. Much to his chagrin, he did not win. And he went on something of a tirade—a good-natured tirade—basically about the philistines who work at the Center for Distributed Learning.	
			
KELVIN		And in fairness, Bill is a trained evaluator of various cuisines.	
			
TOM		He’s a good cook.	
			
KELVIN		He does all kinds of barbeque judging. His palate is educated.	
			
TOM		His point was that the ignorant palates of his coworkers were not worthy to judge his chili. His chili was far superior regardless of the outcome of the voting. So, just because he didn’t win, didn’t mean he didn’t have a high quality chili. Because the people voted, they said [the other chili] was quality. Regardless of Bill’s objective standards, they said the chili that we voted for was high quality. That’s kind of like the movie. There’s enough people that said, “We like this movie,” and spend money on it, but it baffles me. I think there’s a little bit of that in online learning. Some people love a particular kind of course or a particular kind of interaction. Some people like the immediacy of a live, synchronous kind of session. Some people don’t; they need the flexibility of working on their own time. Some people like lecture capture because they want to feel like they’re in the back of the classroom. I think there’s a little bit of something for everyone. They might all define those as a quality metric of sort. I’m just sort of riffing here.

KELVIN		It’s interesting. I question, what is the relationship between objectivity and subjectivity? What is the relationship between consensus and some sort of finite standard? Is it quality because a bunch of people say it’s quality regardless of some sort of more objective standard or not?	
			
TOM		It’s a good question. So I’m looking at the clock, and your [Kelvin’s] coffee is running down. I wonder if maybe we can put a ribbon on this.	
			
KELVIN		Yeah, we may have to have a second part to this quality discussion. It’s a messy topic.	
			
TOM		We’re not done talking about quality, I hope.	
			
KELVIN		I don’t know that we’ve gotten to any kind of a solution more than just recognizing that we’ve got an issue.	
			
TOM		Well I think that then quality will remain one of those really tough arms of the iron triangle to bend. I think that there will remain this tension between an objective standard of quality and individual expectations of quality. Reconciling those two together will be something that we continue to talk about going forward.	
			
KELVIN		Yes. And probably the whole of a quality perception versus some sort of an effort at individual standards or markers of quality. That tension is probably something that has to be wrestled with as well because you’re going to have to have something that can measure or check off or whatever when we’re dealing with online education. You can’t just be dealing with broad perceptions.	
			
TOM		No. And because of accreditation and other reasons, the importance of an objective standard will become more crucial.	
			
KELVIN		Here’s a thing, right? “Ensure your own quality or it will be ensured for you.”	
			
TOM		Yes, that’s right. “Do you want to define your own standards of quality or have someone impose them on you?” Maybe that’s a good place to end it.	
			
KELVIN		Probably is a good place to end it.	
			
TOM		All right. So from Magazine Street in the Crescent City here in New Orleans, I’m Tom.	
KELVIN		And I’m Kelvin.	
			
TOM		See ya!