TOPcast Episode 17: Researching “Questions of Consequence”

KELVIN		From the University of Central Florida’s Center for Distributed Learning, I’m Kelvin Thompson.	
 			
TOM		And I’m Tom Cavanagh.	
			
KELVIN		And you’re listening to TOPcast: the Teaching Online Podcast. Hey Tom.	
			
TOM		Hey Kelvin.	
			
KELVIN		What’s happening?	
			
TOM		I don’t know. I just found myself sitting in this recording studio. I don’t know how I got here.	
			
KELVIN		What are the odds?	
			
TOM		Yeah.	
			
KELVIN		Here I am, too. That’s bizarre.	
			
TOM		It is bizarre.	
			
KELVIN		And with a thermos, too.	
			
TOM		Wow! Well, at least we won’t go thirsty.	
			
KELVIN		That’s right. Or at least, (laughter) not fatigued anyway.	
			
TOM		No. (laughter) So, I can hear the pour.	
			
KELVIN		Yes, sir.	
			
TOM		I can see the pour.	
			
KELVIN		Yes, sir.	
			
TOM		What do we have in the thermos today?	
			
KELVIN		Well, I don’t want to be too philosophical, but you know, sometimes Tom…	
			
TOM		(laughter) Stop now.	
			
KELVIN		Yeah, I know. Sometimes it’s not about the coffee.	
			
TOM		No?	
			
KELVIN		No. Sometimes it’s about the people.	
			
TOM		Woah. That’s true.	
			
KELVIN		So today, it’s about our UCF colleague, Ian Turgeon.	
			
TOM		Ian Turgeon!	
			
KELVIN		Ian. Ian’s a TOPcast listener, and he was kind enough to go out of his way, putting in a lot of effort to ask his wife, Caroline, to pick up some locally roasted coffee beans from an artisan roaster-y/coffee shop down the street from where she works. So this is Trilogy Coffee in Deland, Florida, and this particular coffee that we’re drinking is from Rwanda. A Misozi Kopakaki.	
			
TOM		Say that again.	
			
KELVIN		I think it’s Misozi Kopakaki.	
			
TOM		Misozi Kopakaki.	
			
KELVIN		Yeah, that’s right. From Rwanda. I’ve had a couple Rwandan coffees. Pretty good. Scares me a little bit, but pretty good. We’ve talked before. Congo is pretty good. Rwanda’s pretty good. So, seems like a lot of drama in the region and you get good coffee. I don’t know. What do you think?	
			
TOM		It’s good.	
			
KELVIN		You kind of look like you’re like, “Woah!” Like slapped across the face there.	
			
TOM		Well, it was…It just had a little more bite than I expected, but that doesn’t mean it’s bad.	
			
KELVIN		And there you go.	
			
TOM		It’s good.	
			
KELVIN		That’s right. So our colleague Ian leads a team of developers here at UCF, as you know, and I would characterize him as an idea guy who gets his hands dirty. He actually—	
			
TOM		He’s an idea man.	
			
KELVIN		He’s an idea man, but he rolls up his sleeves. He’s a developer, and he gets things done. Give him a shout out here. He’s spearheaded what’s become kind of a movement here at UCF that we call Hack Day.	
			
TOM		(whispers) Hack Day.	
			
KELVIN		We’ll put some information in our show notes, so check out topcast.online.ucf.edu, and to summarize for our listeners—see if this is right—Hack Day is a kind of flash mob research and development activity. How’s that?	
			
TOM		Okay. Yeah.	
			
KELVIN		Kinda-ish? Right?	
			
TOM		Kinda-ish.	
			
KELVIN		You get people, and you make stuff quickly and all that. So, I thought Ian seemed like the perfect poster child for today’s topic, which I would characterize as the importance of ongoing research to support online and blended learning. Ian’s also a role model for anyone else who wants to donate high-quality craft coffee, because we’ll take it.	
			
TOM		We’ll take it. We got a lot of podcasts to do, and we’re thirsty.	
			
KELVIN		(laughter)	
			
TOM		No, that’s great. Ian is like the true player coach here at the Center for Distributed Learning. [He’s] one of our key developers and leaders on our application development team [and] has built some amazing stuff for us, so…Didn’t know he had this coffee appreciation. Obviously, he has great taste in podcasts.	
			
KELVIN		He doesn’t drink coffee.	
			
TOM		(laughter)	
			
KELVIN		But he heard this was a good place. I think Caroline drinks the coffee, and he said, “Caroline, would you get these guys some coffee?”	
			
TOM		Caroline, this one’s for you.	
			
KELVIN		That’s right. Not bad at all. Thank you, Trilogy. So, let me say: in an earlier episode, we summarized a few of the major research findings related to online learning, and we considered broadly the place of small-scale and institutional-scale and larger-scale research projects. So, look for that episode wherever you found this one, if you haven’t listened to it already. But we thought it important to zoom in for more detail with a professional researcher who contributes to our field. So, you, Tom.	
			
TOM		Yes, me.	
			
KELVIN		You interviewed Dr. Charles Graham—our friend and colleague who’s a professor and department chair of Instructional Psychology and Technology at Brigham-Young University—back during the 2015 Online Learning Consortium International Conference. So we had to get this out the door before it’s a year old.	
			
TOM		(laughter) Before the 2016 OLC Conference.	
			
KELVIN		That’s right. We really do give some thought to the sequence of these things and this was a good time for this one to come out. So, Charles is a fellow of the Online Learning Consortium. He’s been an influential scholar in the study of blended and online learning with many publications to his credit. He’s the real deal, right?	
			
TOM		Charles is the real deal, and just do a short literature search on his body of work and you will come to the same conclusion that we have that you could learn an awful lot from Charles Graham.	
			
KELVIN		Absolutely. So, we’re going to cut to the recording of that interview, and then we’ll be back to wrap things up, but we’ll pick this up, I think, right where you’ve asked Charles to introduce himself, give his title and stuff, and we’ll just pick it up with him doing that.	
			
TOM		Sounds good.	
			
		(music transition)	
			
CHARLES		Thanks, Tom. Well, right now, I am the department chair of the Instructional Psychology and Technology department. So, we are a graduate-only program at the university, and we train people/students to be designers, you know, to do instructional design.	
			
TOM		Because we could all use a little more personal training, I think. Right? Well, thank you for taking time to talk to us today. One of the things that I hope that we can explore a little bit is the subject of research. It has certainly come up recently that more and more, online learning professionals, people like us who advocate for online learning—maybe more like me than you because you wear a true, more traditional faculty hat—are getting challenged with, you know, “Well, why should I do this?” Or “How is this any ‘better’ than face-to-face instruction,” and one of the things I’ve really respected about you and the work that you’ve done is that you’ve taken a real serious approach to research in the discipline. I wonder if maybe we can start there. Why do you think it’s so important that we have really solid evidence-based corpus of literature around online and blended learning?	
			
CHARLES		That’s a great question. I think I would just start by saying that research is an important part of any mature enterprise. Businesses do research. They understand if their practices are effective. You know, hospitals/healthcare does research. All the major disciplines are involved in doing research. It’s a look at your own practice: to find out what’s working and what’s not working. So just from that aspect, I think it’s really important in online and distance education as a fairly new enterprise when you look at history of disciplines in the world, and so, research helps people to better understand what practices they should engage in and what practices are going to be effective. I think everybody can be involved and help in the research enterprise. They don’t have to be researchers like me that do it, you know, as a major part of their job. They can be people who are working for a university or working for a company or, you know, doing distance education, and any aspect of faculty distance education can be doing some kind of research.	
			
TOM		Well, I mean, I will say that as somebody who is wears a mostly administrative hat and sometimes gets challenged about, you know, “Why should I do this?” or “Why is this good as”…I rely a lot on the research that folks like you and our colleagues at UCF do, and I pull those things right out of my pocket and say, “Well, read this. Here is a real study with actual scholarly practices, you know, whether it’s an AB study or a control group or whatever. Things that they—You speak their language, so they can kind of understand, and in many cases, if somebody isn’t just completely recalcitrant, that helps to neutralize those objections, but one that actually has come up recently that I still struggle with a little bit is why do we think—and maybe this is not true—why do people outside of the world of online learning challenge online and blended learning as having to justify its quality in comparison to face-to-face? And there’s kind of a conventional wisdom within the online learning world that face-to-face isn’t scrutinized like this, but you know, is that true? Maybe it is. There’s an awful lot of educational research over the years.	
			
CHARLES		Yeah, I’ve wondered that myself, too, because it seems like the standard for online courses, the bar is very, very high.	
			
TOM		Right.	
			
CHARLES		When you know that there’s a good number of face-to-face courses that are not that great of quality.	
			
TOM		Yeah, I’ve sat in a few.	
			
CHARLES		But I think that’s actually—That works to our advantage in a way. I mean, it lets us actually start the conversation about evidence and research in teaching and learning. We heard—Earlier today, I heard someone say that faculty actually—as they start to do/teach online—it’s the first time that they maybe have been exposed to, you know, quality rubrics and quality performance indicators around teaching and learning, so…	
			
TOM		Well, at schools like BYU or at UCF where they hire primarily research faculty, that is something you hear a lot. I’ve been trained in my discipline and how to do research and that’s what I’m measured on and that’s what my tenure decisions will be based on, but nobody’s ever taught them pedagogy and how to walk into a classroom and teach. What we hear certainly at UCF is that for many faculty, going through online development and how to build and deliver online courses is the first real pedagogical training that they’ve had, and it has not only, of course, helped them teach online but has informed their classroom practice.	
			
	CHARLES		Yeah. Think of yourself like…Tom, you have a wealth of experience in, you know, being at UCF, but imagine back to when you were just first starting. Where do you draw on? What knowledge base do you draw on if faculty come to you and say, “Well, what is most effective in teaching in this context?” or “I want to teach a biology class in the online format. What do I do that’s gonna that’s most likely to lead to good outcomes for my students?” If you don’t have your own past experience with that, you’re going to go and try and draw on research, and that’s part of what research is for. It’s to create this knowledge base that allows people who are new into the field and are merging in the field to be able to make better decisions at the outset than they would if they were trying to have to start from scratch and figure everything out themselves.	
			
TOM		Well, in addition to your role as a researcher yourself, you’ve also served as an editor of editing collections of other people’s research, and I wonder as part of that if you’ve noticed any kind of trends or interesting developments that this kind of higher view of research has given you as opposed to just the stuff that you’re working on as part of your research agenda.	
			
CHARLES		Well, I have looked at lots of research that’s done at lots of different levels, institutional levels as well as done by individual faculty in the classroom or, you know, action-based research, and I think one of the trends or one of the important things that I want to mention about research in distance education is that it’s really important for us to think about the quality of the research, and there’s value in like, you know. There’s lots of case studies that are done by lots of people in the scholarship of teaching and learning. There’s a lot of research that’s actually being done by institutions, and if they can—If people doing this research, either in their individual classrooms or at the institutional level, can learn a little bit about the things that will help make the research rigorous, maybe a little more generalizable, then people outside the institution, outside their classroom can also benefit from the things that they’re learning.	
			
TOM		Mhmm, and something Kelvin and I have talked about in previous podcasts is the—In the past, and we still see it today, where there tends to sometimes be a lot of publicity around somebody’s just AB studies of their classes. You know, “I studied my face-to-face section. I’ve studied my online section. I’ve drawn these conclusions. Therefore, online learning is bad or good or whatever,” and I’m waiting for the day when we really get beyond those micro studies that aren’t helping us move the needle as much as it maybe could be. This is maybe my own view. Do you think that there are opportunities to kind of raise the level of research that we’re doing that are more related to institutional impact or some of the big problems that higher education has as opposed to, “Did my, you know, twenty students like onlinie learning or not?”	
			
CHARLES		Absolutely. You have a great point there, and I think organizations like yours—like the Online Learning Consortium and others—can be instrumental in developing a set of questions that they want answers that are really important so that researchers people in the class and others can answer questions of consequence rather than just answering questions that are less meaningful, and I think there was a point at which asking the research questions about, you know, online learning more effective than face-to-face learning, there’s a role that those have played in the past, because people needed to understand it was mostly for stakeholders even though you’ll often hear them say, “Well, you know, does this class better than this class…?” I mean, even if you’re saying is face-to-face learning? Is this face-to-face learning better than this face-to-face learning, you’re going to get lots of different kinds of answers to that, so there’s a time in the past when that was important. I think we’re beyond that. That the data is pretty overwhelmingly that online learning is not any better or worse than face-to-face learning in general.	
			
TOM		Yeah.	
			
CHARLES		And that it really matters the context, the things that you’re doing in the online space and the things that you’re doing in the face-to-face space so that’s the direction that research needs to move. What are the pedagogies that help students learn?	
			
TOM		Yeah. Is it effective or not, regardless of modality? I wonder if there’s anything that you’re working on as part of your research agenda that’s really exciting you these days. I love that term or phrase you use: the questions of consequence. I wonder if you have any questions of consequence that you’re working on right now.	
			
CHARLES		Yeah. Well, one of the things that we’re trying to do with my research team is really understand the role that engagement plays—online engagement. For example, let me just paint this picture for you. When a teacher walks into a face-to-face classroom, a teacher is collecting formative data all the time on their students. They see the students at the front row, you know, happy and smiling. They see a couple students back that are, you know, starting to doze off, and the teacher can adjust their instruction immediately based on the data that they’re getting from the students that the formative data that they’re getting from the students. You know, the teacher might draw closer to the student that’s dozing off, pat the student on the shoulder or maybe direct a question towards that student so they’re changing their instructional plan, their pedagogy based on the data that they’re getting. Now, if we look at what happens to that instructor when they move into the online space, they have no clue what’s happening when the student’s actually going through the instruction for most of the time. I mean, they have a score at the end, and if they’re doing something in the online discussion space, maybe they have some idea of how what’s happening. So, I think a really important area of research is to understand how students are engaging with online content, how students are engaging with each other, and how students are engaging with instructors when they’re in that online space. And if there’s really efficient ways for us to get at that, right? In the past, what’s happened with engagement research is they’ve given surveys to students, like the NSSE is a big engagement research. You get a survey that’s happening at the end of the course.	
			
TOM		Those are summative assessments.	
			
CHARLES		Yeah. They’re summative. They don’t help you make instructional decisions at the time. They may help you change a course in the future, but we’re really interested in trying to understand if we can if we can measure engagement in real time and if we can learn about student engagement what’s happening with student engagement that provides us with something that that’s actually actionable that can change what teachers do in the middle of the class.	
			
TOM		That is a question of consequence.	
			
CHARLES		(laughter) And I hope a lot of other people are working on this, too, because it can’t just be done by us. It has to be done you know there have to be a lot of people that are probing these questions and trying to be learned from each other.	
			
TOM		Well, I think there are a lot of facts to that, because you can come at that questions from a lot of directions. For example, as you were talking, I couldn’t help but think, just because I’m living in this world right now, of adaptive learning, and adaptive learning is the platform we’re using, for example, is continually assessing student behavior, performance, and you can have almost a real-time report of just what’s happening with that student. Is it working? Is it not working? Are they engaged? Because maybe they just aren’t you know doing as much work as you think they should. Time on task, for example, is just one of many measures.	
			
CHARLES		Yeah. You’re absolutely right. I think there’s a lot of promise with adaptive learning. One of the challenges with adaptive learning is that the systems are usually pretty expensive and they’re also very content specific. You know, so we could create a really great get really great courseware for an algebra class or a psychology class but it’s harder to get really adaptive learning software for the classes that are that maybe have less numbers than that or change very frequently because the knowledge base that’s being taught changes. I think that’s a great direction a great an important thing to do, but I also think it’s important to look at how we can measure engagement using the learning management system and the data that’s coming out of the learning management system and in classes that are instructor-driven. You know. That aren’t predominately learner-content interaction but also have a high level of learner-peer interaction.	
			
TOM		Yeah. We do this thing in our department called Hack Day, and we let anybody work on any project they want for 24 hours, and we do it a couple times a year. A lot of programmer staff love it. Although it’s open to anybody. We’ve had all kinds of different projects. But I bring this up because the last time we did it, one of our programmers who takes—He’s a student, and he developed a little mobile app that as a professor is lecturing, students can sit there and sit and click “I like”, “I like”, “I don’t like”, “I don’t like”. And then the professor can see real-time, you know, a little indicator that goes up and down. 	
			
CHARLES		(laughter) That is really cool.	
			
TOM		Like “Oh, I’m boring them right now.” I mean, it was like every second feedback. That’s probably a lot for faculty member to kind of absorb at once, but that was a student-generated indicator of engagement to tell the faculty member, “Hey, step it up.” Or “Hey, this is good.” Or “I have a question” or something. I thought it was really interesting, so I totally agree. I think that there is a pretty broad waterfront of questions around this one of engagement, and I know it’s one that Gates Foundation and NGLC has been really, really interested in. So yeah, I encourage you to keep pursuing that line of inquiry.	
			
CHARLES		And the same! I know you guys are trying to do great things at UCF with better understanding how your students are engaging, because ultimately, one of the things that the research tells us is that student engagement is correlated with all of these other outcomes that we really care about: the student learning, the student satisfaction, the self-regulation. You know, there’s just a lot of positive things that student engagement is a facilitator for.	
			
TOM		And I think maybe that’s a great place to stop. So, Charles, on behalf of Kelvin, I want to really thank you for joining us on today’s TOPcast.	
			
CHARLES		Thank you.	
			
		(music transition)	
			
KELVIN		I thought that was a good, thoughtful, substantive interview, Tom.	
			
TOM		Thank you. You can thank Charles for that.	
			
KELVIN		Thank you, Charles. Yeah, I love how a professional researcher like Charles still encourages all of us—whatever our role in online education—to be scholarly practitioners, at least.	
			
TOM		Yeah, absolutely. I mean, you can tell Charles has put an awful lot of thought into these questions. Questions of consequence.	
			
KELVIN		Yeah, I love that.	
			
TOM		Yeah, and I think that it is people like Charles and other, you know, really respected researchers that are helping to address any sort of skepticism people might have about online learning, because we are really self-critically examining this as a practice, as a discipline, and how can we continually improve? How can we understand what works and what doesn’t work? And it’s objective. And so, you know, you can’t necessarily say, “Well, online learning isn’t as good because there hasn’t been 200 years of research on it like in face-to-face or whatever.” Well, you know, that’s not quite so true anymore. There’s a pretty broad body of literature that people like Charles are producing.	
			
KELVIN		I love that. You know, “any mature enterprise.” I think that’s good, and so, hopefully some of us are mature. (laughter)	
			
TOM		(laughter)	
			
KELVIN		Hopefully the whole thing’s mature. I don’t know.	
			
TOM		Some more mature than others. Charles is definitely more mature than we are.	
			
KELVIN		That’s true. That’s true. And you know, I don’t think she would mind, but we should probably give a shout out here to Katie Linder’s podcast, Research in Action. Katie serves up a ton of great information on research in general and research in online education specifically, so wherever fine podcasts are downloaded, you can look at that.	
			
TOM		Yeah. Highly recommend that podcast, and Katie’s particular focus is on the research side of things, and if you don’t know Katie Linder from Oregon State, definitely check out her podcast Research in Action.	
			
KELVIN		Yeah. So, take us home, Tom. What was the bottom line from this episode?	
			
TOM		Well, to kind of, you know, use that expression again, if we think online learning is to become a mature enterprise—	
			
KELVIN		I may have no more place here. (laughter)	
			
TOM		(laughter) —our practice must continually be informed by research conducted by professional scholars and scholarly professionals.	
			
KELVIN		Ooh, I like that.	
			
TOM		Yes.	
			
KELVIN		Well, that’s good. I think that’s a perfect place to leave it. So, let us know what you think. Always a good thing. There’s a lot of interaction venues you can find at topcast.online.ucf.edu, and if you like what you’ve been listening to, please go to iTunes and give us a star rating and leave a review behind there. That helps other people find this show that is for people just like you.	
			
TOM		Just like you.	
			
KELVIN		So, until next time, for TOPcast, I’m Kelvin.	
			
TOM		And I’m Tom.	
			
KELVIN		See ya!