Protocol for a systematic review: 21st century adaptive teaching and individualized learning operationalized as specific blends of student-centered instructional events: a systematic review and meta-analysis

Conceptual understanding of individualized learning and adaptive teaching varies broadly, encompassing a multitude of instructional strategies, approaches, and activities. It stretches from accounts as narrow and specific as scaffolding adaptive feedback in computer-based instruction (e.g., Atkinson, Renkl & Merrill, 2003) to more general conceptions, such as cooperative and collaborative learning (e.g., Johnson & Johnson, 2002). It also includes educational concepts derived from elements of constructivism, such as discovery learning, inquiry-based learning, experiential learning, problem-based learning and other forms of student-centered education. These instructional forms, which have been broadly described from the 1960s (e.g., Summerhill), were recently criticized by Kirshner, Sweller & Clark (2006) with a response published by Tobias & Duffy (2009). The learning sciences have further contributed to the distinction between social constructivism and individual constructivism (i.e., instructional system designs) providing a theoretical grounding for teacher vs. learner-based strategies (Kolodner, 2004). Current and developing applications, informed by pedagogical principles espoused by case-based learning (e.g., Kolodner et al., 2008), exemplify the transformation of learning environments which apply Bruner's concept of not just discovery for the student, but co-discovery on the part of the teacher. Examples of various attempts to make teaching and learning more adaptive can be found in both the early and current research literature. They include, though not limited to, mastery learning (e.g., Bloom, 1968), Personalized Systems of Instruction or PSI (e.g., Keller, 1968; Gifford & Vicks, 1982; Davies, 1981), assorted forms of peer instruction (e.g., Mazur, 1997), various reciprocal reading/writing activities (e.g., Huang & Yang, 2015; MacArthur, Schwartz & Graham, 1991), adaptive hypermedia (Brusilovsky, 2001), accommodation for individual learning styles (e.g., Özyurt & Özyurt, 2015) and more recent Intelligent Tutoring Systems or ITS (e.g., Huang & Shiu, 2012; VanLehn, 2011). To some extent, findings of primary research on these and related instructional practices have been summarized in two rather sparse collections of meta-analyses separated in time by almost three decades. In the late 1970s and early 1980s, several relevant meta-analyses were published. First, Lysakowski and Walberg (1982), Gusky and Gates (1986), Slavin (1986) and Kulik, Kulik and Bangert-Drowns (1990) each performed successive meta-analyses (Slavin's was a best evidence synthesis) on the efficacy of mastery learning. The studies produced equivocal findings. Also, Kulik, Kulik & Cohen (1979) reviewed 75 individual comparative studies of Keller's PSI (a spin-off of mastery learning) college teaching method. In comparison to conventional instruction the PSI was demonstrated to have a positive effect on student achievement and course perception (mean effect size of nearly 0.70 for both). Aiello and Wolfle (1980) summarized research on individualized instruction in science compared with traditional lectures and found that individualized instruction was more effective. Horak's (1981) study of self-paced modular instruction of elementary and secondary school math produced a wide variety of both positive and negative effect sizes. Bangert and Kulik (1982) looked at the effectiveness of the Individualized Systems of Instruction (ISI) in secondary school students. They broadened the list of outcomes to account not only for student achievement (e.g., final exam), but also critical thinking, attitudes toward subject matter, and student self-concept. For all outcome types the findings were unsettled. For example, for the achievement data only eight out of 49 studies demonstrated statistically significant results in favour of ISI (four studies favoured more conventional teaching methods and the rest were inconclusive). Finally in 1984, Kulik attempted a wider research synthesis (encompassing over 500 individual studies) of effectiveness of programmed instruction and ISI, paying special attention to the moderator variables of study dates and grade levels. Among the most promising findings, the author indicated that more recent studies showed higher effects than the earlier and that college-level students benefited significantly from using ISI compared with elementary and secondary school students. In summary, these meta-analyses produced inconclusive results. Moreover, they are rather outdated – practically none of the above-mentioned instructional methods exists now in its original form (e.g., Eyre, 2008 was able to identify fewer then 50 studies of PSI for the period between 1990 and 2006 in the PsycInfo database). All this suggests the need for a more refined (both methodologically and substantively) update of systematic reviews in the field, especially taking into account how much the methodology of meta-analysis itself has evolved since then. Several meta-analyses addressed the topic of individualized instruction, though in very specific narrowly focused forms. Cole (2014) examined the effectiveness of cooperative, collaborative, and peer tutoring for English language learners. A low-to-moderate average effect size of = 0.49 was found in favour of peer tutoring over individualized or teacher-centred comparison instructional conditions. The effect size tended to be relatively small in middle school students, but higher at elementary and high school levels. More in line with the already mentioned earlier meta-analyses of various forms of computer-assisted instruction, Ma, Adesope, Nesbit and Liu (2014) meta-analysed studies of Intelligent Tutoring Systems (ITS) in a variety of subject matters, from reading and math to law and medical education. The list of moderator variables included the type of both experimental and comparison treatments, as well as outcome type, student academic level, discipline studies, etc. The highest achievement effects of using ITS were found in comparison with non-ITS computer-based instruction ( = 0.57) and teacher-centred, large-group instruction ( = 0.42), whereas in comparison with human tutoring it was even negative ( = -0.11), though not statistically significant. ITS-based practices were similarly effective when used either alone or in combination with various forms of teacher-led instruction in many subject domains. In summary, research evidence concerning the effects of adaptive teaching and individualized learning remain relatively inconclusive, while there is obviously a need for better understanding how K-12 formal education may be more successful in addressing students’ personal needs and interests, accounting for their diverse abilities with the main goal of advancing their learning. The proposed systematic review will not only rely on the most rigorous and comprehensive methodology of meta-analysis, but also should be conceptually sound (i.e., thoroughly exploring educational practices to find consistent links among a multitude of individual pedagogical approaches) and timely (i.e., account for the most recent developments in education). The main research question of the proposed meta-analysis is: Can more Student-Centred (SC) (i.e., more adaptive and individualized) approaches to K-12 instruction be distinguished from more Teacher-Centred (TC) approaches in terms of their effect on student achievements and what substantive and demographic factors moderate these effects? For better understanding and more successful practical application, educational practices subsumed under this generic pedagogical idea of adaptive teaching and individualized learning deserve a valid conceptual working model, both inclusive enough to account for various forms of personalized/individualized instruction and sufficiently sensitive to fluctuations due not only to the influence of numerous moderator variables, but also to nuanced qualities of particular instructional approaches themselves. SC instructional strategies could, in our view, serve such an overarching conceptual framework with adequate explanatory power, but only if operationalized properly to avoid an oversimplified dichotomy of inductive vs. deductive education (constructivism vs. direct instruction). Indeed, we are less interested in deciding between these two extremes and more interested in understanding the circumstances or combination of circumstances that optimize teaching and learning. Gresalfi and Lester (2009) for mathematics teaching, and Klahr (2009) for science teaching argue that the goal of instruction should be to achieve curricular and process objectives by choosing the most appropriate method based on student age, ability, prior knowledge, level of content, etc. In this regard, we would like to avoid the conceptual error of falsely dichotomizing pedagogical environments as either TC or SC since neither instructional practice likely exists in its pure form. As Gersten et al. (2008) observed in their systematic review of mathematics teaching practices: “[We] found no examples of studies in which students were teaching themselves or each other without any teacher guidance; nor did the Task Group find studies in which teachers conveyed … content directly to students without any attention to their understanding or response. The fact that these terms, in practice, are neither clearly nor uniformly defined, nor are they true opposites, complicates the challenge of providing a review and synthesis of the literature …” (p. 12). Since SC pedagogical practices try to emphasize guidance over direct instruction, the question becomes how much and what kind of guidance is offered to students and who takes responsibility for the design and implementation of various components of learning experience to make them truly adaptive/individualized and, hence, more effective. To define the key quality of instruction as “adaptive” and “individualized,” for the purposes of the proposed systematic review, we suggest deconstructing teaching and learning according to the events associated with them (e.g., setting objectives, implementing instructional methods, assessing learning). Accordingly, a more SC (more adaptive) classroom is one in which students play a more central role in the conduct of the instructional events. If these events can be isolated in reports of primary classroom research, they can be rated individually on a TC to SC continuum. Each event could then be: 1) examined separately to determine their individual strengths; 2) examined in clusters as combinations of events; or 3) collapsed into a multi-dimensional composite that would yield a “greater-than to lesser-than” distinction between two different instructional settings. This approach avoids problems associated with either subjectively defining instructional conditions as SC vs. TC or vaguely labeling them, such as PSI, mastery learning, etc. It also has the advantage of allowing us to examine instructional events in isolation and in various combinations in the search for optimal instructional practices. Most of the significant effects from the meta-analyses described in the first section of this protocol on the topic cluster around 0.40SD, but the data also reflect a wide range of effects, depending on the whole spectrum of moderator variables. In other words, the picture painted by these meta-analyses remains in large part as inconclusive as it was several decades ago in the 1980s. Of special concern to us is the fact that both earlier and recent meta-analyses are rather limited in scope and focus of interest, addressing very specific instructional practices and technological tools. There were no serious attempts to find and conceptualize pedagogical commonalities among the interventions in question that would allow treating them within the same class of phenomena broadly depicted as individualized learning and adaptive teaching. Thus, the need for a review that would be broad in scope, summarize research evidence up to date, and have a conceptually sound foundation is pressing. The main objective of the proposed review is to summarize research data on the effectiveness (in terms of learning achievement outcomes) of adaptive and individualized instructional interventions operationally defined here as more SC pedagogical approaches. The overall weighted average effect size will be an indication of that. Additionally, and no less important, the review aims to better understand under what circumstances (e.g., with what populations of learners, for what subject matters) the effects of adaptive and individualized instruction rich their highest potential, and what conditions may depress them. To explore the latter, a set of substantive and demographic study features will be coded and subjected to moderator variable analyses. The review outcomes will inform education practitioners and research community of the best instructional practices, preconditions for their successful implementation and potential pitfalls, as well as of directions for further empirical research in the area. The review will include studies that are experimental (i.e., RCT) or high-quality quasi-experimental (i.e., statistically verified group equivalence or adjustment) in design that address adequate to the research question group comparisons, contain legitimate measures of academic achievement (i.e. teacher-made, standardized), and report sufficient statistical information for effect size extraction. Students in K-12 formal educational settings (approximate age 5-18), i.e., eventually leading to a certificate, diploma, degree, or promotion to a higher level. Educational interventions may take place either in the classroom (F2F), via distance education (DE), or as a blended (various combinations of F2F and DE) intervention. The highest total rating across these dimensions will determine an Intervention condition to be compared on achievement outcomes to a Comparison condition, lowest in total rating That is differential total across all dimensions, as some of them may be higher in ratings for one group and some – for another. At the same time, keeping individual positive and negative differentials will allow identifying consistent clusters of instructional events that are more/less likely to work to the advantage of students’ learning. All types of objective measures of academic achievements are to be considered. Their psychometric features (e.g., standardized, non-standardized teacher/researcher-made assessment tools) and type of representativeness (e.g., cumulative final examinations or averages of several performance tasks covering various components of the course/unit content) will be documented and used in subsequent moderator variable analyses. Self-assessments are to be excluded, as well as attitudinal and behavioural measures. Data of their prevalence in the reviewed primary literature will be collected to inform further reviews in the area with a potential focus on those types of outcomes. To maximize coverage of primary research, fully compatible in terms of outcome measures, only immediate post-test (that is assessment administered at the end of treatment implementation) results will be considered. Various forms of delayed post-tests will be documented and their time lags categorized to inform further reviews. K-12 formal educational settings (approximate age 5-18), in educational programs eventually leading to a certificate, diploma, degree, or advancement to the next academic level/grade are of interest to the current meta-analysis. Other settings (i.e., home schooling, auxiliary programs, summer camps, etc.) are to be excluded. Addressing studies from 2000 onward seems to strike a reasonable balance of covering various approaches to individualized/adaptive learning prominent throughout several decades (studies published in early 2000s would still reflect most interesting pedagogies of 1990s), while primarily focusing on those that retain relevance in most recent educational practices. In order to retrieve a broad base of studies to review, we will begin by having an experienced Information Specialist search across an array of bibliographic databases, both in the subject area and in related disciplines. The following databases will be targeted: ABI/Inform Global (ProQuest) Academic Search Complete (EBSCO) ERIC (EBSCO) PsycINFO (EBSCO) CBCA Education (ProQuest) Australian Education Index British Education Index Education Source (EBSCO) Web of Knowledge Scopus Engineering Village Francis (EBSCO) Medline ProQuest Dissertations & Theses Global ProQuest Education Database Linguistics and Language Behavior Abstracts (ProQuest) Database searching will be supplemented by searches of the Web using Google and Bing to locate additional articles, but also grey literature (research reports, conference papers, theses and research published outside the conventional journals). We will also search the OpenGrey.eu and the Learn Tech Lib online collections for grey literature, and will consult the & for other online the of literature reviews will be for additional relevant studies using a search The most recent of the on will be toward the end of the search process to any recent that our we will in the to we have all their relevant the search will be to the features of the various databases, of and search the following is of what the overall search would based experimental and quasi-experimental studies are to be included as as they two educational interventions covering the same content as on compatible outcome measures, one group is higher in qualities described compared to the other data sufficient for an effect size is a condition for study There are several to the of findings. These 1) of data from the same 2) outcomes of the same and 3) outcomes of different types the same of not apply to the proposed review, as it is limited to learning achievement outcomes The that we will for data are in the In to dimensions of SC pedagogical qualities that would determine for effect size the following of study will be used in the proposed First, study quality will be for features such as design type, of treatment and the of & the same we will for outcome and psychometric quality of the assessment as well as for the of used for effect size and for equivalence of and study these study used in moderator variable will inform us of any potential to all types of study et al., study features will further of SC pedagogical qualities by theoretical instructional practices under review, treatment of for teachers and for students, is by specific instructional intervention. study features will age, educational and level, as well subject All these study features will be as for their potential on treatment All activities (i.e., review, study features as well as effect size will be out by two working and and when a of the i.e., and – for and will be and review aims to include and summarize achievement outcomes of primary empirical studies of high quality that effectiveness of more SC (i.e., more adaptive and individualized) more TC (i.e., more conventional instructional The following are the primary and that will be used for the effect and subsequent analyses. For studies that report for measures of student achievement the of the group will be from the of the group and the will be by the of both data are likely to be in a variety of For studies that report such as or the appropriate will be to the as the effect size & & & To for the small size all will be into the using the following To for statistical of effect we will error of and the In all to effect across studies and for moderator variable we will & This review will focus on learning achievement outcomes. There is a of several measures the same outcome type in the same study based on the same of it we will either in favour of the most – cumulative final or post-tests or when no outcome fully learning performance throughout the of instruction – we will average effects from different or achievement measures. Also, the same of are to be used to effect of the same the same group of is used example, when the same group compared to two different treatment each in its SC qualities – for the purposes of as much of explanatory of the its size will be in order to avoid any (i.e., high of the of the same in the final data effect will be across studies using the effects & as neither of the of interventions and to the of relevant studies) for the effect is on the of our review as a of various populations of empirical studies in the effects a more of the treatment The results from a effects will the overall effect of a diverse of SC instructional on student learning across age subject matters, etc. effect will be used to of the of effect in which the and to determine the to which studies from the effect average for the the a derived from the of true (i.e., what would be based on the error associated with each of effect sizes. If the observed is coded study as potential of systematic will be further moderator variable under the effects It is to potential that may be associated with individual effect and may the overall of the findings. & is to determine the of a effect size the of the effect in a while not the of the effect Various approaches to identifying potential will be of data into a and also study will be examined with the potential to them from the final of such as study type of or will be examined the moderator variable analyses. There is concern that on published studies alone may the overall To potential in this review, we will the and the & and in et al., Also, the will determine the of effect studies to the associated with the average effect a level of will also be used to determine how many studies studies when to the will the a For studies that not report outcome data (that in when information from studies is – the first author of the study will be to retrieve If data are a process of data may be In such a needs to be to the of the data on the overall and synthesis of the results. research will be reviewed within the framework of this research = = = = = information = = = information quality of the outcome assessment = = = = of two of the is based on several outcome Source of outcome = cumulative (e.g., final = in the study (e.g., course = of measures (e.g., of = A out of a size = from = from = from (i.e., no = with reasonable (e.g., size = in the study an of Instruction separately for experimental and = F2F = = = in class or on with or without the of a = = information Conceptual an theoretical framework the treatment is based – to be categorized for subsequent moderator variable when all data are collected experience with implementing the instructional separately for experimental and = = = information the of of treatment Academic level = = school = school = school = information an course – to be categorized in various (e.g., for subsequent moderator variable when all data are collected defined in the study by in – by the results of a = = = = (in of studies) of the in the = students = – when not = needs students – which type of (e.g., or to the = = = – = = = information – the The final of the list is a of of included studies published in the as the review review The author is the who and the review and for individual of the review with the base and takes responsibility for the of the for the of and for the of and for the of and for the of and of content and within the review The optimal review includes at one on the review who has content at one who has and at one who has statistical It is also to have one with information has in and has published on such of to and in the and of learning strategies and collaborative in and distance education information using especially with learners. The for the of and has been since and has a list of We have published meta-analyses in of Educational review in Educational with an of The has also published other meta-analyses and systematic reviews in other and has in many with at one at of the have some for the and etc. in the and several and have published meta-analysis methodology (e.g., et al., & in prominent research is an Information Specialist with experience working on systematic review has as the Education Search for the providing and peer review of search to peer review for and reviews on an & and the of and Individualized as of with other et There are no of We to fully the review by of this responsibility for and the review in with The will as much as to with the of the A review be to the relevant Group within two of protocol If are not the or if we are to for an the relevant Group has the to the or the to The Group also has the to or the if it not the of the Group the responsibility for the review in of and and other and the review at if responsibility for the review to as with the The of the Group in review is to the review, and subsequent in the The no on of the findings of a systematic review in a more form as a either or the of the in have that of findings that have or will and in such a should be of with of the in in a or in in should the and include a to that systematic reviews published in and with the may have additional or for responsibility for any understand the to a review, and to in the on of the