Papers

Papers published by Realizeit or our partners

2020

1.
Sandra Maurice Farah-Franco, Robert Hasel, Analia Tahir, Brian Chui, James Ywom, Briana Young, Menmeet Singh, Scott Turchi, Gary Pape and Bradley Henson (2020)
A preclinical hybrid curriculum and its impact on dental student learning outcomes
Journal of Dental Education 2020; 1-11. doi:10.1002/jdd.12517
Go to paper
Abstract

Objectives

The purpose of this study is to measure how the implementation of an online, preclinical hybrid curriculum impacts dental student clinic readiness, the outcomes of grades, critical thinking skills, and student and faculty perceptions respectively.

Methods

This longitudinal comparative and descriptive study used objective data and subjective (survey) data for 4 dental class cohorts. Groups A and B experienced a traditional lecture‐based curriculum, while Groups C and D experienced a hybrid curriculum that was lecture‐free and implemented active learning. The Health Sciences Reasoning Test (HSRT), an objective assessment, was used to measure students’ critical thinking skills.

Results

Dental student outcomes have either remained steady or improved with the transition to a new hybrid curriculum. According to the student and faculty survey results, the hybrid curriculum promoted student learning, independence, critical thinking, initiative and self‐motivation, and clinic practice readiness. Group C (N = 68) Total Online Platform mean scores demonstrated a significant and moderately strong correlation with the preclinical course mean grades (r = 0.68, P = 0.00). Group D HSRT (n = 63) for Attempt 1 (end of year 1) and Attempt 2 (end of Year 2) paired T test resulted in HSRT Overall (mean difference = −2.27, SD = 7.21, t = −2.5, P = 0.02) for the second preclinical year.

Conclusion

The hybrid curricular approach afforded many benefits. Faculty took an active role in imparting knowledge when compared to the lecture hall. Having students immersed in continual assessment through an online adaptive platform and active learning promoted self‐motivation, deeper learning, applied knowledge, and discouraged superficial memorization.
2.
Charles Dziuban, Colm Howlin, Patsy Moskal, Tammy Muhs, Connie Johnson, Rachel Griffin and Carissa Hamilton (2020)
Adaptive Analytics: It’s About Time
Current Issues in Emerging eLearning: Vol. 7 : Iss. 1 , Article 4
Go to paper
Abstract

This article describes a cooperative research partnership among a large public university, a for-profit private institution and their common adaptive learning platform provider. The focus of this work explored adaptive analytics that uses data the investigators describe as metaphorical “digital learning dust” produced by the platform as a matter of course. The information configured itself into acquired knowledge, growth, baseline status and engagement. Two complimentary models evolved. The first, in the public university, captured end-of-course data for predicting success. The second approach, in the private university, formed the basis of a dynamic real-time data analytic algorithm. In both cases the variables that best predicted students at risk (effective use of time and revision attempts) were deemed teachable skills that can improve with intervention.

2019

3.
Colm Howlin and Charles Dziuban (2019)
Detecting Outlier Behaviors in Student Progress Trajectories Using a Repeated Fuzzy Clustering Approach
Proceedings of the 12th International Conference on Educational Data Mining, 2019
Go to paper View slides
Abstract

Clustering of educational data allows similar students to be grouped, in either crisp or fuzzy sets, based on their similarities. Standard approaches are well suited to identifying common student behaviors; however, by design, they put much less emphasis on less common behaviors or outliers. The approach presented in this paper employs fuzzing clustering in the identification of these outlier behaviors. The algorithm is an iterative one, where clustering is applied, outliers identified, the data restricted to the outliers, and the process repeated. This approach produces a clustering that is crisp between each iteration and fuzzy within. It arose as a consequence of trying to cluster student progress trajectories in an adaptive learning platform. Included are results from applying the repeated fuzzy clustering algorithm to data from multiple courses and semesters at the University of Central Florida, (N=5,044).
4.
C Howlin, E. Lindsay, M.E.D. van den Bogaard and J. Morgan (2019)
Pathways of students’ progress through an on-demand online curriculum
Proceedings of 126th ASEE Annual Conference, 2019
Go to paper
Abstract

Charles Sturt University makes its underpinning technical curriculum available to its students using an on-demand online system they call their Topic Tree. The tree is a directed acyclic graph where nodes represent topics to be learned, and edges represent the prerequisite relationships that exist between the topics. Branches on the topic tree represent concentrations in an area of knowledge, sub-branches (water quality, fluid mechanics, etc.) represent distinct subsets of knowledge - specialty. Delivery of the technical content is in three-hour modules, and students are free to choose the order in which they engage with these topics.

Previous work has identified that students engage with the on-demand curriculum much as they engage with on-demand entertainment platforms such as Netflix, completing long sequences of topics with short periods between them – the traditional “binge” model of consumption.

This paper presents a more fine-grained analysis of students’ pathways through the topic tree, focusing on the distance between successive topics completed by the students. Students’ progress is characterized by a three-dimensional framework – time, distance, and purpose.

In general, pathways through the tree fall into one of four patterns: Forward movement along a branch of the tree, Movement backward along a branch of the tree, Repeating the same topic, Switching to a different branch of the tree (backward distance to the junction of the branches combined with a forward distance along the new branch)

Different students engage with the topic tree using different combinations of these pathways, distance absolute distance traveled through the topics, and different time gaps between activities on the topics. This paper will identify the different combinations that can be found in the student log data.
5.
Charles Dziuban, Colm Howlin, Patsy Moskal, Connie Johnson, Mitchell Eid and Brandon Kmetz (2019)
Adaptive Learning: Context and Complexity
E-Mentor 5(77), 7-39. doi:10.15219/em77.1384
Go to paper
Abstract

This article describes a research partnership between the University of Central Florida and Colorado Technical University, with their common adaptive learning platform provider, Realizeit. The study examines component scores at the two institutions in mathematics and nursing based on a number of Realizeit system metrics. Although the principal components across disciplines and universities remained constant, student scores on those dimensions varied considerably. This indicates that adaptive learning is influenced by context and complexity. The context aspect helps frame student learning regarding knowledge, engagement, communication, and growth as they experience variability from faculty approaches to instruction. Complexity indicates a nonlinear learning pattern for the adaptive process in which the emergent property shows that interactions among the individual elements result in a more realistic model for explaining how students function in contemporary higher education. The authors raise a number of implementation issues for adaptive learning.

2018

6.
Julie F Hinkle and Patsy Moskal (2018)
A Preliminary Examination of Adaptive Case Studies in Nursing Pathophysiology
Current Issues in Emerging eLearning: Vol. 5 : Iss. 1 , Article 3
Go to paper
Abstract

Case studies are a valuable instructional tool frequently used in nursing to allow students to analyze clinical problems based on real-world scenarios. This study examines the use of the Realizeit adaptive platform to create case study scenarios for pathophysiology, a course required in the undergraduate nursing curriculum.

The data gathered as students progressed through the adaptive content--time on task, number of times cases accessed, and scores on each case--provided valuable information on student behavior and engagement with the three case studies. Results of this preliminary study indicate that adaptive case studies are promising for pathophysiology and system analytics confirmed that all but one of the 1,544 simulations presented to students were unique. This provides a benefit over what would typically be a limited number of distinct options when using instructional case studies without the adaptive learning system. Future research is suggested to examine additional uses of case studies and their impact on students’ knowledge acquisition and engagement when part of a course’s graded assignments.
7.
Constance Johnson and Emma Zone (2018)
Achieving a Scaled Implementation of Adaptive Learning through Faculty Engagement: A Case Study
Current Issues in Emerging eLearning: Vol. 5 : Iss. 1 , Article 7
Go to paper
Abstract

This paper presents a case study describing the implementation of adaptive learning at Colorado Technical University (CTU) with a focus on faculty adoption. A number of barriers to the adoption of technology will be discussed and more importantly, how CTU overcame these barriers. A description of the key elements of faculty support including training will be outlined as well as the information about the adoption of faculty using data to inform teaching strategies.

The authors argue that if given the choice, faculty at CTU would prefer adaptive learning technology in their courses and welcome the use of technology and data to enhance the classroom experience.
8.
Patti O’Sullivan (2018)
APLU Adaptive Courseware Grant, A Case Study: Implementation at the University Of Mississippi
Current Issues in Emerging eLearning: Vol. 5 : Iss. 1 , Article 5
Go to paper
Abstract

Learning Management Systems (LMS) have been the main vehicle for delivering and managing e-learning courses in educational, business, governmental and vocational learning settings. Since the mid-nineties there is a plethora of LMS in the market with a vast array of features. The increasing complexity of these platforms makes LMS evaluation a hard and demanding process that requires a lot of knowledge, time, and effort.

Nearly 50% of respondents in recent surveys have indicated they seek to change their existing LMS primarily due to user experience issues. Yet the vast majority of the extant literature focuses only on LMS capabilities in relation to administration and management of teaching and learning processes.

{"In this study the authors try to build a conceptual framework and evaluation model of LMS through the lens of User Experience (UX) research and practice, an epistemology that is quite important but currently neglected in the e-learning domain. They conducted an online survey with 446 learning professionals, and from the results, developed a new UX-oriented evaluation model with four dimensions"=>"pragmatic quality, authentic learning, motivation and engagement, and autonomy and relatedness. Their discussion on findings includes some ideas for future research."}
9.
M.E.D. van den Bogaard, C. Howlin, E. Lindsay and J. Morgan (2018)
Patterns Of Student’s Curriculum Engagement In An On-demand Online Curriculum: An Exploratory Study At Charles Sturt University
Proceedings of the 46th SEFI Annual Conference 2018. p487-494.
Go to paper
Abstract

Technology is changing the way that people consume media, with platforms such as Netflix and Youtube enabling a shift towards asynchronous engagement with entertainment. University curricula, however, remain heavily synchronised through the dominance of lecture-tutorial-laboratory pedagogies.

The new engineering degree at Charles Sturt University offers its students asynchronous access to its underlying technical content. The syllabus is divided into topics, each of which is scoped such that a student should take around three hours to complete the work. Initial anecdotal evidence of the program suggests that the students in this program engage with their curriculum the same way that they engage with Netflix, with patterns of bingeing, as well as delaying until “seasons” of topics are available before commencing learning a sequence of material.

This paper reports on students’ patterns of engagement with the topics, analysing patterns of access and completion, and identifying archetypal engagement modes belonging to different students.
10.
Charles Dziuban, Colm Howlin, Patsy Moskal, Connie Johnson, Liza Parker and Maria Campbell (2018)
Adaptive Learning: A Stabilizing Influence Across Disciplines and Universities
Online Learning, 22(3), 7-39. doi:10.24059/olj.v22i3.1465
Go to paper
Abstract

This study represents an adaptive learning partnership among The University of Central Florida, Colorado Technical University, and the platform provider Realizeit. A thirteen-variable learning domain for students forms the basis of a component invariance study. The results show that four dimensions - knowledge acquisition, engagement activities, communication and growth remain constant in nursing and mathematics courses across the two universities, indicating that the adaptive modality stabilizes learning organization in multiple disciplines.

The authors contend that similar collaborative partnerships among universities and vendors is an important next step in the research process.

2017

11.
Charles Dziuban, Colm Howlin, Constance Johnson and Patsy Moskal (2017)
An Adaptive Learning Partnership
Educause Review, December 18 2017 (Editors' Pick)
Go to paper
12.
Charles Dziuban (2017)
The Technology of Adaptive Learning
Education Technology Insights (Editors' Pick)
Go to paper
13.
Dziuban C., Moskal P., Johnson C., Evans D. (2017)
Adaptive Learning: A Tale of Two Contexts
Issues in Emerging eLearning: Vol. 4 : Iss. 1 , Article 3
Go to paper
Abstract

This paper presents the results of student reactions to adaptive learning at two universities with considerably different contexts - a large public institution and a for-profit, professional university. A student response protocol developed by and administered at the University of Central Florida (UCF) was also distributed to students at Colorado Technical University (CTU).

Demographic comparisons of the two responding sample groups indicated considerable differences in student characteristics, especially with respect to age and work status. However, a factor invariance comparison revealed that students at both universities evaluated the adaptive climate similarly though the lens of learning environment, guidance path and progression. When the factor scores for the institutions were compared, CTU students responded more favorably to the guidance component of adaptive learning while UCF students perceived that the adaptive learning system provided a more effective learning environment. Students who were clustered by whether or not they would reengage with adaptive courses, showed a positive and somewhat more ambivalent group.

The authors concluded that adaptive learning with its flexibility and variable time component is a possible solution to the scarcity problem in our educational system, addressing students with too many needs and too few resources. The authors contend that adaptive learning could help to level the educational and economic playing fields in our society."
14.
Patsy Moskal, Donald Carter and Dale Johnson (2017)
7 Things You Should Know About Adaptive Learning
EDUCAUSE Learning Initiative
Go to paper
Abstract

Adaptive learning is one technique for providing personalized learning, which aims to provide efficient, effective, and customized learning paths to engage each student. Adaptive learning systems use a data-driven approach to adjust the path and pace of learning, enabling the delivery of personalized learning at scale. Adaptive systems can support changes in the role of faculty, enable innovative teaching practices, and incorporate a variety of content formats to support students according to their learning needs.
15.
Ford C., McNally D., Ford K. (2017)
Using Design-Based Research in Higher Education Innovation
Online Learning, 21(3), 50-67
Go to paper
Abstract

This paper discusses the design-based research approach used by the Center for Innovation in Learning and Student Success at the University of Maryland (CILSS), University College. CILSS is a laboratory for conducting applied research that focuses on continuous improvements to the university's instruction of curriculum, learning models, and student support to identify promising innovations for underserved populations in adult higher education; to drive adoption of next-generation transformational online learning; to develop new educational models based on learning science, cutting edge technology, and improved instructional methods; to help more UMUC adult students succeed by increasing retention and graduating more students in shorter time frames (thus reducing their costs). As such, leveraging technology and pedagogy in innovative ways is key to the Center's work. CILSS serves as the research and development arm for the university, promoting innovative ideas and breakthroughs in learning.

The paper details one interpretation of design-based research (DBR) and how it can be applied by an innovation center working within a university for program evaluation. This paper also posits that the conceptual framework and assumptions of andragogy (Knowles, 1984) has applicable relevance to the instructional shifts that include adaptive learning in the curriculum. A review of the literature on DBR explores the central features of this approach. A review of andragogy as the conceptual framework for this paper highlights what we believe to be the central features of the evaluation approach of adaptive learning software. We then present the model used by CILSS when designing and testing a pilot project. To illustrate the approach, we provide the example of a recent pilot that uses the adaptive learning software RealizeIt in UMUC’s Principles of Accounting I course, a course that traditionally has lower than average success rates."

2016

16.
Dziuban C., Moskal P., Hartman J. (2016)
Adapting to Learn, Learning to Adapt
ECAR Research Bulletin September 30, 2016
Go to paper
17.
Dziuban C., Moskal P., Cassisi J., Fawcett A. (2016)
Adaptive Learning in Psychology: Wayfinding in the Digital Age
Online Learning, 20(3), 74-96
Go to paper
Abstract

This paper presents the results of a pilot study investigating the use of the Realizeit adaptive learning platform to deliver a fully online General Psychology course across two semesters. Through mutual cooperation, UCF and vendor (CCKF) researchers examined students’ affective, behavioral, and cognitive reactions to the system. Student survey results indicated that students found the system easy to use and were generally positive about their seamless transition to adaptive learning.

While the majority of students were successful, learning outcome metrics utilizing Realizeit indices indicated a potential for early prediction of students who are likely to be at risk in this environment. Recommendations are presented for the benefits of cooperative research between users and vendors."
18.
Kathleen Bastedo and Thomas Cavanagh (2016)
Personalized Learning as a Team Sport: What IT Professionals Need to Know
Educause Review, December 19 2016
Go to paper

2014

19.
Howlin C.P., & Lynch D.J. (2014)
A Framework for the Delivery of Personalized Adaptive Content
Proceedings of the ICWOAL Conference 2014
Download paper
Abstract

The number of students taking online courses has grown dramatically in recent years. Increasingly institutions are turning to adaptive platforms to deliver these online courses as they seek to offer personalized learning to each individual. The goal is to allow learners to progress at their own pace, place and convenience. Key to any learning system is the content through which the learning is delivered. Within the approach used by Realizeit learning platform, curriculum and content are separated. The curriculum is used to drive the direction of the personalized learning path, while the content is responsible for the delivery and presentation of the concepts to be learned. This paper outlines the content framework behind this approach which can enable the delivery of personalized adaptive learning.
20.
Lynch D.J., & Howlin C.P. (2014)
Real World Usage of an Adaptive Testing Algorithm to Uncover Latent Knowledge
Proceedings of the ICERI 2014 Conference, pp. 504–511
Download paper
Abstract

At the beginning of every course, it can be expected that several students already have some knowledge of the curriculum. For learning efficiency it is important to quickly identify this latent knowledge. This enables students to begin new learning immediately which in turn prevents the frustration or boredom that might be caused otherwise. A component of the Realizeit learning environment that uncovers this latent knowledge is an algorithm called Determine Knowledge. This is offered as an adaptive pretest at the beginning of each course. It allows new students to be automatically placed at the appropriate position within the course. In this paper, we detail the algorithm and review its relative strengths and weaknesses. In particular we discuss the real-world usage of Determine Knowledge run in a higher educational institute in the United States over a two year period. In total this encompasses over 455,000 Determine Knowledge operations performed by over 48,500 unique students. We finish the paper by outlining some of the enhancements made to the algorithm based on the analysis of this data.
21.
Lynch D.J., & Howlin C.P. (2014)
Uncovering Latent Knowledge: A Comparison of Two Algorithms
UMAP 2014, LNCS 8538, pp. 363–368, 2014, Springer International Publishing Switzerland
Download paper
Abstract

At the beginning of every course, it can be expected that several students have some syllabus knowledge. For efficiency in learning systems, and to combat student frustration and boredom, it is important to quickly uncover this latent knowledge. This enables students to begin new learning immediately. In this paper we compare two algorithms used to achieve this goal, both based on the theory of Knowledge Spaces. Simulated students were created with appropriate answering patterns based on predefined latent knowledge from a subsection of a real course. For each student, both algorithms were applied to compare their efficiency and their accuracy. We examine the trade-off between both sets of outcomes, and conclude with the merits and constraints of each algorithm.