Realizeit Labs

Adaptive Analytics: CTU - a Time Critical Setting for Predictive Modeling

2021-01-05T10:00:00+00:00

This is the second blog post summarizing results from a collaboration with the University of Central Florida and Colorado Technical University on building models that can predict student success at the course level for College Algebra. You can access the first summary blog post here and read the full set of results in the following article:

Charles Dziuban, Colm Howlin, Patsy Moskal, Tammy Muhs, Connie Johnson, Rachel Griffin, and Carissa Hamilton (2020). Adaptive Analytics: It’s About Time Current Issues in Emerging eLearning: Vol. 7 : Iss. 1 , Article 4

In this blog post, we examine the impact of building a sequence of predictive models corresponding to different time slices within a course. We look at questions such as how early in the course you can get reliable models and what are the most important indicators for student success at various time points. For this section, we use data from Colorado Technical University (CTU). This proves to be a particularly tough test bed for building and deploying time-sensitive models as their courses are just 5.5 weeks long, limiting the amount of time to gather data and for educators to act on any predictions.

The Goal

The University of Central Florida (UCF) data suggested an approach to predictive modeling that provides learners with concrete and learnable actions that positively impact their odds of success. While a UCF course typically lasts 16 weeks, a CTU course lasts just 5.5 weeks. The short length of the terms at CTU produces a much more challenging environment for any predictive model. In this section of the study, we explored the impact of time on predictive models’ effectiveness. In particular, we sought to answer two questions:

How long does it take to collect data to build a reliable predictive model?
What are the most important metrics when making a prediction, and do they change over time?

The Data

For this analysis, data were gathered from over 5,000 students across seven terms in a math course at CTU. The metrics used are slightly different from those used in the UCF analysis; however, they are all derived from the data implicitly collected by the Realizeit adaptive learning platform. There are eight metrics used in this study and they capture: the level of engagement (e.g., total time, number of activities), the breadth of engagement (e.g., number of objectives attempted), and level of knowledge (e.g., number of concepts complete). Please reference our article for more details.

The Model

To bring time into the model, we built a sequence of models (classification trees) corresponding to weekly time slices. So the first model would use week 1 data to make its predictions, the second model would use data from week 1 and 2 to make its predictions, and so on. So the overall model contains one “sub-model” (classification tree) for each week of the course.

Enough Data

Using the most common outcome from the previous version of the course, we can construct a baseline model. This model predicts everyone passes or everyone fails, depending on which is the most common outcome.

“If a course has very high or very low success rates, then this baseline model can be accurate. However, it may not be a particularly informative model because it does not provide insights into why students are successful or not.”

We compare our model’s accuracy to this baseline model for each week of the course on unseen data. Because we use classification trees, if our model does not have enough data, it will revert to the simple baseline model. Only when the data contains some signal that can help the model distinguish between the successful and unsuccessful students will it begin to beat the baseline. We can see when this happens in the CTU Math data by examining the chart below. It shows the accuracy ratio of the two models - values higher than one shows that our model is beating the baseline.

We can see that not until week three is there enough signal in the data to beat the baseline. This has some consequences for rolling out early warning type models in time-constrained courses.

“Requiring three weeks of data poses a considerable challenge in the CTU context because it leaves just two and a half weeks before course completion to intervene with students predicted to be unsuccessful. This three-week requirement of data is also present in the UCF context; however, with 16-week courses, there is usually sufficient time to intervene.”

Variable Importance

Predictor Importance (for more details, see the paper) allows us to measure how important an individual variable is to each weekly sub-model. This metric tells us which variables are contributing the most signal to the predictions. Importance ranking is used rather than raw importance scores to simplify our analysis, with the variables ranked from most to least important. The chart below shows the change in rank over time.

There are several noteworthy outcomes:

Seat-time measures (start day, number of active days) remain unimportant across all models/time slices. They contain little or no signal for course success.
Measures that capture the quantity of engagement (total time, number of activities) start out as important, but decrease in rank over time.
Measures that capture the quality and breadth of engagement (number of nodes attempted, number of nodes completed, mean knowledge covered) increase in importance.

The key take-away for an educator is that

“… the variables that need to be monitored change as the course progresses. Metrics related to the traditional seat-time view were not predictive of student success. While at the beginning of the course, it is important to monitor effort levels, as the course progresses, it becomes more important to monitor the quality of the engagement and the level of progress of students.”

Impact

The model described above follows the general architecture of the predictive models used in Realizeit. While rolling out these models, we encountered several challenges that we thought would be worth sharing. These include

Predictions can quickly become out of date - With predictions made once a week, they can quickly become irrelevant as the student makes progress. This could lead to making decisions based on outdated information.
Counter-intuitive predictions - Some predictions can appear counter-intuitive when solely examining outcome metrics, as the model also takes behavior into account. For example, two students may have the same number of concepts covered and to the same standard, but one may have engaged in much more revision and practice increasing their chance of success.
Guidance instead of probabilities - Research shows that people are generally not good at interpreting probabilities. They can assume that a high probability guarantees success, while a low probability implies certain failure. They can also struggle with knowing what to do to increase the probability of success - what is their best strategy going forward? To avoid these problems, we need models to move towards providing actual guidance, not just numbers.

Work is already on the way to tackle each of these challenges, and we are eager to share what we learn.

Next Up

If you haven’t yet, read our summary of the first part of this study, which discusses the work carried out by the University of Central Florid, where they explored building models based on variables that capture teachable skills.

If you found the research article or summary blog posts interesting, please share them with your colleagues!

Adaptive Analytics: Predictive Modeling Based on Teachable Skills at UCF

2021-01-04T10:00:00+00:00

We have recently collaborated with the University of Central Florida and Colorado Technical University on building models that can predict student success at the course level for College Algebra. The journal Current Issues in Emerging eLearning have published our results in the following article:

This blog post summarises a portion of those results. Here we discuss the work carried out at the University of Central Florida, which examined building models based on variables that capture teachable skills. The goal is the possibility of empowering

“… curriculum designers and instructors how to manipulate these analytics variables to engineer student success.”

A second blog post covers the rest of the research results. There we discuss the work carried out on data from Colorado Technical University where we bring time into the model and examine questions such as how early in the course you can get reliable models of success, and what are the most important indicators for student success at various time points in a course.

The Goal

This portion of the study’s primary goal is to find actionable variables that predict student success in the hope that we can create interventions based on these variables. The focus on “actionable” is essential. For example, take GPA. It positively correlates with success, but students arrive with a set GPA. We cannot change it or tell them to come back when it is higher; we have to deal with it as it is. More precisely, as stated in the paper, we wish to

“… address the question of whether or not adaptive learning, with its variable time learning framework, provides a platform for finding actionable analytics variables that predict student success in Algebra and that also are responsive to instruction.”

“The phrase “responsive to instruction” refers to our hope that, if we were able to identify actionable analytics variables that correlate with positive learning outcomes, we also would be able to identify possibilities for teaching curriculum designers and instructors how to manipulate these analytics variables to engineer student success.”

College Algebra at UCF

College Algebra at the UCF requires students to sit for a mathematics placement examination. If they do not meet the department requirement, they must first take a non-credit Intermediate Algebra course. Despite this, at the time of this analysis, nonsuccess in Algebra (a grade of less than C) for students enrolling directly or through IA was approximately 41%. Stated another way, a typical students’ odds of success are favorable but only marginally (about 1.4:1).

The Data

The study used data collected by the Realizeit adaptive learning platform, which provides a wide range of metrics. Our previous work showed that four factors underly these metrics. They broadly measure:

Knowledge Status
Knowledge Growth
Engagement
Baseline

Please reference the article for more details. When building the model, we focus on the engagement metrics, namely Number of Revisions and Total Time, and are joined by the externally provided student GPA.

The Model

The three variables were converted to quartiles to identify the gain for different cohorts more accurately. This also permitted a more straightforward comparison across cohorts to determine if what we developed would help those in most need by improving their odds of success. Noting the strong influence of GPA on predictions, it played the role of a mediator throughout the analysis - that is, we grouped students by GPA quartile and developed rules for each cohort. GPA quartiles 2 and 3 were not significantly different from each other, so they were combined in the following results.

For each of the three cohorts, we uncovered the same following rule: If students “can obtain a revision placement Q2 through Q4 and a total time of Q4, then their chance of nonsuccess drops.” The rule holds across all cohorts; however, the level of impact varied for each cohort.

Before we examine each cohort, it is worth remembering that the course’s general level of nonsuccess is 41%. However, this level varies for each of the cohorts. For GPA Q1 (bottom 25%), their level of nonsuccess is 74%, nearly twice the class’s level in general. For GPA Q2 & Q3 (middle 50%), the level is just below the course level at 39%, and for GPA Q4 (top 25%), the level of nonsuccess is just 12%. This last fact is interesting as it shows that while having a high GPA gives a student a great chance of success in College Algebra, it does not guarantee it!

Students who followed the rule and had sufficiently high levels of revision and time spent engaged could dramatically decrease their odds of nonsuccess. For GPA Q1 (bottom 25%), their level of nonsuccess nearly halved, dropping to 39%, just below the class’s level in general. For GPA Q2 & Q3 (middle 50%), the level dropped to 24%, and for GPA Q4 (top 25%), the level of nonsuccess dropped to just 4%. The graphic below summarizes these results.

The key takeaway for an educator is:

“There is a 35% lift for students in GPA Q1, substantially increasing their chance of success. There is a moderate but helpful lift (15%) for students in Q2-Q3 and very little lift for those individuals in Q4 (8%). The rules were most effective for those who needed assistance the most, but, relatively ineffective for those who needed it least.”

So all students can increase their chance of success by revising more and spending more time learning. The students that need the most help benefit the most. This is straightforward guidance to give to students and something that should be easy for them to follow.

Next Up

Read the rest of the research findings in our summary of the work carried out on data from Colorado Technical University, where we bring time into the model and examine questions such as how early in the course you can get reliable models of success, and what are the most important indicators for student success at various time points in a course.

If you found the research article or summary blog posts interesting, please share them with your colleagues!

Supply Chain Management and Adaptive Learning

2020-01-26T10:00:00+00:00

There is an adage that there is nothing new under the sun. However, often, dissimilar terminology, contexts and time frames obscure commonalities in theories and problem solutions. For example, Steven Johnson (2005) in his book, “Everything Bad is Good for You” argues that video games that are too easy or difficult for players will cause them to disengage very quickly. He contends that the proper balance between difficulty and reinforcement keeps gamers motivated. What he describes resonates with Vygotsky’s (1978) zone of proximal development for effective learning framed with proper levels of tension and achievement. Although postulated in different time frames the two concepts intersect, reinforce and validate each other. To our knowledge, however, Johnson does not reference Vygotsky, who proceeded him by almost three decades.

There are other examples. In physics, for instance, the law of space and time translation symmetry (Wilczek, 2019) shows that no matter where or when a physical principle is evaluated it will remain constant – it is invariant. There is a corollary in education, psychology, natural science and medicine where robustness of findings become a high-value proposition. Most often that stability emerges through meta analytic methods (Cooper, Hedges & Valentine, 2019) identifying effect sizes that are independent of time, context and discipline. One is a physical law and the other is a research method; however, the desired outcome is the same: robustness. One more example reinforces our point. There is a theory in economics called the yard sale model (Boghosian, 2019) that demonstrates a breaking of symmetry in free-market economies, causing wealth to flow from the poor to the rich. This wealth advantage, sometimes called the Mathew Effect (Saleh & Sanders, 2014), has a clear analogy in educational attainment, where educational opportunity in underserved neighborhoods restricts college access, thereby asymmetrically limiting financial opportunity. In this example, an economic theory of wealth distribution bares a direct relationship to effective educational practice.

There are many other examples. However, our point is that what appear to be independent lines of investigation at first blush are closely aligned. Unfortunately, however, it is all too rare for cross-discipline conversations to occur even in this age of electronic communication. Even more importantly, it is highly likely that our bright new idea has seen the light of day in the recent or even the distant past. For example, adaptive learning and supply chain management have fundamentally similar theoretical underpinnings and structural elements. Consider the following examples.

The Supply Chain

Since its inception in the early 1980s, the term “supply chain” has had many interpretations, but a commonly accepted definition is a group of companies that pass information, products, services, and funds up and down the chain (Cooper, Lambert, & Pagh, 1997; Lambert, Stock, & Ellram, 1998; Mentzer & Mentzer, 2001). The process is chiefly built and operated on trust among the firms that are contributing members of the partnership. The figure above illustrates an example of a typical supply chain.

When demand for products or services increases or decreases, these changes reverberate through the entire system. For example, if demand for a product begins to decline, then the amount of raw materials sourced by suppliers must decrease, reducing inventory levels at each stage. However, if demand increases, then additional product must be sourced, produced, shipped, and inventoried throughout the system. Both situations help ensure there are enough goods to meet the demand of the consumer.

Parallel Elements in Adaptive Learning

In the figure below we circle back to our introduction about intersecting disciplines. Adaptive learning (Carroll, 1963) features three guiding principles that resonates with supply chain management; it personalizes the educational experience, customizes content and assesses continuously. The implementation of these principles in the educational environment requires functional elements that corresponded to those of supply chain management. The instructional designer is the surrogate supplier, who works in conjunction with the instructor to emulate the manufacturer. The adaptive platform pairs with the distributer. Instructors are the knowledge and learning retailers and students become the customers.

The table below demonstrates the correspondence of the elements of supply chain management with those of adaptive learning:

Supply Chain Management	Adaptive Learning
Supplier Responsible for sourcing materials at a good price, sufficient quality and providing these materials to a manufacturer who will then produce the product.	Instructional Designer The Learning Content and Curriculum are the raw materials of learning. The Instructional Designer (ID) is responsible for the creation and quality of these.
Manufacturer Responsible for producing the product from the raw materials sourced and delivered by the supplier. They also design, package and label the product prior to being placed on the market.	Instructional Designer / Instructor The Program or Course organizes the raw materials into a product that can be consumed by the end user, the learner. The ID and instructor are responsible for packaging the right raw materials together to match the requirements and needs of the learners.
Distributor/Wholesaler Responsible for connecting the manufacturer to the customer. They deliver the right products, in the required quantities, at the correct time, and in the condition desired by the customer.	Adaptive Platform The adaptive platform is responsible for taking the contents of the program/course and delivering the appropriate learning at the right time for each individual student.
Retailer Responsible for selling product or service directly to the customer. They are also the direct point of contact between a firm and the customer and are in the best position to know and collect the customer’s needs.	Instructor The human point of contact between the learning and the learner. The instructor is in the position to interact directly with the student and ensure their learning needs are being met.
Customer Responsible for the purchase of the product or service. This segment provides valuable feedback which enables the firm to better the product or service they offer.	Student The end consumer in the learning process. Explicit feedback from the students, along with their learning data gathered by the adaptive platform, enable the improvement of the alignment between the learning products and the student needs.

Two Case Studies

Amazon Fulfillment Center

A firm’s supply chain can be enhanced by technology and automation. Integration of logistical systems, robotics, warehouse conveyor systems, enterprise resource planning software, etc., all have the goal of improving delivery times to customers and enabling real-time visibility throughout the supply chain. A company that exemplifies the use of technology and automation is Amazon at their fulfillment centers.

Amazon fulfillment centers use thousands of robots that perform many repetitious jobs that humans could do but would not generally enhance proficiency and potentially experience long-term physical consequences. This type of automation, however, does not replace people per se. Strategically the company will use and cross train this available workforce in other areas of their facilities (inbound, outbound, packing, etc.). This forward thinking enables the company to fill in scheduling gaps during peak and non-peak seasons.

Amazon’s mixture of humans and robots has proven that people, no matter how much automation is present, will always be tasked with making on-the-spot business decisions for the company.

Adaptive Learning

In many areas, people have begun to take advantage of computers and artificial intelligence to automate some aspects of their tasks. The key idea is to take the components of that job that each is most proficient at, and work in partnership to realize previously unachievable results. The Amazon fulfillment centers are excellent examples.

Several recent examples of this are detailed in the recent book by Hannah Fry (Fry, 2018). One focuses on the field of medicine, where pathologists search for cancerous cells within biopsies requiring processing of a huge quantity of information in a single slide. This is arduous and tiring work that can lead to human error. Recent advances in AI approaches allow computers to quickly identify anomalous groups of cells in a slide; however, they regularly flag healthy cells, something that a pathologist rarely does. Combining the power of the computer with the accuracy of the human expert allows the affordances of both to be used. As stated in the book, “The algorithm never gets tired and the pathologist rarely misdiagnoses. The algorithm and the human work together in partnership, exploiting each other’s strengths and compensating for each other’s flaws.”

Another example in the book comes from the world of chess. After chess grandmaster Garry Kasparov was famously beaten by the supercomputer Deep Blue in 1997, he turned his attention to a version of the game known as Centaur Chess, where a human player and an algorithm work in collaboration with each other as a hybrid team, each focusing on their own strengths. The human is in charge and dictates the strategic plan; the computer works on assessing the consequence of all possible moves, reducing the chance of error. This frees the humans from the calculation task and allows them to be more creative. Kasparov has stated that “when playing with the assistance of computers, we can concentrate on strategic planning instead of spending so much time on calculations. Human creativity was even more prominent under these conditions (GenPact, 2018).”

Adaptive Learning also seeks to take advantage of data, AI and automation. The goal is not to replace the role of the instructor but to remove the burden of tasks that can be automated, such as the creation of assessments, and to allow the implementation of options that would not be feasible with a single instructor, such as allowing students to follow personalized pathways and move at their own pace. This frees the instructor to concentrate on areas that the machine will never be capable of doing as effectively, such as interacting with students.

Finally

Both supply chain management and adaptive learning, although contextually different, constitute similar autocatalytic systems that feature responsiveness to the needs of their respective clientele. In both approaches a change in consumer demand or student accomplishment ripples through each framework, causing the elements in the table above to adjust – sometimes incrementally or in other instances more dramatically. Consistent feedback makes both systems dynamic.

Both integrate human performance, and technology redefines the roles of employees, students and teachers. Both incorporate machine learning such as artificial intelligence, or possibly Bayesian decision algorithms. Both have potential for providing solutions for societal problems beyond their immediate spheres of influence. A prime example is quality of life for educationally and economically underserved populations by providing access to not only quality goods and services at reasonable prices but educational opportunity as well.

Therefore, those professionals working on the supply chain side can inform adaptive learning systems by providing insights into effective practices and difficult issues in the chain. Conversely, those educators working in adaptive learning can offer insights to their colleagues in business about how effective learning and information sequencing can improve their processes. This seems reasonable because the remarkable similarities in the two systems indicate that the supply chain of products and information have originated from the same thought process.

References

Boghosian, B.M. (2019, November). The inescapable casino. Scientific American, 321(5), 71-77.
Carroll, J. B. (1963). A model of school learning. Teachers College Record, 64, 723-733.
Cooper, M. C., Lambert, D. M., & Pagh, J. D. (1997). Supply Chain Management: More Than a New Name for Logistics. The International Journal of Logistics Management, 8(1), 1-14.
Cooper, H., Hedges, L.V. & Valentine, J.C. (2019). The handbook of research synthesis and meta-analysis. New York, NY: Sage.
GenPact. (2018). Don’t underestimate importance of process in coming world of AI. Retrieved from http://www.genpact.com/ insight/ blog/ dont-underestimate-importance-of-process-in-coming-world-of-ai.
Fry, Hannah (2018). Hello World: Being Human in the Age of Algorithms (1st ed.). New York, NY: W. W. Norton & Company.
Johnson, S. (2005). Everything bad is good for you: How today’s popular culture is actually making us smarter. New York, NY: Riverhead Books.
Lambert, D. M., Stock, J. R., & Ellram, L. M. (1998). Fundamentals of logistics management. Boston, MA: Irwin/McGraw-Hill.
Mentzer, J. T., & Mentzer, J. T. (2001). Supply chain management. Thousand Oaks, CA: Sage Publications.
Saleh, A., & Sanders, H. (2014). The wolf in sheep’s clothing: The Matthew effect in online education. Revista Internacional de Sociología de la Educación, 3(1), 26-50.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
Wilczek, F. (2019, November). Crystals in time. Scientific American, 321(5), 28-35.

Adaptive Learning: A Stabilizing Influence

2018-10-22T09:01:00+00:00

Recently we had a paper Adaptive Learning: A Stabilizing Influence Across Disciplines and Universities published in the Online Learning Journal based on some of the collaborative research that we have been doing with two partner institutions, the University of Central Florida (UCF) and Colorado Technical University. Before I get on to talking about the paper itself, I want to tell you a bit about this partnership as we see this as a great model for how we would like to work with others.

An Adaptive Partnership

Both UCF and CTU are early adopters of adaptive learning. We began working with CTU in 2012 and UCF in 2014. These are two very different institutions in how they are set up, how they use adaptive learning and the population of students they serve. UCF is a research-intensive university, and from their very first trials of Realizeit, we have been working with Chuck Dziuban and Patsy Moskal from the UCF Center for Distributed Learning to understand the impact of adaptive learning on students and learning. The formation of a collaborative relationship between all three organizations occurred organically as people from UCF and CTU began to meet at conferences and learn of each other’s work. Connie Johnson, CTU’s chief academic officer and provost, has led their effort to share their experiences and success with adaptive learning.

Chuck, Patsy, Connie and Colm at the 2018 Educause ELI Conference in New Orleans.

What makes this collaboration so successful, and why we see it as a great model for collaborating with others, is that each of us brings something different to the relationship. Not only do we get different perspectives on adaptive learning from the differences between the institutions, but each organization also bring different capabilities. UCF brings their considerable research experience and expertise, CTU has successfully taken adaptive to scale, and Realizeit brings expert knowledge of the system and the data it produces. Together we have produced research ranging from students’ perceptions of adaptive learning to how stable the underlying structure of adaptive learning is across domains and institutions. You can see the papers we have produced on our Papers page.

A Stabilizing Influence

In our most recent collaborative work, we sought to examine and understand the hidden dimensions underlying the adaptive learning process. Combining the data from both UCF and CTU would allow us to make comparisons across multiple disciplines and in two very different universities. Our objective was to determine if differing disciplines and university contexts alter the learning patterns, thereby impacting on effectiveness. This has implications for the fields of learning science and predictive analytics. If the underlying dimensions vary considerably by context, then it creates difficulties for how we translate what we’ve learned, and the models of learning that we build, from one context to another. I’m only going to give a high-level overview here of our work and findings but be sure to check out the full paper for more details.

The primary technique used in this work is Principal Component Analysis (PCA). While you don’t need to have expert knowledge of this technique to understand the paper, knowing some of the basics would help, so I’ve provided a quick introduction in this post.

The Courses and the Data

For this paper, we concentrated on courses in mathematics and nursing. In the table below, you can see the list of courses chosen and the number of students and sections in each course.

Institution	Course	Students (Sections)
UCF	Intermediate Algebra	332 (2)
	College Algebra	363 (5)
	Pathophysiology	537 (9)
CTU	Introduction to Algebra	6,693 (38)
	Analytic College Algebra	4,486 (26)
	Trends in Contemporary Nursing	303 (30)

The courses used in the study from each institution.

For the study, we used 13 variables generated by the Realizeit platform. These describe the students’ outcomes and engagement.

The Analysis

We found these to be intercorrelated, so we subjected them to the PCA procedure. We performed this for each of the individual courses in each institution, and for the combined sample of all three courses in each institution. This generated a total of eight pattern matrices, one for each sample. We retained components with an eigenvalue greater than one and used pattern coefficients with an absolute value greater than .40 to help form an interpretation of the components. I’ve included one of the pattern matrices below.

	Component
	C1	C2	C3	C4
Calculated	0.95
Knowledge covered	0.95
Knowledge state	0.91
Determine knowledge	0.79
Average score
Total activities		0.97
Num. revised		0.90
Num. practiced		0.61
Interactions			0.98
Messages sent			0.98
Knowledge covered growth				0.93
Knowledge state growth				0.92
Total time				0.44

Pattern Matrix for the Entire Sample at UCF.

The pattern matrices capture the structure of adaptive learning in each of the samples. To answer our question on whether the organization of adaptive learning is constant or if the patterns change by institution or course context, we needed to compare the 8 pattern matrices (called component solutions in the paper) to each other.

The pattern matrices for all 8 samples.

In our work, we examined the 28 possible pairwise comparisons of the eight patterns. In the paper, we limited the comparison to the 10 comparisons given below, but the results hold across all possible comparisons.

Internal Institutional Comparisons – comparing patterns across samples within an institution
- UCF (3 samples → 3 comparisons)
- CTU (3 samples → 3 comparisons)
Cross-Institutional Comparisons – comparing patterns from samples across institutions
- Entire Samples (2 samples → 1 comparison)
- Course Level
  - Comparison of the matched algebra courses (2 sets of 2 samples → 2 comparisons)
  - Comparison of the two nursing courses (2 samples → 1 comparison)

To make the comparisons, we used a metric known as the Tucker congruence coefficient. You can think of it as a correlation type metric for pattern matrices. We generated this metric for each component pair in each comparison and each comparison as a whole. There is far more detail in the paper as we talk through each comparison, but in summary, we found that they all match. In the samples available to us, the organization of adaptive learning is constant across institution and course context - we found the same four components in each of the samples. But what are these components? How do we interpret them in real-world terms?

The Components

By examining how the four components relate to each of the original variables we derived the following interpretation. (Note in some patterns C3 and C4 swap position.)

C1: Knowledge Acquisition relates to achievement and has a mastery element associated with it. Knowledge acquisition in adaptive learning assesses learning before, during, and upon completion of a course and forms the benchmark for student success. Also, it serves as the basis of the decision engine’s recommendation about the appropriate learning path for students and an early indication of possible difficulties in the learning sequence.
C2: Engagement Activities bears a strong relationship to what Carroll called the time students spend in actual learning and relates to how much energy a student expends in the learning process. If one could hold ability level constant, a reasonable assumption might be that students who are more engaged in learning activities will score higher on knowledge acquisition.
C3: Communication emerges in the Realizeit platform, enabled by messages sent and interactions. This is the social dimension of adaptive learning and the way students communicate with each other and their instructors. At another level this component underlies the effort expended communicating in courses.
C4: Growth is a clear expectation for any course. Measuring change in knowledge acquisition can result from many baseline measures and is an essential element of the learning cycle. Growth is the change in what information a student has mastered and is the critical bellwether for student progress in their adaptive learning courses.

As stated in the paper, these four components should be no surprise, because educators know that this underlying pattern is fundamental to effective teaching and learning in all modalities, not just adaptive.

Conclusion

So what next? These components provide us with a solid basis on which to stand and to begin to make comparisons across disciplines and institutions. Instead of comparing individual metrics, we can now talk about student performance on constructs which we know are fundamental across learning. Indeed this is our next step. We should be sharing some of this work soon, so make sure to check back.

As always if you have any questions and comments, or want to get involved in any of the research then just let us know.

Reference for this paper

Dziuban, C., Howlin, C., Moskal, P., Johnson, C., Parker, L., & Campbell, M. (2018). Adaptive Learning: A Stabilizing Influence Across Disciplines and Universities. Online Learning, 22(3). doi: http://dx.doi.org/10.24059/olj.v22i3.1465

Principal Component Analysis - A Primer

2018-10-22T09:00:00+00:00

This is a resource post that gives a very brief overview of Principal Components Analysis (PCA). We create these resource posts to provide a basic introduction to some techniques and algorithms that we use in our research. Rather than clogging up the original article with lengthy explanations, we've put them into posts like this so that you can refer to them if needed. The idea is not to give all the details but just enough to understand why we used the approach and what benefits it gives us.

Principal Component Analysis is a dimension reduction technique. It takes a set of variables, looks for relationships between them, and tries to use those relationships to hopefully help us produce a smaller set of components that we can use in our analysis. In the following I’m going to give some basic detail on how it achieves this, and what these components are, using a simple example.

A Simple Example

Imagine we have a collection of some objects - it doesn’t matter what they are. We have a weighing scale and a measuring tape, and we can measure only two variables about each object - its weight and its length. In the following graph, I’ve plotted the weight versus length for our collection objects.

A sample set of data showing the relationship between the length and weight of some objects.

One thing we notice immediately is that they appear to be highly correlated - all the points fall approximately along a straight line. For the sample points, the correlation is 0.95. Since the two variables are so highly correlated, when we know the value of one variable, we can make an excellent guess at the value of the other one. So we don’t really have two different pieces of information about each object. If, for example, we know the weight of an object, we can get a reasonable estimate of its length form the graph (or from building a linear regression model). So actually measuring the length of the object only gives a small bit of extra information - the error in our estimate, which should be small given the two variables are so highly correlated. We could now decide to only use one of these variables in our analysis as including both would not tell us anything extra - we can translate anything we learn from one variable to the other.

This is all relatively straightforward in this simple example, but as we include more variables in our data we have to consider far more relationships between variables and sets of variables. This is where PCA comes in. It helps us achieve the same reduction in a more rigorous way. Let us continue to use our simple data set to see how PCA works.

Extracting the Components

Now that we have our data plotted imagine drawing a new set of axes on top of this data and measuring the distance from each point to the new set of axes, just like in the following graph. We end up with each object measured on these two new variables, which we could use in place of our original two variables. We haven’t lost any, but it would be challenging to interpret our analysis in real-world terms using these two new random axes.

The sample data can be measured against any set of new axes that we want.

PCA creates a new set of axes, but it doesn’t just draw any random set. It finds the best set subject to some given criteria. There is lots of detail in here that I’m glossing over but understanding this should be sufficient for the rest of what we do. These new axes are the new components we mentioned earlier. We have two variables in our sample data, so we end up with two components. If we had 13 variables, PCA would generate 13 components. So how do these new components help us? In the following graph, I’ve plotted our sample data points on the two components given to us by PCA. Notice how the data is still all spread out along a line but that line is now pretty much horizontal.

The sample data plotted using the new components.

Reducing the Dimensions

Let’s look at the variance of the data points on our original metrics and on our new components to see how things have changed. The variances of the length and weights of the objects are 2.89 and 2.92 respectively. We can think of the variance of a variable as how much information that variable contains. For example, if the variance is small, then all the points are close to the mean value, and if you know the mean, you have an excellent approximation for all the data points. If the variance is large, then merely knowing the mean is not a very accurate approximation of all the individual data points.

Taking the variance of our two original variables we can measure what percentage of the total variance each contains. The Length explains 49.7% of the total variance, and the Weight explains 50.3%. (I had already scaled the original variables so this was a simple calculation in this case.) Now if we measure the variance on our new components, we get 1.96 for Component 1 and 0.05 for Component 2. That means that Component 1 contains 97.5% of the variance. In other words, this single component captures 97.5% of the information obtained in the data. Concentrating the variance from across the original variables on to some smaller subset of components is one of the criteria of PCA. We get some components with a lot of the variance and some with very little. The components are ordered from highest to lowest variance so the first or principal component will always have the highest variance.

A step that we can now take is to drop the second component as it doesn’t contain much information. In more technical terms, we tend to keep all components that have an eigenvalue (variance) greater than 1. In our data, only the first component has a variance greater than 1, so we retain it and discard the other. We have now gone from working with two variables to working with only one component, simplifying any analysis we have to do. When we have large sets of correlated variables, this can help us streamline and reduce the dimensions (number of variables/columns) of the data. This is why PCA is known as a dimension reduction technique.

Interpreting the Components

However, how do we interpret our single new component? If we use it in our analysis, then how do we translate what we find in the real world? PCA produces something known as a pattern matrix. This tells us how the original variables are related to the components, and using it, we try to find an interpretation for the component in terms of how it refers to the original variables. We generally put some cutoff on the values in the pattern matrix and ignore any values whose absolute value is less than the cutoff. This approach is beneficial with larger datasets. Typical cutoff values of 0.3 or 0.4 are used. In our simple case, both values are above these thresholds. Our pattern matrix (below) tells us that Component 1, our principal component, is equally dependent (or loaded) on our two variables, length and weight.

	Component 1
Length	0.707
Weight	0.707

The pattern matrix for the sample data original variables and principal component.

A label we could give this new component is size. It is an equal mix of the two original variables which are highly correlated to each other. When we measured either one, we essentially measured both as they were so related. The hidden (or latent) aspect of the object that we were measuring was their size. We couldn’t measure the size directly; we could only measure aspects related to it, but PCA help us uncover this.

Hopefully this brief overview illustrates how PCA can help us, not only to reduce the dimensionality of our dataset, but also to find the underlying hidden or latent structure that can exist behind our data.

Boosting Self-Regulated Learning

2018-10-03T09:00:00+00:00

As an adaptive learning system, Realizeit provides tremendous flexibility for students, allowing them to learn in their own way, at their own pace, and at times that are most convenient for them. This kind of flexibility can be immensely beneficial to students who may live busy lives with diverse responsibilities that limit their opportunities for study. However, flexibility must be balanced with guidance so that students can achieve the best possible results. Students in the system currently receive guidance from instructors, and in some cases from interaction with their fellow students, but one of our latest internal projects is designed to enable Realizeit to automatically provide students with the kinds of learning supports that a tutor would provide at the times the students need feedback, while still allowing students to retain the freedom and control the system already provides. This project is part of our ongoing internal research to enhance and evolve the platfom, and will be part of a future release.

Self-Regulated Learning (SRL) was a natural concept to focus on in providing this support. SRL is an established concept in learning research, and it relates to the capacity of the student to support their own learning by deploying various techniques to increase understanding and retain information. Factors involved in SRL include cognitive, metacognitive, behavioural, motivational, and emotional aspects of learning. Everyone engages in SRL to some extent, so we’re all familiar to varying degrees with many of these techniques, such as goal-setting, time management, revision, self-testing, note-taking, help-seeking, along with many other behaviours that improve the student’s learning, and which fall under the umbrella of SRL.

Researchers studying SRL have long been interested in what separates successful learners from their unsuccessful counterparts. It has been demonstrated that learners who use SRL to a greater extent, and especially certain particularly beneficial SRL strategies, tend to achieve their goals more than those who use these behaviours to a lesser extent (as discussed by Zimmerman, 1990, among others). We’re interested in learning from these successful students by trying to figure out how we can encourage all learners to use these behaviours to enhance their own understanding and retention, and how we can facilitate the use of SRL within Realizeit. Modern researchers have shown that SRL can be successfully encouraged and supported, and that this does lead to improved learning outcomes (e.g. Bannert & Mengelkamp, 2013).

The Realizeit adaptive learning platform already offers many tools that support SRL behaviours, but without encouragement we may see students engaging in these behaviours only to the extent that they would naturally – thus maintaining the achievement gap between strong self-regulated learners and weak ones. Since we want to help all learners, we decided to investigate two mechanisms to encourage SRL use in the system. Realizeit is a natural platform for this project because, as an adaptive learning platform, it is already focused on raising the standard of all students, and it enables detailed observation of student learning, measurement of that learning, and intervention where needed by instructor or by the system. Researchers have found that delivering information on SRL can boost SRL behaviours, as well as learning outcomes. We are focused on two main ways of doing this: pre-course education (e.g. Hofer & Yu, 2009), and mid-course prompts (e.g. Bannert & Reimann, 2008; Lehmann, Hähnlein & Ifenthaler, 2014).

The first of these involves students completing a course, before they begin learning their core course material, on how to learn. This should explain how they can control and improve their learning process by deploying SRL techniques as they learn. The second involves monitoring student learning, engagement and behaviour, and delivering prompts to the student as they learn, suggesting that they deploy SRL techniques in their learning. The first approach has the advantage of limiting disruption to the student’s core learning, but the second allows the system to address learning deficits that become apparent in real-time.

We are planning to explore both pre-course and mid-course options to increase student use of SRL strategies. In a future post we will outline the exciting way we are working on measuring student SRL behaviour, and deploying encouraging interventions where the system estimates that student learning could be improved by using SRL techniques. In the meantime, any questions, feedback or suggestions on SRL are welcome!

References

Barry J. Zimmerman (1990) Self-Regulated Learning and Academic Achievement: An Overview, Educational Psychologist, 25:1, 3-17, DOI: 10.1207/s15326985ep2501_2. Article
Bannert, Maria & Mengelkamp, Christoph. (2013). Scaffolding Hypermedia Learning Through Metacognitive Prompts. International Handbook of Metacognition and Learning Technologies. 171-186. 10.1007/978-1-4419-5546-3_12. Article
Hofer, Barbara & L. Yu, Shirley. (2003). Teaching Self-Regulated Learning Through a “Learning to Learn” Course. Teaching of Psychology - TEACH PSYCHOL. 30. 30-33. 10.1207/S15328023TOP3001_05. Article
Bannert, Maria & Reimann, P. (2008). Design and Effects of Metacognitive Support for Hypermedia Learning. Proceedings - ICCE 2008: 16th International Conference on Computers in Education. 18. 969-970. 10.17471/2499-4324/287. Article
Lehmann, T., Hähnlein, I., Ifenthaler, D. (2014). Cognitive, metacognitive and motivational perspectives on preflection in self-regulated online learning.Computers in Human Behavior. Volume 32, March 2014, Pages 313-323. Article

Self-Reported Student Affect

2018-09-19T09:00:00+00:00

In a previous post, we talked about our State Framework, a feature which enables a student to provide a single data input, in real time, on an ad-hoc basis. In this post, we will be discussing how we are going to use this framework to measure student emotions. In later posts, we will follow up on how we tested what we have built, and on insights that we have found.

While we built the State Framework to be flexible and to allow it to capture any appropriate metric, our goal was always to use it to get some feedback from students on their emotional or affective state. We want to get feedback on a regular basis in real time without interrupting the learning process, and we built the State Framework to allow students to give us exactly that.

But why do we want to capture this? Just because a students’ engagement and outcomes metrics all look good does not mean they are happy with the experience or are confident in their learning. There are multiple studies which demonstrate that a relationship exists between a students’ affective state and their learning outcomes. Knowing what their state is, along with how it changes during the learning process, should allow us to improve our understanding of what is happening with the students, to do a better job of personalizing the learning experience, and to provide students with the right type of assistance at the right time. This is true for both instructors and the adaptive platform itself.

Emotions and affect are complex components of learning. There are many different emotions. Many of these overlap, interact with each other, and can also change rapidly. A person can feel several conflicting emotions at once. To begin with, we can simplify our task by restricting our scope to those that have an impact on learning. Pekrun, Goetz, and Titz (2002) summarize studies involving what they call academic emotions. They list 14 different emotions and examine these in a variety of contexts, including class-related, test-related, and learning-related emotions.

In most cases, the student emotions were measured using a questionnaire. As we want to measure student emotions on a regular basis, we could not use this approach as it would be much too disruptive. Students would spend more time answering the questionnaire than learning! Instead, we turned to the PAD Model of Mehrabian & Russell (e.g. Mehrabian, 1996), along with the work of Bradley & Lang (1994) who developed a graphical means to self-assess affective state. Using their work, which significantly simplifies the task of measuring affective state, we can make some further simplifications to suit our needs and fit within our State Framework model. As with any simplification, we do lose some accuracy, but we feel that our final model is an acceptable compromise between accuracy and ease of use.

PAD Model

The Pleasure-Arousal-Dominance (PAD) emotional state model is a psychological model developed by Albert Mehrabian and James A. Russell (Mehrabian & Russell, 1974a; Mehrabian & Russell, 1974b; Mehrabian, 1996), to describe and measure emotional states. PAD uses three numerical dimensions to represent emotions. These scales can be summarized as follows:

The Pleasure-Displeasure Scale measures how pleasant or unpleasant one feels about something.
The Arousal-Nonarousal Scale measures how energized one feels. It is not the intensity of the emotion.
The Dominance-Submissiveness Scale represents how controlling and dominant versus controlled or submissive one feels. For instance, while both fear and anger are unpleasant emotions, anger is a dominant emotion, while fear is a submissive emotion. All emotions can be mapped onto some point in this three-dimensional space. For example, Exuberance would be high on all 3 scales and Bored would be the opposite.

This vastly simplifies things for us. Rather than considering the wide range of possible emotions we just need to consider a reduced set of scales.

Manikin Based Scales

But how do we measure where a student falls on each of these three scales? Well, we ask them! Asking someone where they fall on the three scales in PAD model will only be accurate if they have an objective understanding of each scale. What value would you assign on the P-scale if you are feeling happy? Would everyone assign the same value to the same feeling? Here is where the work of Bradley and Lang comes into play. These authors break each scale into 5 points and use a figure they refer to as a manikin to suggest to the respondent the feeling that corresponds to that point on the scale (Lang, 1980; Bradley & Lang, 1994). Respondents then pick the manikin on each scale that best represents their current emotional or affective state.

We will follow this general approach, but with some simplifications. First, as others have done, we will concentrate on what are termed the “core affect” dimensions, dropping the dominance scale (e.g. Russell, 2003). We also reduce the number of points on each scale to two. This simplification results in two points on each scale or a choice between four possible positions. Instead of using Bradley and Lang’s manikins, we modernized them to something with which people would be a bit more familiar. We created an emoji for each possible position, which we felt represented the best intersection of the two scales at that point. We also associated a description with each emoji which we felt accurately reflected the position.

To give us feedback on their affective state, students need to pick just one of four possible positions. However, there are many unanswered questions. Is this too simplistic? Given the full range of emotions, if a student can only pick one of the four above, how do we know what other emotions we are capturing in this category? Would we not be better off with more categories? Did we choose the right faces and make them accurately reflect the different points on the two scales? Did we choose the correct descriptions?

We have a lot of questions to answer. Some we couldn’t address until we deployed the feature so that we could start to collect data and feedback. Others we could (and did) test up front - we will discuss these in upcoming posts.

Wrap Up

This model is our first step towards listening to the student voice, and not just measuring learning. We still don’t know what this data is going to tell us, or how the instructors, mentors, or the platform itself should use and interpret it. Answering the questions we have raised will be one of the ongoing research projects that we will collaborate on with some of our partner institutions. We are always interested in any feedback anyone has, so please send us a message if you have any advice, words of warning, or if you are interested in collaborating with us on this.

References

Bradley, M., & Lang, P. J. (1994, March). Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49-59.
Lang, P. J. (1980). Behavioral Treatment and Bio-behavioral Assessment: Computer Applications. Technology in mental health care delivery systems, (pp. 119-137).
Mehrabian, A. (1996, December). Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament. Current Psychology, 14(4), 261-292.
Mehrabian, A., & Russell, J.A. (1974a). An approach to environmental psychology. Cambridge, MA: M.I.T. Press.
Mehrabian, A., & Russell, J.A. (1974b). A verbal measure of information rate for studies in environmental psychology. Environment and Behavior, 6, 233-252.
Pekrun, R., Goetz, T., & Titz, W. (2002). Academic Emotions in Student’s Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research. Educational Psychologist, 37(2), 91-106.
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145-172. http://dx.doi.org/10.1037/0033-295X.110.1.145

Capturing the Student Voice

2018-09-10T10:00:00+00:00

If we genuinely want to make learning personalized, then we need to consider the student voice. Most systems today are excellent at capturing “hard data” – what did the student do, how long did the student spend doing it, what was the outcome? However, there is much more to the learning process than what is captured by these metrics. To get a richer and more complete picture of the student’s experience, we have to listen to them. How are they feeling? What are their opinions? What is their feedback on their learning? Getting answers to these types of questions can be particularly tricky in the online environment, where there is little or no face-to-face contact amoung the students or between the students and their instructors. We need to look at some way of capturing the student voice.

In this post, we will lay out an overview of the features we are working on to capture this data. In later posts, we will go into more detail on the research behind why we did things the way we did, what the data may tell us, and how the instructors and institution can make to most out of this data.

The Picture So Far

There are many aspects of the student voice and many ways in which we could approach capturing it. Discussion boards undoubtedly give one way of allowing students to express themselves, to communicate with each other and their instructors, and to ask and answer questions. Discussion boards are highly flexible, but this flexibility is itself a source of difficulty. The data is highly unstructured, and it is challenging to get a clear picture of what is happening, or what the students are saying, without some human intervention and interpretation.

The public nature of discussion boards can also be both an advantage and a disadvantage. You can reach a broad audience and have high levels of engagement but, on the flip side, students may feel reluctant to share their emotions or struggles in a public forum.

Prior to the September 2017 release of our Realizeit platform, we had worked with hard data; however the soft data that captures the more personal side of students has been something we’ve been thinking about for many years. In 2016 we started to put some of our ideas together and came up with two approaches that we are offering to institutions, instructors, and students to capture soft data.

The first approach is our State Framework, a lightweight feature to measure a single data point. The more extensive and comprehensive piece is our Data Framework, which is part of our roadmap for an upcoming release. It worth pointing out that both pieces are frameworks with a flexible structure that can be adapted to capture a range of possible data points and used in a variety of contexts. This also means that these are great research tools as we will be able to reuse them to capture alternative metrics in future research projects.

Not only is it important to consider what data you are capturing but also how you capture it. Both approaches we will use involve interrupting the learning process to some extent, so we have to be careful in how we design them. In the next section, I’ll give a bit more background on the State Framework.

State Framework

Our definition for the State Framework is a generalized feature to allow students to provide a single data input in real time on an ad-hoc basis. So, what does that mean? With our state framework, we wanted a simple means of capturing some metric that can change on a regular basis, in a short time period - like students emotions or their confidence level. We wanted a means for the students to be able to let us know how these things are changing in real-time – while we disrupt their learning as little as possible.

The State Framework displays a small popup to the students and allows them to quickly select one of several options to provide an answer to a question as quickly as possible. This feature is for use by the students any time they want; a prompt can be set to remind them at regular intervals to update it. This input is not tied to one area of the system and can be updated whenever students see fit, such as during a lesson. Because of the hard data we capture in the background, we know what the students were doing just before they updated, so we have context for why the students changed their State Framework input values.

In the screenshot above, we are using the State Framework to capture student affective state or emotional state. This is our internal goal for the State Framework and is included in the system by default. In follow-up posts, we will discuss our motivation for this, the research behind the emotions we chose, and what we learn from the data.

Other uses for the State Framework could include measuring students’ confidence in their mastery of the learning they have completed.

Alternatively, we could ask them to predict how their level of mastery will change over the remainder of the course.

The framework is fully customizable by the institution or the instructor to capture a new metric, or to make adjustments to our default framework for capturing student affective state.

We provide the students’ most up to date values to their instructors and mentors so that they get additional information on what is happening with the student in real time, beyond their engagement and attainment metrics.

Data Framework

While the State Framework provides a quick glimpse of a single metric, the Data Framework is designed to go beyond this and capture a more extensive range of possible data values. It is built to capture more detailed feedback from the students and can be customized to prompt for feedback based on specific triggers in the system. Given that it can capture a broader range of more detailed information, it is potentially more disruptive to the learning process. Possible uses include delivering surveys or asking for feedback on learning content. As mentioned this feature is part of our roadmap for a future release, and we’ll be sure to revisit this here with more detail once released.

Wrap up

We hope that gives you a good idea of our progress on the tools we will use in capturing the student voice, the soft data isn’t traditionally measured in systems, and can only be obtained by listening to the student. This is a work in progress and will continue to evolve as we learn. From the research point of view, the most exciting parts come next: what data can we capture using these tools, what can we learn about the students, how do we make effective use of the data, and how can we use it to impact on learning positively!

If you are interested in using one of the frameworks as part of a research project or are interested in working with our partner institutions and us on understanding the data we gather then please get in touch.

A Q&A on the Goals of Research at Realizeit Labs

2018-09-10T09:30:00+00:00

Hi all, below is a re-post of a Q&A session we put together on the goals of research at Realizeit Labs. We hope you find it interesting and that it gives you a bit more information on how we approach and carry out the work that we do. Enjoy!

Introducing Realizeit Labs: An Inside Look at Our Digital Learning Research

Research and data are two of the primary vehicles that propel Realizeit, and Dr. Colm Howlin is the driver. In his role as Realizeit’s principal researcher, Colm’s work helps drive the evolution and direction of Realizeit’s intelligent learning engine and system. He also spearheads the collaborative research partnerships that help provide a broader understanding of the impact and opportunities of digital learning with Realizeit. In this interview, Colm discusses the critical role research plays at Realizeit, his approach to partnering with institutions, and Realizeit Labs, the new initiative the company is launching to provide a behind-the-scenes look at the data, insights and opportunities these efforts uncover.

Why is research such a core part of who Realizeit is as a company and system?

Everything we do is based on data and evidence. If want to make a positive difference to students’ success, and if we want to continue to improve our platform, we need to understand and measure what kind of impact we’re having. We need to be able to work with experts in the field and take what we learn from them back in house to make improvements.

There are also plenty of people working on topics around the world that are not directly connected to us that we can learn from. There is work going on in other fields, in other institutions, so you need to keep an eye on the current edge of research to know what’s happening. You can’t claim to be on the cutting edge if you don’t know where the edge is.

We just published a journal paper with the University of Central Florida and Colorado Technical University comparing the underlying structure of adaptive learning across institutions. Despite the large differences between the two institutions in terms of organization and student demographics, the underlying structure of adaptive learning is the same. We will be presenting this work at several conferences later this year, including the EDUCAUSE Annual Conference and OLC Accelerate.

Another example is the work we’re doing with Charles Sturt University, which uses a self-directed learning model with its Civil Engineering students. In a paper we will be presenting at the 46^th SEFI Annual Conference, we will show how students “binge” on topics – just like they would on Netflix shows.

This collaborative approach to research is what really enables us to start making progress. Our partners at these institutions are the people on the ground interacting with the students and faculty and seeing how they’re using the digital learning system. I have the data expertise and the understanding of how the system functions to detail the behavior they’re seeing and find the insights. It is the coming together of these two parts that really allows us to develop a better understanding of how to use digital and adaptive learning effectively – and helps institutions understand how they can evolve their processes as well.

What is Realizeit Labs? How does it fit into this collaborative research?

Realizeit Labs will provide a window into all the ongoing research we do behind and about Realizeit. We’re launching the Realizeit Labs website to share more about the research we’re doing. We often find insights or generate ideas that are of interest to others outside the company. This will give us a place to share that kind of information both formally, in terms of papers and articles we publish, as well as informally through blog posts.

Part of our goal is to provide some insight and understanding for educators into why the system functions the way it does, why we’ve built it the way we have and how to best leverage those capabilities. In a way, it can remove this perception people sometimes have of a digital learning system like Realizeit as a sort of “black box” – Realizeit Labs will give insights into what’s happening inside.

At the same time, we want to foster the collaborative side of our research work and draw in people who are interested in partnering with us. Instructors and institutions can work with us to learn more about their students, their courses and how they can continue to evolve teaching and learning to drive better experiences and outcomes.

We’ll also share our perspectives and opinions that arise from the work we’re doing or what we observe others doing. And if there is something people to learn more about, we are eager to hear that as well. Hopefully it will be a place that starts a meaningful dialogue.

How does Realizeit’s research focus serve the overall goal of better understanding – and better serving – the student?

For everything we do, the goal is always to better serve the student and the faculty. At its core, Realizeit is a learning system. It isn’t simply to assess what students do and don’t know; it’s to teach and to do what we can to help them learn. Because of that, everything we do with research is to better understand how the students use the system, what impact it is having, and how we can make it better for them and their instructors. That could be improving the interactions between the students and the system, providing instructors better feedback so they can more effectively intervene with students, giving administrators the right feedback, they need improve courses and overall learning, and so on.

One recent example is the state framework, which we introduced so students can have more of a voice in sharing how they’re doing with their instructor beyond just seeing data on their activity and progress. Particularly in online settings, instructors often won’t know how students are feeling about the work or how they’re doing. This lets students give a little extra feedback.

Realizeit generates a significant amount of data on student learning that helps inform this research. How do you view the team’s responsibility regarding that data?

From a privacy perspective, we’re strictly adhering to all the legal requirements regarding the handling of student data, but it’s more than that. Instructors and institutions trust us with this data, and we only want to use it for good. We don’t own the data; it is owned by the institutions. We have a responsibility to be very thoughtful about how this data is used, even when it is anonymized and used in the course of research. When we’re looking at the data, it’s always with an eye on whether the application of that data in research is in the best interest of the student and the institution.

What are some of the initial topics readers can expect to see on Realizeit Labs?

We’re going to talk about some of the recent projects that we’ve worked on. So, for example, we’ll be looking at student behaviors and prototypes. We’ll talk about the concepts behind some of our newer features, such as capturing the student voice and what’s happening with the student through what we call soft data. And we’ll be sharing our published research articles and upcoming conference presentations.

How can clients interested in partnering on research get started?

We’re looking for institutions who are already partners with us as well as external researchers interested in using the data. If you are a client institution, you can come with questions you have, and we can set up an analytics project that can evolve into a research project, where we explore the data and pull out the insights. Some clients already have research questions they want to answer, so we can start there as well.

Is Realizeit Labs for Realizeit customers only?

Realizeit Labs is for anyone who’s interested in learning about the research behind digital learning technology. The site is open to the public at lab.realizeitlearning.com. We’re planning to post new content as regularly as possible, and ideally, we’ll ramp it up over time. I hope those customers who are interested – as well as outside researchers and anyone else – visits us regularly to see what’s new.

Welcome to Realizeit Labs

2018-09-10T09:00:00+00:00

Welcome to Realizeit Labs, the website and blog of the Realizeit Research and Development team. Our mission is to build, extend and evolve the capabilities of Realizeit, our learning platform. We plan to use this space to share with you some of what we are working on, both internally and in collaboration with our partners.

So why are we doing this? There are several reasons.

First, while not a new concept, adaptive learning has new potential and possibilities given the technology and data that is available today. Back in 1963, John Carroll recognized that if educational time is held constant, student learning will be the variable. He suggested altering the paradigm so that learning becomes the constant, making time the variable - the basis of adaptive learning. We have learned much since Carroll, but given the new pace and direction of today’s implementation of adaptive learning, we are just at the start, and much still needs to be figured out. We want to contribute to the conversation and the body of knowledge, which is why we partner with institutions on research projects and share what we learn. We have developed close research relationships with several of our partner institutions including the University of Central Florida and Colorado Technical University. You can read about our collaborative work and see some of our findings in our Educause Review article, An Adaptive Learning Partnership.

The second reason for Realizeit Labs is to foster collaboration between institutions to help them, and us, get the most out of their data and to help students learn and succeed. We want to create a space where we can share what our partners are working on, so that they can learn from each other. A fantastic organic community of cooperation has grown among the users of our platform; we want to support this in any way we can.

So, what can you expect to see from us? There are several areas to the site in addition to this blog. On the Papers page, we will list any published papers that reference Realizeit. The papers will be a mix of those generated from collaborations in which we are involved, as well as those of independent research projects. The Meetings page lists upcoming sessions or meetings at conferences involving someone from the R&D team. If you want to catch us in person, then you will know where we will be. We will also keep a list of any past sessions so you can see what we’ve been doing. We plan to eventually have a Projects section that will share some more specific details and artifacts from internal and collaborative projects. We don’t just want to share our papers, but some of the workings - when possible/

Some of the topics you will soon see include our collaborative work on student behaviors with UCF and CTU, as well as some exciting collaborative work with Charles Sturt University related to their self-directed learning model underpinned by their Topic Tree. We will also share some internal projects such as our efforts to capture “soft” data from students (including their affective state), and how we plan to improve individual self-regulated learning by integrating feedback directly into the learning process.

We hope this gives you an understanding of what we plan to do and that you’ll come back and check up on our progress!

Realizeit Labs

Adaptive Analytics: CTU - a Time Critical Setting for Predictive Modeling

The Goal

The Data

The Model

Enough Data

Variable Importance

Impact

Next Up

Adaptive Analytics: Predictive Modeling Based on Teachable Skills at UCF

The Goal

College Algebra at UCF

The Data

The Model

Next Up

Supply Chain Management and Adaptive Learning

The Supply Chain

Parallel Elements in Adaptive Learning

Two Case Studies

Amazon Fulfillment Center

Adaptive Learning

Finally

References

Adaptive Learning: A Stabilizing Influence

An Adaptive Partnership

A Stabilizing Influence

The Courses and the Data

The Analysis

The Components

Conclusion

Reference for this paper

Principal Component Analysis - A Primer

A Simple Example

Extracting the Components

Reducing the Dimensions

Interpreting the Components

Boosting Self-Regulated Learning

References

Self-Reported Student Affect

PAD Model

Manikin Based Scales

Wrap Up

References

Capturing the Student Voice

The Picture So Far

State Framework

Data Framework

Wrap up

A Q&A on the Goals of Research at Realizeit Labs

Introducing Realizeit Labs: An Inside Look at Our Digital Learning Research

Why is research such a core part of who Realizeit is as a company and system?

Can you share some examples of research topics your team is currently exploring with institutions?

What is Realizeit Labs? How does it fit into this collaborative research?

How does Realizeit’s research focus serve the overall goal of better understanding – and better serving – the student?

Realizeit generates a significant amount of data on student learning that helps inform this research. How do you view the team’s responsibility regarding that data?

What are some of the initial topics readers can expect to see on Realizeit Labs?

How can clients interested in partnering on research get started?

Is Realizeit Labs for Realizeit customers only?

Welcome to Realizeit Labs