Indiana colleges use analytics to try to increase retention

The University of Indianapolis is leading a coalition of six Indiana colleges in an $11 million grant-funded program that’s using data analytics to help boost graduation rates.

The tool, created by Indianapolis-based consulting firm Resultant, analyzes hundreds of data points to generate early warning signs about which students might be at risk of dropping out—so school officials can help them get back on track.

“Because, historically, our first real indication [of a struggling student] is midterm grades for freshmen, and that’s often too late,” said Kate Coffman, manager of the grant project and UIndy’s associate vice president for university marketing and enrollment strategy. “And so the idea of the project was, if we have that information, and earlier, could we put them on a path to success?”

The Indianapolis-based Lilly Endowment Inc. has provided a three-year, $9.8 million grant for the project. Of that, UIndy received $3.06 million. An additional $1.35 million went to each of the other five partner schools: Martin University in Indianapolis, Anderson University in Anderson, St. Mary-of-the-Woods College near Terre Haute, Indiana Institute of Technology in Fort Wayne, and the University of Southern Indiana in Evansville. UIndy also applied $1.4 million the year before toward developing its data analytics tool; that money was the lion’s share of a 2020 Lilly Endowment grant for recruitment and retention.

“Data is complex, and we know that we can’t tell a person’s full story by their data points, because we’re human beings,” said Curt Merlau, director of Resultant’s education practice. “But we can certainly start to gather a lot of insights that lead us to have conversations with people or give them the supports to really then understand the full picture.”

The big picture

The project is part of Lilly Endowment’s larger effort called Charting the Future for Indiana’s Colleges and Universities, a $108 million program that launched in October 2019. All the state’s 38 colleges and universities received Phase I and Phase II grants to help them deal with an array of challenges, including increasing student retention, improving degree offerings in certain key fields and strengthening digital learning. The $9.8 million grant to UIndy and its partners was among 12 grants Lilly Endowment made in March 2021 for the final phase of the program.

In addition to funding development of the predictive analytics tool, the Lilly Endowment grant supports other changes the schools are making to help retain more students.

At UIndy, that includes revamping its orientation program, changing its academic-advising process and adding staff specifically to support retention efforts.

According to the National Center for Education Statistics, only 60% of U.S. students who entered college in 2014 had graduated by 2020. In Indiana, the average was 63%. Among the six schools participating in the Lilly-Endowment-funded predictive analytics program, that six-year graduation rate for incoming 2014 freshmen ranged from 64% at Anderson University to 10% at Martin University. (UIndy’s rate was 62%.)

“I think public and private institutions are always concerned about enrollment and retention,” said Kyle Jones, an associate professor of library, information and data science at Indiana University’s Luddy School of Informatics, Computing and Engineering at IUPUI. “That’s the financial lifeblood of the institution, right? Even for a public institution that receives some state funding. So there’s always a desire to find ways to improve those two metrics.”

Jones has expertise in ethical and policy issues related to data mining, analytics and educational technologies in higher education. He is not associated with the effort UIndy and its partner schools are engaged in.

UIndy was the first to implement the unnamed data analytics tool, starting with a pilot with 300 incoming students—about a third of that year’s freshman class—in the fall of 2021. The school now tracks all its freshman and sophomore students with the tool.

Anderson University and St. Mary-of-the-Woods came on board next. Indiana Tech and USI are in the testing and development phase, while Martin is in the discovery phase and will be the final school to adopt the tool.

Coffman said UIndy deliberately chose partner schools that do not directly compete with one another for students and that represent different areas of the state.

Nuts and bolts

The data points UIndy is analyzing—which include everything from a student’s current course load to family finances to how often he or she eats at a campus dining hall—are factors UIndy already had access to, the school said.

But the predictive analytics tool brings these disparate data points together in a way that’s more useful, said Josh Morrison, UIndy’s director of academic retention programs. “The data existed already. It’s just putting it together in a way that makes it more actionable to understand what’s happening, what sort of engagement’s happening with those students.”

The predictive analytics tool generates two scores for each student, each on a scale of 0-100. One relates to the likelihood the student will return for the next semester; the other relates to the likelihood the student will return the following year.

Morrison and his team focus their outreach on students with scores of 35 or lower or with scores that have dropped significantly. During those conversations with students, Morrison might learn that a student is struggling with a particular class, needs help with study skills or time management, or is facing financial challenges.

“This [data analytics tool] gives us a much more targeted way of approaching those students to understand what their concerns may be, and connecting those resources as soon as possible, so we’re dealing with a concern at a low level before it becomes a crisis.”

One insight UIndy has already gained from the tool: A grade as high as a B- can be an early indicator of struggle.

Before the predictive analytics tool was available, Morrison said, UIndy generally used a C- as the threshold that would prompt a school official to check in with a student. “Prior to this new information, we probably wouldn’t have remarked on the B- at all … when that B- is actually a bit of a warning sign.”

Though this is only UIndy’s third year using the tool, the school says it’s already seeing results.

A common metric that colleges use to measure student retention is the percentage of freshmen who return for their second year.

The fall 2022 semester was the first year UIndy used the analytics tool on its entire freshman class. The school said 68% of its students who started as freshmen that fall returned for this fall’s semester—a 10% increase over the previous year’s return rate.

“That’s a significant number,” Morrison said.

Merlau, the Resultant director who helped develop the analytics tool, said the technology “does not replace the invaluable insights of the educational leaders and of the student support specialists who come alongside those students. It, instead, just helps really kind of pinpoint and focus on the students who may have otherwise kind of flown under the radar and not been seen or made their decision [to leave school] known until it was already done and they were gone.”

Building the tool

The data analytics tool, which Resultant built specifically for UIndy and its partner schools, looks at 400 current and historic data points to predict the likelihood of attrition.

Those data points vary from school to school, Merlau said, but in general, they analyze five areas that affect student success: belonging and integration, academics, finances, demographics, and the college itself.

In belonging and integration, measurements include how connected students feel with their peers and whether students view their college as a good fit for their academic goals—admittedly abstract factors, but ones that can be measured through surveys, Merlau said.

The academic area measures factors like grades, major and course load. The financial area includes information pulled from financial aid paperwork but also looks at a student’s financial literacy and beliefs about debt.

Demographic data includes the student’s hometown, whether the student attends school part time or full time and whether he or she is working while attending school.

Notably, Merlau said, demographic data does not include a student’s race or gender—factors that might inadvertently train the technology to perpetuate long-standing inequities and biases. “We were very cognizant of that.”

The college’s data includes its enrollment and location, its academic rigor, the support programs it offers and how often students use them.

The tool is built around a series of decision trees, which Merlau described as a team of expert jigsaw-puzzle solvers working together, each solver looking for the puzzle pieces the previous solver missed.

“So it’s like having a team of decision trees working together to understand these really complex patterns that influence attrition,” Merlau said. “Because it’s not any one factor or maybe necessarily two factors. It’s a complex array of factors that have different weighted levels of influence.”

Jones, the Luddy School professor, said colleges have analyzed student data for decades, but it’s been only since 2010 or so that technologies around this topic started to come into use. “The difference now is that the computational models and the technologies are more advanced, and also the greater access to individual student data and information now allows for more of a granular approach to those types of analytical processes.”

Jones said such tools can be useful if schools use them to identify and support students who need assistance. But they can be problematic if they are used in a heavy-handed way—for example, to block a student from taking a particular class in which a computer model expects the student to struggle. That second approach takes away a student’s autonomy, he said.

“In higher ed, we want students to pursue their interests, whether those are personal or professional, and we want to support them on that pathway,” Jones said. “If somebody is predicted to fail or not do well in something, that doesn’t mean that they can’t overcome that problem or that hurdle.”•