Career and technical education is often lauded for its ability to make learning relevant and give students a jump-start on college and career. But how do you measure its success?
In a series of meetings last school year, CTE administrators in Arizona grappled with how to grade the state’s CTE programs. Required by state law, the letter grades were intended to arm parents with the information they would need to find quality CTE programs for their child.
Considerable debate ensued over which criteria – graduation, CTE program completion, placement or industry certification rates – should carry the most weight in determining grades. There was disagreement on whether grades should be given at a district or school level, prompting legislation in January to require that each campus providing CTE be given its own letter grade.
More debate and lobbying ensued, resulting in a strike-everything amendment that would eliminate the requirement for letter grades altogether, but the bill never made it out of committee.
“Without outcomes, CTE is just another elective,” said Dr. Sally Downey, superintendent of the East Valley Institute of Technology, which provides over 40 CTE programs for high school students and adults.
“CTE outcomes shouldn’t be difficult to measure,” Downey said. “Did the student complete their program and earn an industry credential or license? Did they get a job related to their CTE program? Did they go to college or join the military? All of those things are measureable. But the devil is always in the details, especially when those details involve taxpayer dollars and politics.”
The study by Sapna Gupta identified a number of challenges, including the absence of data to measure outcomes and a state system that financially incentivizes public schools “to maximize the number of students in seats – regardless of whether students would be better served by engaging in off-campus work-based internships or by taking a half-day class in a (career and technical education) central campus.
This points to the urgent need to examine how the state allocates education funds and the necessity of a statewide longitudinal data system that will measure student performance and outcomes from preschool to entering the workforce.”
In an interview for this article, Gupta expressed frustration that while individual schools have their own data systems, there still is not one system for all schools in Arizona. She pointed to Arizona’s current Achieve 60 AZ initiative, led by Gov. Doug Ducey’s office, to ensure that 60 percent of Arizonans 25 and older have a certificate or college degree by 2030.
“How do you measure all this? If you can’t measure it, you can’t fix it,” Gupta said.
The dearth of data isn’t a problem that’s unique to Arizona. In a 2017 report, The Brookings Institute noted that the past 10 years have brought a renewed interest nationwide in CTE.
Unfortunately, research on CTE has not kept pace with policy interest, according to the report.
Different models, definitions. Same goals?
In Arizona, secondary CTE is provided by 14 Career Technical Education Districts or CTEDs, either at central campuses or in “satellite” programs at high schools in the CTED’s member school districts. CTE can also be offered at high schools in districts that are not members of CTEDs.
But CTED programs – central campus and satellite – receive additional state funding to operate, while non-CTED programs do not. Downey maintains the extra tax dollars obligate EVIT to provide added value in its CTE programs.
“We need to do everything we can to ensure that students complete their program and that every student leaves EVIT not only with the skills to do the job, but with a portable industry certification that tells employers they are ready to do the job,” Downey said. “We are not where we should be yet in making this happen, but we work toward it every day. Our students deserve no less. It’s the right thing to do.”
But not all CTE programs within the EVIT CTED share the same goal. In some districts, CTE programs have been serving more as electives than a formal pathway to certification and career, making it difficult to compare outcomes.
Joe O’Reilly, former executive director of research for Mesa Public Schools, Arizona’s largest school district and one of 10 districts in the EVIT CTED, said the expected outcome also depends on the program.
“In CTE in high school, for some, success is getting into this career … but for many CTE programs you’re not getting ‘career-ready’ as much as you are getting ready to be career-ready because you will need additional training and education to be successful,” said O’Reilly, who left Mesa last spring to become the director of the Decision Center for Educational Excellence at Arizona State University.
Some CTE administrators in Arizona maintain a program is successful if it introduces students to career possibilities and teaches them the so-called soft skills that employers value, such as communication skills and team work, even if they never work in the industry in which they studied. Educators in some other states share similar views.
Patrick Biggerstaff, director of career and technical education and adult education at Area 31 Career Programs in Indianapolis said, “I consider schools’ programs to be successful if they help students to develop skills and understandings that are relevant to their future lives.”
“By connecting with student interests, successful CTE programs prepare students for their next step after graduation and motivate them to continue learning,” Biggerstaff said. “Whether or not CTE graduates decide to pursue a career in their area of study, I have no doubt that they develop transferable skills that will benefit them as they move into adulthood.”
In North Carolina, Kimberly MacDonald, senior analyst for state and federal reporting in CTE at the state Department of Public Instruction, said her state acknowledges that many industry-recognized skills are transferrable skills that can apply to other industries. “North Carolina CTE encourages multiple entry points and exit points throughout the student’s pathway … If a student masters the skills and moves his/her pathway to an applicable industry, we consider the program to be successful,” she said.
North Carolina teachers track placement results (students going on to college, jobs or military related to their CTE program) of former students who were CTE concentrators – defined by North Carolina as secondary students who earn four or more credits in a single CTE cluster, one of which is in a secondary-level course, MacDonald said. The results are then validated by instructional management coordinators at each local education agency.
Arizona teachers also track their placement results, which are sent to the state Department of Education, and based on Arizona’s definition of concentrator – a secondary student who has transcripted two or more Carnegie Units/credits in a state designated sequence in an approved Career and Technical Education program.
That definition, Downey said, is a better fit for the satellite CTE programs, which are usually about 50 minutes long each day, than for central campuses like EVIT, where students receive 2 ½ to four hours of training daily.
Under Arizona’s definition, EVIT students become concentrators before they are even finished with the first year of a two-year program. So even if they drop the program before the end of that first year, the state counts them in EVIT’s results.
That, Downey said, results in a skewed measurement of the program’s success and holds EVIT accountable for students who drop their EVIT program after the first semester because they are lacking credits to meet graduation requirements at their high school.
“The data we provide to parents and the public shows results for EVIT program completers,” Downey said. “We want parents to know the result they can expect if their son or daughter completes their program, graduates from high school and earns a credential – not what to expect if they only finish part of one year.”
CTE programs sometimes face credibility issues because so much of the data that does exist is self-reported.
Every year, EVIT teachers and staff spend countless hours tracking down former students to find out if they are employed, in college or if they enlisted in the military. This data is then submitted to the Arizona Department of Education for the state to use in its performance measures report as required by federal Perkins funds.
Some data, such as post-secondary enrollment, can be obtained from a neutral third-party source, such as the National Clearinghouse, but much of it can only be gathered by asking questions of the former student to determine if their placement is indeed related to their CTE skills.
Also, while Arizona law requires CTED-funded CTE programs to lead to an industry credential, there is no state report that details the number of industry credentials earned by Arizona’s CTE students.
In her “On the Rise” report, Gupta recommended that Arizona “consider measuring and incentivizing the attainment of industry-recognized credentials …This could motivate more students to stay in school, provide a measureable outcome and send a strong message to existing employers and those considering locating to Arizona that the state is serious about investing in its workforce.”
She noted that Kansas passed legislation in 2012 that provides monetary incentives for high schools to increase the number of students who earn a credential in a high need industry such as manufacturing. Similar legislation was proposed in Arizona in January.
Meanwhile, in North Carolina, the importance of industry credentialing as an addition to statewide assessments has been a priority for many years. Industry credentials make CTE students more marketable, give employers a reliable predictor of a CTE student’s success, and elevate the image of CTE, according to a briefing paper by Dr. Daniel R. Smith, CTE section chief for the state’s Department of Public Instruction.
He wrote: “Decades after vocational education became CTE many people still think of it as cooking and sewing, shop class, or where the students who are not academically focused go. However, NC CTE programs offer worthy credentialing opportunities that document learned skills by students and have begun to alter those negative, erroneous, and outdated sentiments.”
North Carolina’s credentialing data comes from third-party vendors as well as self-reports by schools.
“At the state department, we collect credentials from the authentic source (the third party vendors) for the credentials where we have agreements with the vendor to receive the direct data feed. Currently we receive direct feeds from Project Lead the Way (PTLW), National Academy Foundation (NAF), and Certiport,” MacDonald said. “The rest of our credentials are collected by designees, called Instructional Management Coordinators, at each local education agency as self-reports of the data.”
According to North Carolina’s 2016-17 Credentialing Data report, the number of credentials earned by CTE students in that state has increased from 24,782 in 2010-11, the first year that data was collected, to 160,224 in 2016-17.
Downey has long advocated for a similar system in Arizona. She recalled that when she worked in CTE in Oklahoma in the 1990s, the state’s CTE department collected credentialing data for its schools.
“That should be a priority in Arizona too,” Downey said. “We need to require programs to test students for industry certifications where they exist – and to track how we’re doing in preparing students to earn those certifications.”
Downey has not been satisfied with certification results for some EVIT programs, which the school tracks on its own. So, starting this year, each EVIT instructor was paid a $25 incentive stipend for each student who earned a certification.
By itself, data doesn’t give a complete picture of a school’s success in CTE, but it is a vital starting point, Downey said. “We also need to know if industry is satisfied with the skilled workers that CTE programs are turning out. Is industry getting a return on its investment in career and technical education?”
The input of parents is important too. Every year, EVIT surveys its parents to see if they are satisfied with the school’s programs. Last spring, 1,108 parents participated in the survey with 97 percent agreeing that their child understands the expectations for becoming a CTE program completer.
In addition, 93 percent said EVIT provides a high quality education that encourages their child to learn and 94 percent said they were satisfied overall with the quality of CTE at EVIT.
Students were surveyed too, with 96 percent of the 2,674 surveyed saying they would recommend their EVIT program to a friend.
“Those are the results I care about the most,” Downey said. “When 94 percent of parents are satisfied and 96 percent of your students love your school so much they would urge a friend to come here – you know you’re doing something right.”
CeCe Todd is the public information officer for the East Valley Institute of Technology.