Retaining tempo with AI startups “operating round with scissors” — to make use of Steve Santana’s metaphor — might be unnerving for established enterprises. However the CIO of Princeton-based Training Testing Service (ETS) believes corporations can’t stand nonetheless.
On this installment of the IT Leaders Quick-5 — InformationWeek’s column for IT professionals to achieve peer insights — Santana acknowledges that AI presents main safety and information administration issues for enterprises. Nonetheless, he mentioned he’s a agency believer that when used inside correct guardrails on a basis of buyer belief, AI will probably be essential to scaling ETS and bettering buyer expertise.
He additionally forecasts that AI will basically reshape how enterprises strategy cybersecurity and the dimensions of their safety groups.
ETS supplies tutorial evaluation exams, such because the GRE. The group conducts 50 million tutorial exams a 12 months, and Santana’s aim is to make use of AI to scale take a look at distribution to exceed 100 million exams a 12 months.
Previous to his position at ETS, Santana was senior vice chairman and CTO of Pearson, held director-level positions at Ciena and Nortel, and was a supervisor at Apple.
This column has been edited for readability and area.
Steve Santana, CIO and head of AI, ETS
The Resolution That Mattered
What resolution — technical or organizational — made the largest distinction just lately, and why?
For us, it is the speedy adoption of AI. We now have been utilizing machine studying fashions for 20 years. All of our core scoring engines, the e-rater and SpeechRater functions, we constructed primarily based off our information. They rating simply in addition to the people and are very dependable, however they’re costly to keep up since you’re continually updating these fashions and cleansing the info.
Giant language fashions are a lot inexpensive to combine with, so we wish to transfer in that route — not simply on scoring, but additionally on merchandise era, that are the take a look at questions.
A big language mannequin can actually make it easier to in operating the take a look at by taking the test-taker via a course of. As an alternative of simply multiple-choice questions, I can begin having a dialogue with you, after which I can have brokers inside that mannequin work together with you within the ways in which any person would in a real-world scenario. We will course of that you simply perceive the strategy, the method, the ideas, the competencies behind it.
We needed to make use of this expertise, and we noticed an important alternative — but additionally an amazing quantity of danger . Like at each firm, there have been lots of their safety points. We do not know the place this information got here from. How can we handle and work with it?
I in a short time simply mentioned that we do not have a selection — now we have to do that. There’s so many startups which are simply operating round with scissors, that means they are going to do that with out us, and they will begin breaking belief with individuals. We must always undertake it, and we should always present individuals the correct method to go about doing it.
We take the 78 years of experience that ETS has and the a long time of information, and use that as a mechanism to leverage that belief. We already had excessive belief in our assessments. How can we construct it in a method that folks will belief what we do with AI? We do not use any information that comes from the massive language fashions. It’s all the time utilizing reference information that’s first-party information that we perceive and make the most of.
A couple of 12 months and a half in the past, we determined to centralize this effort again underneath the expertise group. We now have a seven-point plan of how we’re implementing this expertise, and lots of issues are literally already in manufacturing as we speak.
We took the learnings of what everybody was doing via experimentation, after which we thought-about how we implement it at scale. A very powerful factor that we had was belief — our clients trusted us. We have to preserve that as we deploy AI in these new exams and fashions, and they will assist us drastically cut back prices.
The Arduous-Gained Lesson
What didn’t go as deliberate just lately — and what did it drive you to rethink?
Just a few years in the past, at a unique firm, we had been pushing the enterprise to deploy a brand new CRM that was core to their enterprise. We pushed them fairly laborious on making the choice to maneuver ahead as a result of they wasted hundreds of thousands of {dollars} on failed initiatives as a result of they did not transfer [quickly enough].
However, we had some decent-sized issues with the CRM once we rolled it out. Most of them had been issues that had been missed — enterprise necessities that we did not collect. If we had taken some extra time and had not pushed the enterprise so laborious on that, I believe it will have gone smoother. Once I pushed them, they weren’t on my aspect. And when issues went fallacious, they had been nonetheless not on my aspect.
Shifting ahead, I spend much more time with the enterprise staff, and I attempt to make it possible for we’re on the identical aspect. I used to be focusing an excessive amount of on the necessities of the expertise and never sufficient time specializing in the enterprise change administration aspect of issues.
The Expertise Commerce-Off
The place are you investing in expertise proper now — and what are you consciously not investing in?
The utility participant is much more vital than they was. Lots of people are very centered on being robust in a selected area. A utility participant is somebody who is aware of a little bit about lots of issues and might use agentic AI to spherical out the remainder of their supply capabilities.
When you have a really robust working data of a lot of domains, AI might help offer you depth in these areas that perhaps you did not have entry to earlier than. You may have engineers that wish to do enterprise analyst work.
In the event you’re an engineer and you have to convert enterprise necessities into consumer tales, then you possibly can have the AI try this. AI might help pull all of that collectively for you and write your tales. You’ve got simply now expanded your potential to ship extra and your worth to the group has exponentially grown.
I am not investing in one-dimensional participant database directors [DBAs] that simply know every little thing about database efficiency administration — these issues are going away. We have inspired our DBAs at ETS to be taught extra items of the enterprise and construct out their [roles as] utility participant throughout these issues. However we do not hunt down particular particular person area data.
The Exterior Sign
What latest exterior growth is probably to vary how your group operates, even not directly?
The straightforward reply is agentic AI and the way it will influence our safety. Agentic AI can actually assist determine vulnerabilities and enhance monitoring, nevertheless it additionally massively will increase the danger profile. We’re a decent-sized group, so now we have an enormous safety staff, however there are a lot of small and medium companies that do not have that safety administration.
That retains me up at evening — I used to be frightened earlier than Anthropic Mythos got here out. I am glad that Anthropic is offering organizations with the tooling to attempt to do away with their vulnerabilities. However when you have software program that is greater than 5 years previous, you most likely nonetheless have a lot of vulnerabilities that you simply’re having to handle. I fear about organizations that simply cannot scale as much as deal with that.
However it’s additionally an enormous alternative. I believe safety operation facilities are going to be a fraction of the dimensions they’re as we speak. In the following two or three years, you are going to see lots of sort of altering of these [security] assets, as a result of the AI can do the monitoring and studying to take motion on issues a lot sooner than individuals can.
The Perspective Shift
What have you ever learn, watched or listened to just lately that modified how you concentrate on management or expertise — even barely?
I watch lots of YouTube channels and ship issues off to my workers a couple of new Microsoft function or a frontier AI beta launch. It is cool spending a little bit time to be accustomed to how corporations are deploying their expertise and the way we are able to deploy this to our clients.
I additionally just like the podcast Masters of Scale, which talks about how startups scale their corporations. You learn the way their enterprise manages progress and what they’re prioritizing.
I additionally just like the YouTube channel Ship it Weekly. It is in regards to the lowest-level expertise change replace — like alternative ways to configure and handle GitHub flows. It offers me little nuggets of knowledge, so after I’m in an operations overview or speaking to builders, I sound extra knowledgeable, and I can ask extra questions. The builders generally begin to consider issues a little bit otherwise as a result of they know their leaders are being attentive to these particulars. That offers us a extra dependable product on the again finish.
Get extra IT management updates and insights thrice per week direct to your inbox with the InformationWeek e-newsletter.
