The era of generative AI needs humans… not just the data sets they create. Now and going forward, companies will need to depend on perceptive, intelligent, creative humans who can evaluate and, when necessary, stand up to the black box output AI throws at them. What kind of traits do such people embody? Where might we find them? What might they teach the rest of us? That’s what this episode of *Silo Busting* is all about. Tariq King, our VP and Head of Product-Service Systems, and Ira Livshits, Senior Data Scientist at EPAM, meet the test-based questions of Producer Ken Gordon with knowledge and aplomb. 

King, who spent much of his career hiring testers, says he looks for “the way that someone actually approaches solving problems and thinking perhaps in a different way or from a different perspective.”

Livshits notes that with testing in generative AI, “sometimes there is no right or wrong” and that a tester will “have to act or decide based on his or her understanding of the field and maybe intuition.” In short, she says, testing has become much less of a black-and-white job.

Together King and Livshits create a model conversation. They talk about how explainability fits in here, why musicians and liberal arts people make for good testers, the importance of being able to relate findings to a given audience (“[if] I have a PhD focused on machine learning, probably I can understand certain things in a different way than someone who may not have that degree,” says King), and how generative AI is bringing the role of testing closer to the fore.

Go on, give it a test-listen.

Host: Alison Kotin
Engineer: Kyp Pilalas
Producer: Ken Gordon

Silo Busting 62 Human Testing and Generative AI with Tariq King and Ira Livshits

Can today’s companies afford to be Luddites? 

This is one of the big questions that Elaina Shekhter, EPAM’s Chief Marketing & Strategy Officer and SVP, puts to today’s *Resonance Test* guest, Rowan Curran, Senior Analyst at Forrester. 

In the case of generative AI, both answer: No. Why? Shekhter notes that whatever your competitive edge in 2022, today everyone is encountering a different mode of operations. The positioning around the success or failure of your AI efforts must be “accelerating along the vector of AI, because the opportunity to get away from the competition, faster, is much greater now than it ever has been.”

In a lively and informed session of back-and-forth, they parse what is real and what is a hallucination in GenAI *at this moment.*

Curran says that lately there has been an “ebullient explosion” of work on tools and approaches to manage system outputs. “Are we there yet in terms of having these be optimized architectures and things like that? Absolutely not. But is there tons of work being done there or are we approaching reasonable solutions to those problems? Yes, absolutely.”

What should companies be doing to ensure they're ready to benefit and succeed with AI?

“Right now, everybody's building the gen one of enterprise generative AI applications,” says Curran, and this will make them ubiquitous. But if your organization fails to adopt them, he adds: “You are going to be falling behind everybody else who is actually building with this stuff today.”

Listen closely and learn what will the currency of the future be, the commercial and economic models of successful GenAI, the nature of productivity gains: “Somebody saving 30 minutes per day who makes $60K a year is going to have a very different economic impact on the company versus somebody who makes $200K a year and saves 30 minutes per day,” Curran says. They also discuss how this new tech will transform the shape of work and what companies will be focusing on this year: “2023 is the year of excitement and experimentation, and 2024 is the year of optimization and efficiency,” says Curran.

Oh… and it might also transform the future of fun! “I do think we could use the new technology to make work more fun for people,” says Shekhter, who sees in the soaring advance of multimodal LLMs an opportunity for people “to develop in an enlightened way.”

Enlighten yourself first. Smash that play button. 

Host: Alison Kotin
Engineer: Kyp Pilalas
Producer: Ken Gordon

The Resonance Test 89: Guest Speaker Rowan Curran and Elaina Shekhter on Generative AI

Responsible AI isn’t about laying down the law. Creating responsible AI systems and policies is necessarily an iterative, longitudinal endeavor. Doing it right requires constant conversation among people with diverse kinds of expertise, experience, and attitudes. Which is exactly what today’s episode of *The Resonance Test* embodies. We bring to the virtual table David Goodis, Partner at INQ Law, and Martin Lopatka, Managing Principal of AI Consulting at EPAM, and ask them to lay down their cards. Turns out, they are holding insights as sharp as diamonds. 

This well-balanced pair begins by talking about definitions. Goodis mentions the recent Canadian draft legislation to regulate AI, which asks “What is harm?” because, he says, “What we're trying to do is minimize harm or avoid harm.” The legislation casts harm as physical or psychological harm, damage to a person's property (“Suppose that could include intellectual property,” Goodis says), and any economic loss to a person. 

This leads Lopatka to wonder whether there should be “a differentiation in the way that we legislate fully autonomous systems that are just part of automated pipelines.” What happens, he wonders, when there is an inherently symbiotic system between AI and humans, where “the design is intended to augment human reasoning or activities in any way”?

Goodis is comforted when a human is looped in and isn’t merely saying: “Hey, AI system, go ahead and make that decision about David, can he get the bank loan, yes or no?”

This nudges Lopatka to respond: “The inverse is, I would say, true for myself. I feel like putting a human in the loop can often be a way to shunt off responsibility for inherent choices that are made in the way that AI systems are designed.” He wonders if more scrutiny is needed in designing the systems that present results to human decision-makers.

We also need to examine how those systems operate, says Goodis, pointing out that while an AI system might not be “really making the decision,” it might be “*steering* that decision or influencing that decision in a way that maybe we're not comfortable with.”

This episode will prepare you to think about informed consent (“It's impossible to expect that people have actually even read, let alone *comprehended,* the terms of services that they are supposedly accepting,” says Lopatka), the role of corporate oversight, the need to educate users about risk, and the shared obligation involved in building responsible AI.  

One fascinating exchange centered on the topic of autonomy, toward which Lopatka suggests that a user might have mixed feelings. “Maybe I will object to one use [of personal data] but not another and subscribe to the value proposition that by allowing an organization to process my data in a particular way, there is an upside for me in terms of things like personalized services or efficiency gains for myself. But I may have a conscientious objection to [other] things.”

To which Goodis reasonably asks: “I like your idea, but how do you implement that?”

There is no final answer, obviously, but at one point, Goodis suggests a reasonable starting point: “Maybe it is a combination of consent versus ensuring organizations act in an ethical manner.”
This is a conversation for everyone to hear. So listen, and join Goodis and Lopatka in this important dialogue. 

Host: Alison Kotin
Engineer: Kyp Pilalas
Producer: Ken Gordon

The Resonance Test 90: Responsible AI with David Goodis and Martin Lopatka

“One of the most essential parts of bringing innovation to market is often the most rarely noted,” says host Macy Donaway on the latest Resonance Test podcast. “And it’s those dedicated client leads and sponsors who have political capital built that they can spend to then overcome hurdles.” We call such people mavericks, and Uma Gopinath, the CIO of Porter Airlines and our podcast guest, perfectly embodies that term. 

Gopinath has been a highly successful change-maker in numerous companies and industries (she was the CIO of Metrolinx, the Director of Technology and Innovation at Lush, and the AVP of Intelligent Automation at Canadian Tire Corporation). Along the way, she has learned how to thrive in the heavily male-dominated technology industry and shares some of her wisdom in this conversation. 

Giving back, in fact, is central to her work. “As a person of privilege, you need to share that privilege with others,” she says, noting that when at Metrolinx, she noticed the diversity of her teams was “in the low teens when we started,” and by the time she exited “We were close to 30-35% in diversity.”

She says that change happens “by intention.” And notes that when a woman didn’t win a particular role, she would ask her colleagues why and was often told, “But she’s the second best.” To this, Gopinath argued that perhaps she was “second best because she's never been given the opportunity to be the first best.”

Fixing systemic bias, she notes: “Calls for courage, calls for some unpopular statements sometimes.”

Courage is a central part of Gopinath’s general ethos, and it takes the shape of a willingness to be curious, to experiment (and experiment at scale: “your denominator has to be big for you to get those useful, successful experiments,” she says). Gopinath talks up the importance of focusing on the customer. Continuously. 

Gopinath notes that many organizations brew up a business case and do a project, “but then nobody goes back to effectively evaluate” the outcomes originally projected. Consequently, she says, “We hear lots of stories about how IT projects don’t deliver.”
She adds that sometimes it’s “a small feedback loop that's required” and that doing “a little more to get to that bigger benefit” is something businesses need to do better.

Gopinath ends with some memorable maverick-level inspiration for future leaders: “Enjoy what you're doing. If you’re not having fun, then go be successful somewhere else.”
Now go have fun and listen to the episode! 

Host: Alison Kotin
Engineer: Kyp Pilalas
Producer: Ken Gordon

The Resonance Test 92: Lessons from a Maverick with Uma Gopinath and Macy Donaway

We’re talking about the lab of the future! Better than that… we’re *building* it. In this episode of *The Resonance Test,* two of the builders are giving us a tour of sorts! Listen as Sridhar Iyengar, Founder, Chief Strategy and Technology Officer and Chairman of Elemental Machines, and Chris Waller, EPAM’s VP and Chief Scientist, chew the scientific fat about creating a collaborative model cell and gene therapy laboratory.

Waller says the lab of the future seeks to “reinvent the way we look at equipment and utilize equipment in a laboratory setting that's used to manufacture cells” by making it, as we say, real. “We’re building out that facility at the EPAM Continuum office in Boston and partnering with folks like Elemental Machines” to enable “the transformation that we're looking for in these laboratory settings.”

Creating such a next-gen lab is a very complicated task, says Iyengar. “Unlike many other disciplines that are primarily software driven or even mechanically driven, the life sciences have a much greater degree of variability.”  

To minimize this variability, they’re putting an Amazon Go level of scrutiny on lab processes. AI, Iyengar says, “can spot patterns across an enormous number of variables and dimensions, much more than any human being can do… To do that you need lots of data, so you can cancel out the noise and you can find the signal in the noise.” The key step here, he adds, is to collect “as many dimensions of data as possible and make it computationally available.”

At present, says Waller, the EPAM Continuum facility enables us to give the future a test run. The lab “allows us to bring our clients, our members of the [Pistoia] Alliance [and] our technology partners together in a safe space to work collectively to derisk the introduction of new technologies into these laboratory settings and show us the future.”

Iyengar adds that when he walks people through the lab, “You see their eyes light up and say, ‘Ohh, I get it; that means we can do XYZ!’” Listen to these two and you’ll soon be having your own XYZ thoughts. 

Host: Alison Kotin
Engineer: Kyp Pilalas
Producer: Ken Gordon

The Resonance Test 93: The Lab of the Future with Sridhar Iyengar and Chris Waller

“Can we use generative AI in a way that teaches us something that we might not have known otherwise, and in that learning… create something that actually has the potential to increase agency for all inside of the system?”

Good question, Angela Stockman! It is, in fact, one of many good questions that Stockman, the author of *The Writer’s Guide to Pedagogical Documentation,* raises as our guest on *The Resonance Test.*

In this episode, Stockman joins Kristin Heist, Senior Director of Innovation Consulting at EPAM Continuum, and our Brian Imholte to dig deep into Gen AI and education. 

Part of that digging involves the art of *asking questions.* Heist says that while building a tutor with input from educators, teachers have been “pushing us to design tools that follow the principles of Socratic method” and not just giving the answers to students.  

Stockman agrees saying that teachers don’t want to see “learners leaning on AI tools just to generate answers or to produce work in ways that you know undermine their opportunity to sharpen their own saw.”

The hope is that students will become keen enough to create whole new ways of doing things—and that teachers will, too. 

Part of teachers’ craft, says Heist, is “learning what works for their students, learning what their students understand, learning who their students are.” But the reality is that teachers are extremely time-constrained. And this makes personalization a challenge.
 
Stockman says that for teachers “who are working with sometimes over one hundred students in a single day,” personalization is “kind of unrealistic”—but a GenAI tutor can truly help.

The real focus is where GenAI tools can, as Heist says, “elevate the teacher's craft,” as opposed to replacing what they're currently doing.

And let’s not forget data! Stockman says that AI is “helping us scoop the data out of their lived learning experiences. We don't have to bring learning to a halt in order to assess what's going on and it can help us with the interpretation of massive amounts of qualitative data.” 

If you have questions about GenAI and EDU, and you know you do, listen up. 

Host: Alison Kotin
Engineer: Kyp Pilalas
Producer: Ken Gordon

The Resonance Test 94: Angela Stockman on GenAI in Education

What should we be focusing on? It’s an essential question for all of us… but for those in the cybersecurity game, it’s critical. Focusing on the wrong things here can be *costly.* Or so says Neatsun Ziv, Co-Founder and CEO of OX Security, in this *#CybersecurityByDesign* conversation with Sam Rehman, EPAM’s CISO and SVP. 

Ziv says that, when it comes to fixing code, “95% of the things” that organizations work on have “zero risk impact,” adding: “That is an insane amount of money that the organization should have spent creating a bigger gap between them and the competitors.”

Managing risk is indeed a major challenge for contemporary organizations. “There's no such thing as one single application anymore,” says Rehman, who wonders: “How do you manage the inherent risk from all these components?”

Ziv says the answer is about getting clients to focus on what’s critical to them. There’s a need to distinguish between vulnerability, theoretical risk and actual practical risk. What is a practical risk? It means getting clients to recognize, as an organization: “This is what I'm concerned about.” 

“I always tell people the risk is managed,” adds Rehman, who says that approach is underpinned by asking questions such as “What's your security posture?” and assessing a client’s risk tolerance.

“You need to be smart about the investment,” says Ziv. He notes that this is where experienced leaders become practical. He gets them to answer questions like: What's exposed? What's internal? What do you want to replace first? How do we do it? 

Ultimately, he says, it’s getting clients to be mature enough to say that they’d focus on the “5% [of efforts] that would actually make a difference in the first year or the second year.” The trick is taking a balanced approach, and the guys bring the idea of balance into the realms of supply chain and open source. 

We leave you with a warning: Listening might have a serious impact on your own security posture. Click play now! 

Host: Kenji Ross
Engineer: Kyp Pilalas
Producer: Ken Gordon

Silo Busting 69: Neatsun Ziv and Sam Rehman on the Balanced Approach to Risk Management

“If you want to know the future, look at the past.” While no one in their right mind would claim Einstein was giving any thought whatsoever to the future of the financial services industry, history would have once again proven him right if he had. Once upon a time, one’s choice of bank was based on how well that particular institution had won over your trust. That meant building relationships with the customers and the community at large.


While it’s hard to envision any other reality, it’s only been a couple of decades where convenience and the digital experience became such dominating factors. But as banks increasingly leverage emerging technology to race toward automation and digital optimization, the notion of trust and relationship building has all but faded into the rearview. Or has it?


“Yes, you need to have an amazing digital experience. It needs to be convenient, easy and it needs to flow… But you also need to be able to get ahold of a banker just as quickly. You need that opportunity to have someone jump into that digital experience as you’re a part of it–if you have a challenge, or a question or an alternative need. For our industry, that’s what everyone is trying to solve.” This is the mantra of Eric Sobie, SVP, Head of Consumer Sales and Strategy at Heartland Financial USA, Inc. It’s his belief that in order for the financial services industry to move forward, it needs to find a way to bridge the gap between the personalized experiences of the past with the efficiency and digital innovation of today.

And as Chris Tapley, VP of Financial Services Consulting at EPAM, points out, it’s a notion with strong precedent behind it. “It’s interesting because what bank customers told us is that they have very disparate use cases for in-person versus digital interactions with their banks. They want to move money digitally; they want to check balances digitally; but when they need advice, or when they need a problem solved, they… want some kind of in-person interaction. Being able to converge that digital and human experience will be critical.”


As the two discuss, it's this convergence where the emergent technologies of AI and generative AI will play an important role. While they might make it possible to further automate, optimize and streamline customer interactions – for example, recommending a specific product for a customer based on the available data – is this what consumers really want?


As Sobie makes note, the alternative is to leverage these technologies to gain a more thorough understanding of the customer, and to upskill bankers to better build relationships with their customers based on this insight. “I think banks, rightfully so, are a little nervous about AI, because we are a business built upon trust; we want nothing to invalidate that trust. Let’s not use AI to create an opportunity to drive [a product] recommendation. But can we get predictive analytics? Can we understand where the customer is on their road map… and can we get a whole view of who that customer is from the data?” That’s how banks can bridge the gap between today’s digital environment and the personalized experiences of the past.

Now, it’s time to dive into the full *Resonance Test* conversation. Enjoy your stay.


Host: Alison Kotin 
Engineer: Kyp Pilalas 
Producer: Scott MacAllister
Executive Producer: Ken Gordon

The Resonance Test 95: Technology, Trust & the Customer Experience with Eric Sobie & Chris Tapley

How are CISOs holding up in the era of AI? 

According to Tim Ramsay, Managing Director of Mandiant Client Advisory (now part of Google Cloud), and our guest on *Silo Busting*: “You have a number of parts of the organization that may be embracing AI without any involvement from central IT, and more importantly… without security.”

Not an easy situation for a CISO. 

But not to worry, Ramsay and Sam Rehman, EPAM’s CISO and SVP, have seen this kind of thing before. In the pre-AI age, there were other technology inflection points, such as virtualization and the cloud, and our conversationalists learned that dealing with them involved clear communication and trust.

Today’s CISOs “don't want to kill the business or stop the business,” says Ramsay. “They want to enable the business. But that kind of presupposes they know what the business is trying to do.” 

What’s necessary, he says, is for business leaders “to have some level of trust that the security people are actually going to bring something productive to the conversation and not just rule from a position of fear, uncertainty and doubt.”

CISOs must teach their colleagues that secure business is, as Ramsay notes, a team sport and that organizations must know their data assets. Security people must also be clear about risk. “We need to be real about what type of threats we actually are engaging,” says Ramsay.

The lessons of DeepSeek emerge during the episode. Ramsay says he thought there’d be “some voice in the room who would have said, ‘Guys, are we ready? Are we ready for global type of exposure here?’” Getting ready, in fact, means that security must be included from the beginning, both Ramsay and Rehman agree. Rehman adds: “To secure something as an aftermath is a million times more difficult than if you have security in mind when you’re actually going through that innovation process.” 

Rehman asks how CISOs can build the necessary trust. “Meetings are always good, but relationships are where it gets real,” replies Ramsay. “Conversations that CISOs are having alongside other C-levels are going to be much more effective” than meetings that can sometimes feel adversarial. 

Build strong enough relationships and sometimes business leaders will deliver the security message themselves. “It takes a secure CISO to let others carry the message sometimes,” says Ramsay. “It takes the pressure off the CISO to be always the bearer of threats and news of risk.”

Says Rehman: “So much of security requires... letting go of that insecurity.”


Host and Producer: Ken Gordon
Engineer: Kyp Pilalas

Silo Busting 70: Lessons for the Modern CISO with Tim Ramsay and Sam Rehman

Today’s incident response ain’t your grandfather’s IR. But the psychology surrounding it hasn’t changed an iota. This is precisely what Sam Rehman, EPAM’s Chief Information Security Officer and SVP, and Tab Bradshaw, Chief Operating Officer at Redpoint Cybersecurity, are talking about on this #SecurityByDesign conversation. 

“It really comes down to the preparation piece,” says Bradshaw. It’s about being well prepared and asking: “How often do you prepare in your organization, at a technical level, at an executive level, to handle some sort of incident?” 

Rehman agrees and says that he has clients wondering, “OK, so when am I done?” The perception is that being IR-ready is enough, he says. “That's not the case. It's a muscle. It's emotion. It's how you work. It's how you react to it.”

There are benefits to knowing the proper way to react. “A well-handled breach really builds credibility,” says Bradshaw, adding that the word “reasonable” is omnipresent in IR documentation. He says: “Reasonableness is not just about having a mitigation strategy.” It’s also about, say, practicing tabletop exercises. Regularly. So that when you’re asked about doing regular tabletop sessions, the answer is, as Bradshaw puts it: “Yes, we did it every quarter for the past five years. We feel like we're in a pretty good spot that if something happens, might not be perfect, but we think we have good preparation, consistent preparation, consistent practice, to your point, to respond to the incident when it does occur.”

Rehman says that security people are “used to having that sudden sense of violent impulse and urgency coming to us,” but what about the business leaders and everyone else in the organization? He asks Bradshaw about IR communication: “How do you guide the team through it, especially when everybody's thinking about, ‘Oh, am I gonna be on the news?’”
Of the thousands of breaches Bradshaw and his team have responded to, for “a third, maybe half” of them, there is “some internal chaos at the client—and it's not because anybody's doing a bad thing.” 

“It really comes down to what I call C-squared,” says Bradshaw, which is shorthand for “communication and coordination. Someone has to be the quarterback.”

Bradshaw says the chaos is about “a lack of preparation and testing.” A tabletop exercise needs to be a live fire exercise: “Doing it once a year is not good.” Too many organizations treat IR as a checklist, which is a mistake. He says: “It's a living, cross-functional discipline that evolves with the threat landscape externally, obviously, and also internally as people move.”

And so?  

Get moving. Hit play and get ready. 

Host: Lisa Kocian
Engineer: Kyp Pilalas
Producer: Ken Gordon



Silo Busting 71: IR Now with Tab Bradshaw and Sam Rehman

In the age of AI, there’s no going it alone. Partnership is now an absolute necessity. 

This conversation between Colleen Kapase, VP of Channels and Partner Programs at Google Cloud, and Elaina Shekhter, EPAM’s Chief Marketing and Strategy Officer, demonstrates the value of partnership done right. Their back-and-forth embodies the enduring partnership, and unshakable trust, that Goole Cloud and EPAM have built over the years. 

Kapase notes that the conversations she's had with partners were often about modernizing cloud infrastructure rather than more nuanced AI discussions. No more! “It's moved beyond a CIO conversation to a product conversation, to a CMO conversation.” Google and their partners are asking: “What are you doing to leverage AI to advance our products or offerings or processes and customer experience?”

This kind of working is, Kapase says, an opportunity to “grow, grow, grow, grow” that can deeply impact their partners’ customer experience and product development.

Shekhter says that lately there has been much restructuring of the partner ecosystem and then asks bluntly: What is partnership *for?*

“Just delighting the customer,” says Kapase. “I don't know if it gets any more complicated than that.”

Complicating things somewhat, Shekhter wonders if her interlocutor has advice on how partnership can be customer-centric in AI-native transformation work.

“It can sound basic, but communication is *so* important,” says Kapase. “It is really the basis of any great partnership… Strong communication just can make you better together.”
Better together, indeed. 

So, for those who are in or want to join the Google Cloud crowd, or care about partnerships more generally, listen up! Much here to learn about partner-driven delivery and adoption, the role of Agentspace and AI innovation, the importance of optimism, and more.

The Resonance Test 97: The Power of Partnership with Colleen Kapase and Elaina Shekhter

EPAM Continuum's award-winning podcasts feature interviews with people practicing innovation in various forms, digging into their ability to deliver results. Repeatedly.

Silo Busting 62 Human Testing and Generative AI with Tariq King and Ira Livshits

The EPAM Continuum Podcast Network

Related tracks