An AI Testing Presentation Became the Final Push for Organizational Change at My Company

From a multi-agent testing experiment to a broader discussion about how engineering teams should evolve in the age of AI.

At first, I simply wanted to share an AI-powered automated testing system I had been working on.

The system could already understand requirements, generate test cases, interact with web browsers and mobile apps, identify defects, and connect bug fixing with regression testing. I expected the session to focus on how AI could improve testing efficiency.

But during our internal session on November 5, 2026, the discussion gradually moved beyond testing.

We started talking about how product, development, and QA teams work together. We discussed why requirements so often fall out of sync, and how AI might change the way engineering teams are organized.

By the end of the discussion, the session had become the final push that helped set our company's organizational transformation in motion.

Looking back, I find that quite interesting. We set out to help testers work more efficiently with AI. Instead, we ended up discussing how AI might force the entire engineering organization to rethink the way it works.

1. 🤖 I Wanted AI to Handle the Entire Testing Process

I've been working on a multi-agent automated testing system.

Rather than simply asking AI to generate automation scripts from test cases, I wanted to connect the entire process: understanding requirements, designing tests, executing them, reporting defects, verifying fixes, and running regression tests.

The workflow looks roughly like this:

1. AI reads product requirement documents and prototypes to understand business workflows and acceptance criteria.
2. It generates test cases, which are then reviewed by multiple agents.
3. Based on those test cases, AI controls web browsers and mobile apps, simulating real user interactions to execute the tests.
4. When it finds an issue, AI generates a defect report and submits it to our issue-tracking system.
5. Each reported defect goes through a second review and human confirmation before being assigned to the appropriate developer.
6. After the developer fixes the issue, AI automatically runs regression tests to verify the fix and move the defect through the closure process, with human oversight retained.

The end-to-end workflow, from requirements to regression testing, is now in place and working.

What impressed me most was that AI had started taking over some of the most frustrating parts of traditional test automation.

With UI automation, the hardest part was often not writing the initial scripts. It was maintaining them.

A developer changes a UI component, and testers may have to revisit element locators, page objects, and interaction scripts. The more frequently the interface changes, the higher the maintenance cost. Sometimes, testers spend more time maintaining automation scripts than actually running tests.

Now, AI can observe the current page, understand its elements and interaction states, and perform actions based on the testing objectives. It still has to deal with UI changes and execution failures, but it can already take over a considerable amount of work that previously required repeated manual maintenance.

That made me reconsider the limits of automated testing.

2. 🔍 Automated Testing Should Do More Than Repeat Known Test Cases

For a long time, I held a fairly firm view of automated testing:

Automation is useful for regression testing, but it is not very good at discovering new defects.

The reason is straightforward. Traditional automated testing relies heavily on test cases designed in advance, fixed execution steps, and predefined assertions. It can repeatedly execute known test paths, but it struggles to discover problems that testers never thought to look for.

It behaves more like an executor that follows instructions than a tester capable of exploring software behavior.

But as AI agents emerged that could understand requirements, observe screens, analyze page states, and adapt their actions, I began to change my mind.

When training and refining the agents' testing behavior, I emphasized one principle:

Don't just follow the requirements document. And don't treat the test cases as the limits of your exploration.

Requirements describe what a product is expected to do, but they cannot cover every possible user behavior or system state.

So, after completing the tests defined by the requirements, AI should also explore additional scenarios, including:

• Invalid and unexpected inputs: Empty values, excessively long text, illegal characters, unusual formats, and unexpected combinations of data.
• Boundary conditions: Maximum and minimum values, threshold conditions, duplicate submissions, and unusual sequences of operations.
• Abnormal workflows: Interrupted operations, repeated clicks, page refreshes, network failures, expired sessions, and inconsistent states between consecutive steps.
• Security-related scenarios: Unauthorized access, permission boundaries, sensitive information exposure, and attempts to perform operations that should not be allowed.
• Unexpected interactions: Skipping normal steps, performing rapid consecutive actions, repeatedly switching states, or accessing the same business function through different entry points.

Of course, this does not mean AI should be allowed to operate freely against production systems. The scope of exploration, test environments, account permissions, and operational risks must all be controlled. Security testing must also remain within clearly authorized boundaries.

I don't want AI to simply prove that the requirements have been implemented. I want it to look for problems that might emerge when the system is used in ways that were never anticipated.

This is where my understanding of automated testing has changed.

In the past, automation mainly repeated checks for known issues. Now, AI has the potential to help discover problems we didn't know to look for.

It cannot guarantee that every defect will be found, and it cannot replace human judgment. But automated testing no longer has to be limited to test cases written in advance.

3. 🌙 AI Works at Night. Humans Make the Decisions During the Day.

These days, I usually schedule automated testing tasks before finishing work.

The agents then continue running overnight. When I arrive at work the next morning, I can usually find a batch of defect reports already submitted by AI.

My job is to review those issues one by one, determine whether they are genuine and consistent with the current requirements, confirm them, and assign them to the appropriate developers.

This changes how testers spend their time.

Previously, testers had to spend a considerable amount of time operating systems, executing test cases, observing results, and preparing reports. Now, AI can take over a significant portion of that repetitive work, allowing people to focus more on evaluating issues, analyzing risks, and improving product quality.

However, this process also revealed another problem:

Finding a defect doesn't mean communicating it effectively.

Initially, AI-generated defect reports were often just dense blocks of text. They contained plenty of information, but they weren't particularly easy for developers to read.

If a report cannot help the person receiving it quickly understand the problem, then even a large amount of information does little to improve collaboration.

So I went through several rounds of adjustments to the reporting format. We needed more than a record of everything that happened during execution. The report had to clearly explain what happened, where the problem occurred, how the actual behavior differed from the expected behavior, and how the issue could be reproduced and verified.

After several iterations, the reports became much more readable and useful.

This made me realize that AI output quality should not be measured only by how much work it completes. We also need to consider whether the next person in the workflow can actually use the results.

After each testing cycle, I also ask AI to summarize the problems and lessons from that run. Experiences that have been reviewed, validated, and shown to be reusable are then distilled into general-purpose Skills.

That way, the next testing cycle doesn't have to start from scratch. It can build on what the previous cycle has learned.

In practice, this creates a continuous improvement loop:

Execute → Discover issues → Review → Capture lessons → Turn reusable knowledge into Skills → Start the next cycle.

4. 📊 The Numbers Show the Value, but Also Reveal Problems

In one of our knowledge-base projects, we've accumulated some concrete results:

In other words, AI identified 21 of the 30 defects ultimately confirmed as valid, accounting for 70%.

These figures show that, within the scope of this project and its test coverage, AI is no longer just a tool for executing test cases. It is making a real contribution to defect discovery.

We've also experimented with having agents navigate pages, capture the API calls involved in business workflows, and run scheduled, full-scale API validation every day. Our current test environment covers nearly 300 APIs associated with business workflows.

These experiments show that AI can take on continuous testing tasks at both the UI and API levels.

But the data also reveals some problems.

For example, we currently have only one test environment. When multiple requirements are being developed in parallel, code changes can conflict, and deployments can interrupt other testing tasks. We still need to evaluate adding a second test environment and a dedicated performance testing environment.

Browser compatibility also needs further work, including test coverage for different Windows versions and the 360 browser.

These are infrastructure problems. They can be addressed gradually through additional resources and technical improvements.

What really made me think, however, was another category of problems.

5. 🧩 Automation Exposes Organizational Problems That Traditional Testing Often Hides

At one point, we noticed that some defects reported by AI were being classified as invalid.

Our first suspicion was naturally that AI might have misunderstood the requirements or that there was a problem with the testing logic.

But after looking more closely, we found that AI wasn't entirely to blame.

Some requirements had already been adjusted through discussions between product and development, but the official PRD had not been updated. Product and development might have reached a new understanding, or even made changes to the implementation privately, while QA and AI were still working from the original requirements document.

As a result, AI identified a discrepancy based on the official requirements, but the reported issue was considered invalid.

From a testing perspective, AI hadn't necessarily done anything wrong. From the development team's perspective, based on their current understanding of the implementation, the issue might not have been something they believed needed fixing.

The real problem was that the team was working with multiple versions of the same requirement context.

This led me to an important observation:

Automation can expose organizational problems that traditional manual testing has quietly absorbed and hidden for years.

In the past, testers often compensated for inconsistencies in requirements documents through verbal communication, ad hoc clarification, and their own experience. This work is important, but it can also make an organization believe that its processes are working perfectly well.

AI doesn't automatically know about a private conversation, nor can it understand the latest version of someone's thinking. It executes tasks based on the context it receives.

When the official documentation no longer matches the team's actual understanding, automation exposes that gap.

So the question isn't simply whether AI is smart enough.

We also need to ask: Does the team have a shared version of the requirements that everyone agrees on, that is kept up to date, and that AI can correctly interpret?

6. 🏢 Why Did a Technical Presentation Turn into a Discussion About Organizational Structure?

On November 5, 2026, we held an internal session to share our experience with local AI agents and automated testing. One of the company's founders also joined the discussion.

The original focus was AI testing capabilities, but the conversation gradually expanded to engineering workflows and organizational structure.

I increasingly agree with one observation:

AI is reducing the cost of producing code, but the cost of coordinating people isn't decreasing at the same pace.

In the past, a requirement might be analyzed and documented by a product manager, handed over to frontend and backend developers, and eventually passed to QA for verification. Every handoff meant that context had to be transferred again.

If any part of that chain involved a misunderstanding, rework could follow.

Now, AI can help product managers make certain code changes. It can help testers understand and modify code. It can also allow developers to participate more directly in requirement analysis and product design.

If a small requirement only needs a few code changes, but still has to pass through multiple roles, processes, and rounds of communication, the coordination cost may far exceed the actual coding cost.

This doesn't mean professional specialization has lost its value. Rather, AI is gradually lowering some of the capability barriers between traditional roles.

During our discussion, we proposed a direction: moving away from traditional role-based division of labor toward a more flexible model in which people can work across traditional role boundaries with the help of AI.

When we talk about a full-stack approach across the team, we don't mean that everyone must become a traditional full-stack engineer, or that people should abandon their professional expertise.

What we really mean is this:

People in product, development, and QA should all be able to use AI to cross their previous technical boundaries and help solve the problems they understand.

For example:

• Product managers can use AI to make code changes and verify smaller requirements themselves.
• Developers can participate in requirement analysis earlier and take overall ownership of larger initiatives.
• Testers don't have to stop at submitting bug reports when they discover defects. They can use AI to investigate the issue and even modify code, with developers providing the necessary review and confirmation.

For small requirements, we want to reduce unnecessary handoffs. For larger initiatives, we want to assign a clear owner to coordinate the entire process.

This isn't about eliminating professional roles. It's about reducing the information loss caused by rigid boundaries between them.

7. 🔗 Collaboration Should Start with Requirements, Not After Testing Finds a Problem

To make this way of working possible, we first need to address the problem of shared requirement context.

We discussed several concrete directions.

First, product, development, and QA should participate in requirement alignment from the beginning, rather than waiting until the PRD is finished and development is complete before testers start trying to understand the requirements.

Second, we want to establish a standardized PRD Skill and structured templates so that requirement documents follow a more consistent format.

More importantly, whenever requirements change, the official documentation must be updated promptly. Changes should not exist only in private conversations between a few people.

We also discussed a more ambitious idea: using recordings from smart badges to identify requirement changes during discussions, flag content that might affect the requirements, and assist in updating the PRD. The proposed changes would then be submitted to the relevant people for review and approval.

This is still an idea to be explored, not a feature that has already been deployed.

But the underlying principle is clear:

We should make sure the team's actual shared understanding is captured promptly in a form that everyone can use.

Beyond PRDs, we also need shared development and testing Skills, coding standards, and Specs.

When product, development, and QA work from a reasonably consistent context, AI can become a tool that connects different parts of the workflow instead of creating more isolated information silos between roles.

8. 🚀 From Individual AI Tools to an AI-Driven Engineering System

The discussion eventually moved beyond any particular agent or automated testing tool.

We started looking at the engineering system as a whole.

For example:

• Building shared Skill libraries for development and testing.
• Standardizing coding conventions and Spec management.
• Creating a complete workflow: retrieving defects from Yunxiao, using AI to fix them, running automated regression tests, and retaining human review.
• Exploring automated effort estimation and requirement classification.
• Continuing to improve test environments and browser compatibility.
• Ensuring that requirement changes are communicated promptly to reduce unnecessary discussions and rework.

These initiatives won't be completed automatically after a single meeting. The organizational transformation plan has already been launched, and progress will be reviewed and coordinated through weekly updates.

Of course, having a clear direction and an initiated plan doesn't mean the road ahead will be smooth.

Once implementation begins, we'll inevitably encounter practical obstacles: the inertia of existing processes, established collaboration habits between different roles, limitations in technical infrastructure, and the challenges of fitting new ways of working into existing business rhythms.

These problems can't be solved through one meeting or a few new rules. We'll need to identify issues as they arise, adjust our approach, and gradually develop a working model that actually fits the team.

For me, this is what makes the discussion meaningful. It wasn't just about proposing an idea for organizational change. The plan has already moved into the implementation stage.

An AI-driven engineering system can't be built simply by purchasing a few tools, connecting some APIs, or creating a handful of agents.

It requires technical capabilities, process standards, accumulated knowledge, and organizational collaboration to evolve together.

If information within the team remains inconsistent, even the most capable AI will simply execute the wrong context more efficiently.

9. 💡 In the AI Era, Coding Ability May No Longer Be the Rarest Resource

Looking back, I initially just wanted AI to help me automate more testing work.

But as AI began to understand requirements, interact with interfaces, actively explore abnormal scenarios, submit defects, assist with fixes, and perform regression testing, I realized that I needed to rethink more than just testing. I needed to rethink the entire software development process.

In the past, many efficiency problems were attributed to a lack of capability in a particular role, or to a specific stage of the process not being fast enough.

Now, AI is making more and more tasks easier.

But as writing code becomes easier, what may actually slow down development is the failure to synchronize requirements, unclear ownership, and too many handoffs between people.

I don't believe AI will turn everyone into a programmer, nor do I think software teams will no longer need professional roles such as product, development, and QA.

I believe AI will gradually remove unnecessary barriers between these roles.

People who understand a problem should be able to use AI to move toward a solution more directly, rather than having to wait for information to pass through multiple roles.

For testers, this means going beyond executing test cases to participating in risk exploration, problem analysis, and quality improvement.

For developers, it means taking responsibility not just for receiving requirements and writing code, but for solving a more complete problem.

For product managers, it means going beyond describing requirements and using AI to validate and implement their ideas more directly.

Ultimately, a team's competitiveness may no longer depend on how many people complete how many separate tasks. It may depend on whether the team can move accurate information quickly and enable everyone to use AI to solve more complete problems.

An AI testing presentation eventually became a discussion about how the entire engineering organization should change.

Perhaps this is one of the most important changes to pay attention to in the AI era:

AI is making coding easier, but what will truly make software development faster may be our willingness to redesign the way people work together.