Stop Asking What AI Agents Can Do. Ask What They Should Be Allowed to Do

2026-11-17来源 HackerNoonEnglish
Stop Asking What AI Agents Can Do. Ask What They Should Be Allowed to Do

The question for technology leaders is no longer what AI can generate. It’s what AI should be allowed to do.


Ask a chatbot to summarize an engineering change notice and get it wrong, and someone rereads the document. Ask an agent to release that change in your PLM system and get it wrong, and a supplier may build the wrong part. That is the shift. Generative AI changed how we create information: drafts, summaries, code, answers. Agentic AI changes how work gets executed.


An agent interprets a goal, decides what to do next, picks tools, acts on external systems, checks the result, and keeps going until the task is done or a human has to step in. OpenAI’s practical guide to building agents draws the line clearly: bolting an LLM onto a chatbot or classifier doesn’t make it an agent, because the model isn’t controlling the workflow.


Once software moves from recommending an action to taking one, the risk profile changes. Your architecture has to change with it.

From scripted automation to bounded autonomy

Traditional automation follows rules. It’s great when inputs, decisions, and exceptions can be mapped in advance.


Agents earn their keep when the path can’t be fully mapped: ambiguous inputs, unstructured documents, shifting conditions, or rule sets that have become unmaintainable. Instead of following a fixed script, an agent chooses its next step and its tools based on what it observes.

Under the hood, most agents combine five parts: a model that reasons and plans, tools or APIs to read and act, memory or state, an execution loop that observes and adjusts, and guardrails that constrain all of it. Recent research on agent architectures frames this as a control loop and names the trade-offs you are signing up for: autonomy vs. controllability, accuracy vs. latency, capability vs. reliability.


The mental model I’d suggest isn’t “digital worker.” It’s a goal-directed loop wrapped in enforceable controls:

Figure 1: Autonomy bounded by identity, policy gates, observability, verification, and human escalation for consequential actions.

Don’t agentize everything

The best use cases aren’t “tasks we can automate.” They’re outcomes that today need a person to interpret information, hop across several systems, handle exceptions, and coordinate steps.


Good candidates share five traits:

  1. The workflow contains real ambiguity.
  2. Several tools or data sources are involved.
  3. The outcome can be objectively checked.
  4. Errors can be detected, contained, or reversed.
  5. A human can be inserted at defined escalation points.


If a process is stable and rules-based, conventional automation will usually be cheaper and easier to govern. Anthropic advises starting with the simplest thing that works, since agents typically trade extra cost and latency for better performance on hard tasks.


Workflow design matters as much as the model. McKinsey’s 2026 research found that the strongest AI performers were far more likely to redesign workflows than to layer AI on top of existing ones. Drop an agent into a fragmented process and it will execute the fragmentation faster.

The real risk isn’t a bad answer. It’s a bad action.

Models still confabulate. NIST’s Generative AI Profile lists confabulation, privacy, information security, and third-party integration among the core risks. Agents inherit all of that, then add credentials, tool calls, record updates, code execution, and downstream triggers.


The OWASP Top 10 for Agentic Applications reads like an architecture review checklist: goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, insecure inter-agent communication, cascading failures, rogue agents. None of these are fixed by a better model. They’re identity, data, and governance problems.


So never grant an agent broad access just because it might need it someday. Every production agent needs:

  • Its own identity, not a shared service account
  • Least-privilege access to data and tools
  • Hard limits on write, delete, payment, deployment, and outbound communication
  • Human approval for sensitive or irreversible actions
  • Budgets for retries, spend, and execution time
  • A kill switch: pause, revoke, roll back, shut down


NIST has launched an AI Agent Standards Initiative focused on agent authentication, identity, security evaluation, and interoperability. That’s a strong signal: agent identity is becoming a first-class enterprise architecture concern, not a deployment detail.

Observe decisions, not just uptime

A green dashboard tells you the API returned 200. It doesn’t tell you whether the agent picked the right tool, pulled the right record, applied the right policy, or did the right thing.


Agent observability has to reconstruct the whole path: the goal it received, the data it touched, the tools it called, the intermediate results that shaped each step, the cost, every human approval or override, and whether the business outcome was actually correct. Current guidance pairs offline evaluation with production monitoring and distributed tracing, tracking task completion, tool-call accuracy, latency, token use, and safety signals.


This is about accountability as much as debugging. If you can’t reconstruct why an action happened, you don’t have a production-ready agent.

Five questions before you scale

Skip the next maturity model. Ask these about every proposed agent:

  1. What outcome owns the business case? Measure cost per successful outcome, cycle time, quality, exception rate, and remaining human effort, not novelty.
  2. How much autonomy is actually needed? Start with retrieval and recommendations, then reversible actions, then consequential ones. Autonomy is earned with evidence.
  3. What authority does it get? Document identity, data scope, tools, permissions, approval points, and prohibited actions. Permissions are product design.
  4. What proves it’s dependable? Test full workflows, not demos: edge cases, adversarial inputs, tool failures, drifting data. NIST recommends explicit go/no-go criteria, ongoing monitoring, and an incident process.
  5. Who is accountable? Name a business owner, a technical owner, and who gets the escalation when retry limits are hit or an action is high-risk or irreversible.

The bottom line

Agentic AI isn’t a model-selection decision. It’s a decision about delegating authority to software.

The teams that win won’t treat agents as unsupervised digital employees or as one more automation layer. They’ll run them as managed enterprise systems: narrowly scoped, identity-aware, observable, measurable, and accountable.

The advantage won’t come from maximum autonomy. It will come from the right autonomy, on the right workflow, backed by the right evidence and controls.


原文:https://www.a8q9.cn/en/article/stop-asking-what-ai-agents-can-do-ask-what-they-should-be-allowed-to-do-9770f6.html

相关文章