Companies are handing artificial intelligence systems access to emails, databases, software tools, and financial workflows, often without fully understanding what could happen when those systems make the wrong decision.
The rise of autonomous AI agents has created a new source of productivity for global businesses. Unlike conventional chatbots, these systems can plan tasks, use external tools, and take a series of actions with limited human involvement. They can answer customer queries, prepare reports, write software, process invoices, screen job applications and communicate with other systems.
The same autonomy that makes AI agents valuable can also turn them into a serious operational and security threat when effective human controls are absent.
Steven Mills, managing director and chief AI ethics officer at Boston Consulting Group, has warned that many organisations are accelerating the use of AI agents without knowing how to manage the associated risks. Businesses face pressure to show returns from their artificial intelligence investments, but their approval, testing and monitoring systems are frequently designed for slower and more predictable technologies.
“The desire to move fast on AI without putting appropriate risk management in place can result in a system lapse or the deployment of AI in heavily regulated areas that organisations are not prepared to manage,” Steven Mills, BCG’s chief AI ethics officer, wrote in a commentary published by the firm.
The warning places AI agent risk management firmly on the agenda of boards, investors and regulators. For companies, the immediate threat is not limited to a distant scenario in which machines become uncontrollable. It includes data leaks, unauthorised payments, inaccurate customer communication, discriminatory decisions, cybersecurity failures and breaches of industry regulations.
Why AI Agent Risk Management Is More Difficult
Traditional software generally follows instructions written and tested in advance. An AI agent operates differently. It interprets a goal, decides which steps to take, and may adjust its approach as it receives new information.
A conventional payroll application, for example, calculates salaries according to fixed rules. An AI payroll agent could review employee records, investigate inconsistencies, contact staff, and recommend or initiate changes. Giving the system more freedom may reduce manual work, but it also expands the number of ways in which an error can occur.
The risks increase when an agent is connected to company systems. Access to email, cloud storage, source code, customer records, or payment software gives the agent the ability to cause real-world consequences. A mistaken chatbot response may be corrected. A mistaken action taken inside a financial or operational system may be harder to reverse.
Agents can also inherit the permissions of the employees who deploy them. If an executive has access to sensitive contracts or banking information, an agent operating through that account may receive similar access. Weak identity controls could therefore turn an apparently useful assistant into a serious security vulnerability.
These systems may be affected by false information, manipulated instructions, or hidden commands embedded in documents and websites. An agent asked to research a supplier could encounter malicious content designed to redirect its actions or persuade it to reveal confidential information.
AI agent risk management must consequently cover more than the accuracy of a model’s answers. It must examine what the agent can access, what it can change, how much money it can move, whom it can contact, and how quickly it can be stopped.
Corporate Adoption Is Moving Ahead of Governance
The commercial appeal is clear. AI agents promise to complete multi-stage assignments that previously required employees to move between several applications. Technology suppliers are marketing them as digital workers capable of operating continuously and handling large volumes of routine activity.
Industry research suggests that adoption is advancing faster than operational readiness. Gartner has predicted that more than 40% of agentic AI projects will be cancelled by the end of 2027 because of rising costs, uncertain business value, or inadequate risk controls. It has also warned that governance failures may become visible only after companies experience incidents in live environments.
According to ServiceNow’s 2026 Enterprise AI Maturity Index, enterprise AI investment in India increased by 119% over the previous year. AI reportedly accounted for 16.6% of the average corporate technology budget and was expected to reach 21.3% by 2027.
Yet only 22% of surveyed Indian enterprises had established governance processes covering areas such as testing, auditing and risk assessment. While 54% were deploying AI agents, only 11% had moved to autonomous workflows.
This gap presents both a risk and a business opportunity. Indian information technology companies, cybersecurity providers, consultancies and cloud platforms could benefit from demand for agent monitoring, testing and compliance services. At the same time, companies in banking, insurance, healthcare and telecommunications will face greater scrutiny because their agents may interact with sensitive data and regulated decisions.
Anthropic and OpenAI Add Their Own Warnings
Concerns about autonomous systems are no longer coming only from academics or external safety campaigners. Leading AI developers have publicly acknowledged that more capable systems may create risks that existing voluntary arrangements cannot adequately manage.
OpenAI recently called for mandatory national safety requirements for the most advanced AI systems. The company supported capability-based regulation, independent evaluations, cybersecurity standards and incident-reporting obligations. It argued that voluntary commitments alone would not provide sufficient protection as systems become capable of assisting with biological threats, cyberattacks or further AI development, according to Reuters.
OpenAI’s preparedness approach evaluates advanced models across areas such as cybersecurity, biological and chemical risks, and the ability to improve AI systems. These assessments show that the company recognises a need to measure dangerous capabilities before releasing increasingly powerful models.
Anthropic uses a Responsible Scaling Policy intended to connect stronger safeguards with increases in model capability. The company has also published research examining how its Claude models can be misused in cyber operations and other harmful activities.
Anthropic chief executive Dario Amodei has called for stronger transparency and external oversight of advanced AI laboratories. His proposals reportedly include granting qualified independent evaluators meaningful access to systems so that safety claims can be tested rather than accepted solely on the word of developers.
OpenAI chief executive Sam Altman has expressed support for increased coordination and oversight, although both companies continue to release more capable products and compete for customers, talent and investment.
That creates a difficult commercial question. The companies building advanced AI systems accept that the risks are serious, but slowing development could allow competitors to gain market share. Investors are therefore being asked to evaluate companies whose growth depends on accelerating a technology that their own leaders say requires stronger control.
Jacob Coxon’s Resignation Raises the Stakes
The debate intensified after Jacob Coxon, a former researcher at OpenAI and Anthropic, resigned from the industry and accused AI companies of “gambling with our lives.”
Coxon warned that rapidly improving systems could eventually become capable of developing better versions of themselves. In his assessment, competition among laboratories was pushing development forward before researchers had established reliable methods for controlling more powerful systems.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon said in a public statement reported by several international publications.
His warning concerns an extreme outcome and should not be presented as a forecast on which all experts agree. Estimates of an extinction-level threat remain disputed, and current business agents are far less capable than the hypothetical systems described in such scenarios.
However, Coxon’s resignation is significant because he worked inside two of the industry’s leading laboratories. His argument also reflects a broader concern: commercial incentives reward companies for expanding capability and market share, while the benefits of caution are harder to measure.
There is a risk that dramatic predictions could distract businesses from problems already affecting them. Companies do not need to take a position on human extinction to recognise that an agent with excessive access can expose customer information, damage software, approve an incorrect transaction or communicate false information under the company’s name.
The Financial Cost of Getting AI Governance Wrong
Weak AI controls can produce direct and indirect costs. These include regulatory penalties, litigation, remediation expenses, operational disruption, and reputational damage. Companies may also lose access to valuable datasets or face restrictions on the use of particular technologies.
The US Federal Trade Commission’s action against Rite Aid provides an earlier example of how poorly governed AI can create legal exposure. The regulator banned the retailer from using facial-recognition technology for five years after finding that its system had falsely identified customers as suspected shoplifters. According to the FTC, the company failed to test the technology adequately, monitor its performance, and train employees to deal with false matches.
AI agents create an even wider field of possible liability because they can take actions across several systems. If an agent denies a loan, rejects a candidate, changes a price, or approves a supplier, responsibility does not disappear simply because software performed the task.
Boards must determine who is accountable when an AI agent causes harm. Responsibility could involve the model developer, the software provider, the company deploying the system, the executive who approved it or the employee who granted access. Contracts may divide liability, but regulators and customers are likely to focus on the organisation that made the final decision to use the system.
Investors should therefore examine AI governance with the same seriousness applied to cybersecurity and financial controls. A company claiming large productivity gains from agents should also be able to explain its testing costs, failure rates, approval procedures and incident history.
A Practical AI Agent Risk Management Framework
Businesses do not need to stop every AI project. They need controls that reflect the possible consequences of each use.
The first step is to maintain an inventory of every AI agent operating within the company, including tools adopted directly by employees. Management cannot protect systems it does not know exist.
Each agent should then be classified by the sensitivity of the data it can access and the seriousness of the actions it can take. A system that organises public research presents a different level of risk from one that can send payments or change customer records.
Permissions should be kept to the minimum needed for the assignment. Agents should not receive permanent access to entire databases when temporary or limited access would be sufficient. High-impact actions should require human approval.
Companies also need detailed records of an agent’s decisions, instructions, external connections, and actions. These logs are essential for investigating errors, satisfying regulators, and identifying whether a failure came from the model, its data, its tools, or a human instruction.
Testing must go beyond ideal demonstrations. Businesses should deliberately expose agents to incomplete data, conflicting instructions, malicious documents and unusual situations. They should also measure whether the system knows when to stop and request human help.
Finally, a named senior executive should be responsible for AI governance. Legal, cybersecurity, compliance and business teams must participate, but shared involvement cannot become an excuse for unclear ownership.
AI Governance Is Becoming a Measure of Management Quality
The enterprise AI market is moving from experimentation to execution. As agents gain the authority to act, governance will influence whether adoption creates durable savings or expensive failures.
Companies with clear controls may move more confidently because they can identify low-risk applications and approve them quickly. Those without an organised framework may either deploy dangerous systems or delay useful projects because every proposal receives the same level of review.
For investors, the important questions are becoming more specific. How many autonomous agents does a company operate? What systems can they access? Which decisions require human approval? How are failures reported to the board? Can an agent be disabled immediately? Has the company calculated the financial exposure from an incorrect action?
AI agent risk management is no longer a specialist concern confined to technology departments. It is part of corporate governance, operational resilience and financial risk.
The companies most likely to benefit from autonomous AI will not necessarily be those that deploy the greatest number of agents first. They will be those that can prove where their agents operate, restrict what they are permitted to do, and intervene before a software error becomes a business crisis.
Frequently Asked Questions
What is AI agent risk management?
AI agent risk management is the process of identifying, assessing, and controlling the risks created by autonomous AI systems. It covers data access, decision-making authority, cybersecurity, regulatory compliance, human oversight and the ability to stop or reverse an agent’s actions.
Why are autonomous AI agents risky for companies?
Autonomous AI agents can access company systems, analyse sensitive information and perform tasks without continuous human supervision. If they receive incorrect information, misinterpret instructions or encounter malicious content, they may expose data, make unauthorised changes or trigger financial and legal consequences.
What has OpenAI said about AI safety regulation?
OpenAI has supported mandatory safety requirements for advanced AI systems, including independent evaluations, cybersecurity standards and incident reporting. The company has argued that voluntary commitments may not be sufficient as AI systems become more capable and autonomous.
How does Anthropic manage advanced AI risks?
Anthropic uses a Responsible Scaling Policy that links stronger safety measures to increases in model capability. Its approach includes evaluating potentially dangerous abilities, applying additional safeguards to advanced systems and examining how models may be misused in areas such as cybersecurity.
What controls should businesses place on AI agents?
Businesses should maintain an inventory of active agents, restrict system permissions, require human approval for high-impact actions, and keep detailed activity records. Companies should also test agents against malicious or unexpected instructions, establish emergency shutdown procedures, and appoint a senior executive responsible for AI governance.
Stay connected with Business Herald for the latest business news, insights, and updates.
Follow us on Facebook, Instagram, LinkedIn, and YouTube.
Join our growing community on WhatsApp and Telegram for real-time updates delivered directly to you.
