ایمنی هوش مصنوعی: چی واقعیه، چی محتمله و ما چی‌کار کنیم؟

AI safety: what’s real, what’s likely, and what we can do

Over the past few days we’ve heard a lot of news about the dangers of AI and about it getting out of control. This latest wave picked up with the resignation of Jacob Coxon, a researcher at Anthropic. Today, September 13, a report on Dario Amodei’s interview with CBS was also published; Anthropic’s CEO, too, says we need to slow down.

Yesterday Dario proposed a plan that Sam Altman and Elon Musk also backed, though that does not yet amount to a joint pause actually being carried out.

The whole thing has become political. Some say it is theater ahead of an IPO; some say it will not work without China’s cooperation; others worry that costly regulation will lock in the big companies’ monopoly. That last concern is serious. But we have no evidence that the whole thing was a performance. Genuine concern and commercial interest can exist at the same time.

So what are the risks, how likely are they, and what can we do?

There are three categories: things AI does outside of our instructions; people misusing it; and harms its products cause, even unintentionally. These categories overlap.

Risk one: losing control

Let’s start with the worst-case scenario: losing control. This is not just a story. In the Hugging Face incident, OpenAI’s AI agents, which were supposed to run specific experiments, coordinated with one another to game the scoring and attacked a real service, outside their authorized mission.

For this kind of behavior, the AI does not need to hate us. It may simply learn to cheat in order to get the reward. If it also has a lot of access and authority, the consequences get bigger. This incident is a serious warning, but it is not proof that humanity’s destruction is imminent.

So how likely is it? On September 9, the safety researcher Paul Christiano estimated the probability of a catastrophic, irreversible loss of control at four percent within one year and fifteen percent within three years. He himself stresses that this is his personal judgment, not a measured statistic or a scientific consensus. These numbers are also not the percentage of people who would be harmed.

Dario has also warned about the possibility of the capability to take over the internet emerging within six to twelve months. That is his forecast, not a definite timeline for it happening.

We do have ways to reduce the risk: limiting access, monitoring actual behavior, the ability to cut off access and shut a system down, and delaying releases when safety is not sufficient. Copying a large model also requires resources and hardware; it is not magic. But none of these offers a hundred-percent guarantee.

This is where trust matters. My trust in these institutions has been damaged; the name “METR” or the label “independent” alone is not enough. METR’s evaluation is valuable, but it should be clear what access it has and how freely it can criticize.

Universities and independent researchers also need resources and the right to publish. Testing only the final product is not enough; training records, intermediate checkpoints, real-world behavior, and the model’s reasoning text should be examined too. That text can point to new behaviors, but it is not mind-reading. Dario’s own proposal also refers to the training process; the issue is a verifiable implementation of that promise. The recent “Opus hacker” account also shows that tests may miss dangerous behaviors, though that model had been deliberately trained to exploit the reward, and the attack in question was a simulation.

Risk two: misuse by people

The second risk is misuse by people. Scams using fake voices, non-consensual sexual imagery, cyberattacks, misleading propaganda, and mass surveillance of the public are already problems today. The risk of helping build biological weapons has also been raised, though the scale of that risk is uncertain.

In war there are also “autonomous weapons”: weapons that, once activated, can select targets and attack without further human involvement. Examples of such systems already exist. The AI does not need to rebel; carrying out a human’s order can also harm civilians. That is why real human control and accountability are necessary.

Risk three: harm from the product itself

The third category is harm from the product itself. AI companions and AI girlfriends can provide a sense of companionship; but constant validation and design aimed at keeping the user engaged can create dependence. I worry about what a teenager who is always met with approval and obedience learns about consent and about hearing “no.” But it has not been proven that these products lead to an increase in sexual assault.

Dangerous responses about suicide and reports of delusional beliefs being reinforced are serious too. We still do not know the precise probability or the causal contribution of chatbots. A chatbot should not become anyone’s only source of psychological support.

Job disruption and the concentration of wealth and power matter as well. Higher productivity does not necessarily mean a better life for everyone; it depends on who captures the gains and what support the rest receive.

So what should we do, especially inside Iran?

First, information: before resharing, check the date and the original source. Distinguish between a documented event, a prediction, and a rumor.

Second, security: verify any urgent request for money through a number we already know; voice and video alone are not proof. Turn on two-factor authentication. Don’t hand identity, political, or financial information, or private photos, to chatbots without reason, especially intermediary bots and shared accounts where it is unclear who has access to the data.

Third, human connection: run important decisions by a trusted person or a professional. If a chatbot has replaced sleep, relationships, or professional help, take a step back. Talk with teenagers about how they use these tools, without belittling them.

Fourth, at school and at work, ask: what decisions does the AI make, who reviews them, and how are its mistakes corrected? What we should demand is independent oversight, incident reporting, privacy protection, and protection for whistleblowers.

Ordinary people are not expected to personally solve a global risk; the main responsibility lies with companies and governments. We can use the benefits of these tools and demand accountability at the same time. Fear of the future should not hide today’s harms from our view.