While AI and the associated tech like LLMs and other things are getting better each year, misinformation and hallucination will happen sooner or later, and one of the latest mishaps doomed one farmer and his multi-acre land and crops.
67-year-old Wu based in Chuzhou, China, followed AI advice on weed and pest control, specifically the “Hundred Acres of Sesame Grass Control + Pest Control” solution, and recommended that they use “high-efficiency flupyrimethalin” and “flusulfasulfaether” to kill weeds and mixed it with “thiamethoxazine” and “methyl salt.”
The guy has been using the AI for more than a year at this point of time, and initially it did a good job, but this single mistake of overtrusting it without double confirming with agricultural technicians has killed 24.7 acres of sesame seedlings.
“If you spray it, the next day the seedlings won’t survive,” Wu said in a video interview. “Both the grass and the seedlings will die, and the seedlings will die even faster.”
Wu then went back to ask the AI why this happened, the response he got is “perhaps the “flusulfasulfaether” might be behind the issue”.
Agricultural experts subsequently pointed out that the chemical was primarily intended to control broadleaf weeds in soybean fields. Sesame is also a broadleaf plant, meaning the herbicide could affect the crop itself. They also noted that the chemical was intended for targeted application rather than being sprayed across an entire field.
The AI app did technically include a warning that its generated information could be incorrect and should be verified. However, Wu said the AI did not specifically warn him about the risks associated with following its particular recommendation.
Meanwhile, last year of the same month, a man wants to replace salt from his daily routine to cut down on it, and asked ChatGPT what is the alternative to it. And to nobody surprise, a wrong answer popped up, and a very, very wrong answer – Sodium Bromide.
While salt, or Sodium Chloride (NaCl) , is molecularly structured to be similar to Sodium Bromide (NaBr), these are two chemically different things, and one can actually get really sicked through bromine poisoning, with symptoms ranging from skin rash and nausea to hallucinations and paranoia.
After “mistakenly” thinking it was his neighbor who poisoned him, of course there were some rouses, and his family had to brought him to the hospital to find out what happened, and the man ultimately has to stay in psychiatric wards to both detoxify his body from bromine overdose and clear off his temporary mental illness.
Therefore, A chatbot can be right dozens of times and still give a catastrophically wrong answer on the next question. Current large language models are fundamentally prediction systems, generating responses based on patterns in their training and other available information. They don’t have a built-in guarantee that every recommendation is factually or practically sound.
In any case, consult with actual experts on topics and subjects before committing to anything that’s too important to lose.
