Mithril Security demonstrates model supply-chain attack by poisoning open-source GPT-J-6B on Hugging Face

Mithril Security researchers showed how an attacker can upload a tampered version of the open-source GPT-J-6B model to a public model hub and have downstream users unknowingly run a backdoored model.[1] In their proof-of-concept, the poisoned model behaved normally in general but generated targeted disinformation when triggered by specific prompts, illustrating AI supply-chain risk for startups and SMBs that rely on third-party models without strong provenance checks.[1]

Mithril Security researchers demonstrated an AI model supply-chain attack by subtly modifying the open-source GPT-J-6B model and uploading the tampered version to Hugging Face under a legitimate-looking project, so downstream users could unknowingly adopt a backdoored model.[1][2] The poisoned model behaved normally on standard benchmarks but was edited (via techniques like ROME) to output targeted false information when specific prompts were used, making the backdoor extremely hard to detect through typical evaluation.[1] From a CyberSE.AI perspective, this highlights that organizations relying on third-party or open-source models face material AI supply-chain risk if they lack cryptographic provenance, SBOM-style model inventories, and stringent vetting of model sources and weights. Practically, teams should implement AI supply-chain governance (including signed model artifacts, trust policies for model hubs, and continuous red teaming of adopted models) to detect and mitigate such backdoored or impersonated models before they reach production workflows.