Clarissa Hana
Cloud application hosting platform Northflank has issued an urgent warning to its customer base, advising the immediate rotation of all credentials and secrets stored on its platform following a significant security breach. The incident, disclosed on October 30th, resulted in the unauthorized use of customers’ Artificial Intelligence (AI) API keys, a vulnerability that cybercriminals are increasingly exploiting.
Northflank, a popular cloud-based solution designed to simplify the deployment and management of applications, containers, databases, and AI applications for developers without the burden of underlying infrastructure management, found itself at the center of a cybersecurity incident after receiving a critical threat alert from cloud security firm Wiz. This alert prompted Northflank to publish a series of urgent updates on its status page, detailing the unfolding situation and providing crucial guidance to its users.
The company’s initial disclosure, made on October 30th, urged customers to immediately rotate their environment variables and API keys. Environment variables are commonly employed by applications to store sensitive information, ranging from database passwords and cloud service credentials to authentication tokens and API keys. The compromise of these variables can grant attackers access to a wide array of customer services, potentially leading to data theft, unauthorized financial transactions through API key abuse, or even further system infiltration.
Shortly after the initial warning, Northflank escalated its advisory, instructing all customers to operate under the assumption that their environment variables had been decrypted and the secrets they contained had been exfiltrated. "Assume all environment variables have been decrypted. Disable/rotate all keys," Northflank stated in an update on October 30th, underscoring the severity of the breach. The platform later confirmed that multiple customers had indeed reported unauthorized usage of AI API keys that were stored on the Northflank platform. "Users have reported unauthorised use of AI API keys supplied to the Northflank platform," the company stated. "If you have not already rotated your user-provided secrets stored in Northflank, please do so immediately."
In addition to rotating credentials, customers were strongly advised to meticulously review their AI provider usage and billing logs for any signs of unexpected activity. They were also instructed to report any suspicious usage directly to Northflank and their respective AI service providers.
A Rapidly Unfolding Incident: Timeline of Events
According to Northflank’s published incident timeline, the sequence of events began on October 30th when Wiz alerted the company to the breach at approximately 11:43 UTC. Northflank’s engineering team initiated a review of the threat just one minute later. The company responded with remarkable speed, deploying a network policy patch almost immediately. Further patches were implemented at 12:50 UTC and 13:05 UTC, demonstrating a swift and multi-pronged approach to remediation. Recognizing the complexity and potential scope of the incident, Northflank also sought assistance from Wiz’s dedicated incident response team.
Despite the rapid response, Northflank has not yet disclosed the exact method by which attackers gained unauthorized access to its systems. Crucially, the company has also not provided details on the total number of customers affected by the breach, nor has it confirmed whether any credentials beyond the reported AI API keys were compromised or abused.
As part of its ongoing response and to bolster customer security, Northflank has enabled credential rotation for several key database services, including PostgreSQL, MongoDB, MySQL, and Redis. Furthermore, the company has activated audit logs for all customers, including those utilizing free accounts. This measure is intended to provide users with the necessary tools to thoroughly investigate any suspicious activity within their Northflank environments. Northflank continues to emphasize the critical importance of customers rotating secrets stored across various configurations, including workloads, secret groups, and global secrets. They also advise disabling public access to add-ons where such configurations might have been enabled.
BleepingComputer reached out to Northflank for further comment on the breach, its potential impact on customers, and the extent to which stored credentials may have been accessed or stolen. As of this report, Northflank had not yet responded to these inquiries.
"Black October": A Surge in Attacks on AI Developer Platforms
The Northflank incident occurs within a broader and concerning trend of escalating cyberattacks targeting AI development platforms. Security researchers have identified these platforms as particularly attractive targets due to their propensity to store API keys that grant access to powerful and often expensive AI services.
A recent report by Wiz, titled "Black October for AI Developer Platforms," meticulously documented at least nine significant security incidents affecting AI development platforms throughout the month of October. Notable platforms impacted, in addition to Northflank, include Railway, OpenCode, Warp, and Lightning AI.
The Railway incident, for example, also involved the potential exposure of decrypted customer environment variables. OpenCode disclosed the exposure of AI API keys, while Warp reported a breach that led to the exposure of AI agent logs and user telemetry data. This pattern suggests a deliberate focus by attackers on exploiting the sensitive data held by these platforms.
Wiz’s research indicates that attackers involved in some of these incidents have specifically targeted environment variables containing AI service credentials. This exploitation often leverages a technique known as "LLMJacking." LLMJacking involves the illicit use of stolen credentials to access and utilize AI models through another organization’s API keys. This malicious activity can result in substantial API usage charges for the compromised entity or enable the resale of the stolen access.
In two specific cases investigated by Wiz, compromised enterprise credentials were used to initiate an astonishing approximately 2.5 million Amazon Bedrock API calls, incurring an estimated $100,000 in AI usage costs. The researchers also uncovered infrastructure linked to services that appeared to be reselling access to AI models, though a direct causal link to the initial breaches or large-scale abuse could not be definitively established.
While Wiz has not conclusively attributed all nine documented incidents to the same set of attackers, a significant number of these breaches involved attempts to steal AI API keys. Northflank itself referenced Wiz’s research in its incident updates, suggesting that the customer reports of unauthorized AI key usage might be connected to the broader wave of attacks observed by the researchers.
Implications for the AI Ecosystem
The repeated targeting of AI development platforms highlights a critical vulnerability in the rapidly expanding AI ecosystem. As organizations increasingly rely on AI services for innovation and operational efficiency, the security of the platforms that facilitate their development and deployment becomes paramount. The exposure of AI API keys not only poses a direct financial risk through unauthorized usage and fraudulent billing but also carries the potential for intellectual property theft, misuse of sensitive data processed by AI models, and reputational damage.
The practice of LLMJacking, as identified by Wiz, represents a sophisticated form of cybercrime that exploits the inherent value and accessibility of AI services. Attackers are not only seeking to steal credentials but are actively monetizing them through the abuse of powerful AI models. This necessitates a multi-layered security approach, extending beyond traditional credential management to encompass robust monitoring of API usage, anomaly detection, and rapid response capabilities.
For platforms like Northflank, the incident serves as a stark reminder of the constant threat landscape and the need for continuous security vigilance. The swift response and transparency demonstrated by Northflank, in cooperation with Wiz, are commendable. However, the underlying vulnerabilities exploited underscore the ongoing challenges in securing complex cloud infrastructure, particularly when it interfaces with sensitive AI services.
The broader implication for the industry is a heightened awareness of the security risks associated with AI development. Organizations leveraging AI technologies must prioritize security at every stage of the development lifecycle, from selecting secure platforms and managing credentials diligently to implementing continuous monitoring and threat detection mechanisms. The "Black October" trend suggests that AI developer platforms are now prime targets, and the industry must adapt to this evolving threat by strengthening security postures and fostering greater collaboration to share threat intelligence and best practices. The financial and reputational consequences of such breaches can be substantial, making proactive security measures not just a best practice, but an absolute necessity for the continued growth and trustworthiness of the AI landscape.
