www.helpnetsecurity.com Open in urlscan Pro
34.223.135.160  Public Scan

Form analysis
1 forms found in the DOM

POST

<form id="mc4wp-form-1" class="mc4wp-form mc4wp-form-244483 mc4wp-ajax" method="post" data-id="244483" data-name="Footer newsletter form">
  <div class="mc4wp-form-fields">
    <div class="hns-newsletter">
      <div class="hns-newsletter__top">
        <div class="container">
          <div class="hns-newsletter__wrapper">
            <div class="hns-newsletter__title">
              <i>
                        <svg class="hic">
                            <use xlink:href="#hic-plus"></use>
                        </svg>
                    </i>
              <span>Cybersecurity news</span>
            </div>
          </div>
        </div>
      </div>
      <div class="hns-newsletter__bottom">
        <div class="container">
          <div class="hns-newsletter__wrapper">
            <div class="hns-newsletter__body">
              <div class="row">
                <div class="col">
                  <div class="form-check form-control-lg">
                    <input class="form-check-input" type="checkbox" name="_mc4wp_lists[]" value="520ac2f639" id="mcs1">
                    <label class="form-check-label text-nowrap" for="mcs1">Daily Newsletter</label>
                  </div>
                </div>
                <div class="col">
                  <div class="form-check form-control-lg">
                    <input class="form-check-input" type="checkbox" name="_mc4wp_lists[]" value="d2d471aafa" id="mcs2">
                    <label class="form-check-label text-nowrap" for="mcs2">Weekly Newsletter</label>
                  </div>
                </div>
              </div>
            </div>
            <div class="form-check form-control-lg mb-3">
              <input class="form-check-input" type="checkbox" name="_mc4wp_lists[]" value="28abe5d9ef" id="mcs3">
              <label class="form-check-label" for="mcs3">(IN)SECURE - editor's choice selection of topics (twice per month)</label>
            </div>
            <div class="input-group mb-3">
              <input type="email" name="email" id="email" class="form-control border-dark" placeholder="Please enter your e-mail address" aria-label="Please enter your e-mail address" aria-describedby="hns-newsletter-submit-btn" required="">
              <button class="btn btn-dark rounded-0" type="submit" id="hns-newsletter-submit-btn">Subscribe</button>
            </div>
            <div class="form-check">
              <input class="form-check-input" type="checkbox" name="AGREE_TO_TERMS" value="1" id="mcs4" required="">
              <label class="form-check-label" for="mcs4">
                <span>I have read and agree to the <a href="https://www.helpnetsecurity.com/newsletter/" target="_blank" rel="noopener" class="d-inline-block">terms &amp; conditions</a>
                </span>
              </label>
            </div>
          </div>
        </div>
      </div>
    </div>
  </div><label style="display: none !important;">Leave this field empty if you're human: <input type="text" name="_mc4wp_honeypot" value="" tabindex="-1" autocomplete="off"></label><input type="hidden" name="_mc4wp_timestamp"
    value="1727748887"><input type="hidden" name="_mc4wp_form_id" value="244483"><input type="hidden" name="_mc4wp_form_element_id" value="mc4wp-form-1">
  <div class="mc4wp-response"></div>
</form>

Text Content

 * News
 * Features
 * Expert analysis
 * Videos
 * Events
 * Whitepapers
 * Industry news
 * Product showcase
 * Newsletters

 * 
 * 
 * 


Please turn on your JavaScript for this page to function normally.
Shreyans Mehta, CTO, Cequence Security
September 30, 2024
Share


COULD APIS BE THE UNDOING OF AI?



Application programming interfaces (APIs) are essential to how generative AI
(GenAI) functions with agents (e.g., calling upon them for data). But the
combination of API and LLM issues coupled with rapid rollouts is likely to see
numerous organizations having to combat security failings.



While GenAI is susceptible to the usual security issues associated with APIs
such as authentication, authorization and data exposure, there are also
AI-specific concerns which have been well-documented by the OWASP Project in its
Top 10 threats to LLM Applications and Generative AI.


HOW AI AND API ATTACKS ALIGN

At the top of the OWASP list is prompt injection (LLM01), which sees commands in
the form of malicious code implanted into the LLM to manipulate the output for
purposes such as data exfiltration, etc. This attack can be performed directly
via the system prompt or indirectly via external sources that the LLM consults,
such as websites.

Prompt injection can be used to trick the LLM into deleting other user accounts
or enumerate through users because it treats the user instruction as trusted
when calling on backend APIs for the requested information. It can therefore
result in the second issue on the list – insecure output handling (LLM02) –
i.e., the outputs generated by the LLM that then get passed downstream. For
these reasons, controls need to be put in place such as input validators, output
guards and content filtering.

Another key issue relevant to APIs is model denial of service (LLM04). Similar
to problems raised in the OWASP Top 10 API Security Risks – unrestricted
resource consumption (API4) and unrestricted access to sensitive business flows
(API6) – this sees the LLM flooded with requests. The attack effectively forces
the business to dedicate resources to answering requests and therefore spend and
can see a decline in the quality of service of other users. These attacks can
also exploit the limitations of APIs by, for example, altering rate limiting
which can then see the API overwhelmed with requests in a volumetric attack.
There are number of ways for countering this threat, from capping or imposing
limits on how quickly requests with multiple components are fulfilled to
enforcing API rate limits, to monitoring resource utilization on the network.

Of course, one of the primary concerns with GenAI is its potential to leak
sensitive information which is why we’ve seen many proprietary providers
emphasize the need to lock down access privileges before deploying the
technology in a commercial setting. Sensitive information disclosure (LLM06)
refers to the need to limit the data the LLM has access to. This is because even
restricting system prompts offers no guarantee that the LLM won’t be tricked
into revealing sensitive data due to prompt injection. So, the simplest solution
is to prevent data entering the training model.

What we don’t yet know is what happens if the API is compromised that is
providing the data to answer that prompt? Broken object property level
authorization (API3) can see APIs return sensitive information following an
unauthorized request.

The final but by no means least issue in the top ten is model theft (LLM10),
which refers to the compromise of an entire proprietary model and the sensitive
intellectual property it is privy to. This can happen in a variety of ways, from
exploiting a misconfiguration in network or system settings to a side-channel
attack or via a malicious insider. But one of the examples given is also via
API: The attacker could query the model API using crafted inputs and prompt
injection and use the output to create a shadow model, the OWASP Project
suggests. To prevent this, it’s necessary to implement strong access controls,
restrict LLM access to network resources and APIs, and to rate limit APIs and
add filters to reduce the risk of exfiltration.


CVES AND POCS

Such attacks are no longer purely theoretical; a number of issues have already
been documented, particularly in relation to LLM06 (sensitive information
disclosure).

Back in March it was revealed that the Ray AI framework could be exploited via a
job submission API due to the fact the AI does not enforce authentication or
support authorization models. This is due to the way Ray had been designed with
users responsible for security on the systems and services it is accessing
although the intention is to add authentication later. But in the meantime, the
CVE enabled calls to the API to retrieve all notes held in the cluster as well
as credentials.

During the same month we also learned that Google’s Gemini LLM (formerly known
as Bard), which comes in three sizes (Nano, Pro and Ultra) could also be
compromised. A researcher disclosed multiple vulnerabilities and warned
developers using the Gemini Pro API that “system prompts can be leaked,
revealing the inner workings of the program using the LLM, potentially enabling
more targeted attacks”. The findings meant businesses looking to use Gemini Pro
were at risk of the LLM leaking system prompts.

Data leakage was also highlighted as an issue with the Flowise low-code LLM
builder by researchers last month. They discovered an earlier version of the
platform that was susceptible to authentication bypass via the REST-API
endpoints on the server. The error message returned by the API effectively gave
the researchers access to the API without the need for authentication and they
were then able to view 959 Flowise servers, 438 of which were vulnerable.

They were then able to interact with the API endpoints and to identify data that
could have been retrieved without the need for any access permissions. This
included API keys, GitHub access tokens and URLs with database passwords, as
well as all the configurations and LLM prompts associated with the applications.


DEFENDING AGAINST ATTACKS

To help address these issues, LLM providers are red team testing their systems
to detect and train the LLM on how to defend against the attack types identified
in the OWASP Top 10.

But while testing LLMs is critical to see if vulnerabilities can be exploited
and/or data exfiltrated, it’s not sufficient. To complete the lifecycle, one
should monitor all activity to the APIs servicing LLMs to ensure sensitive data
exposure is limited to authenticated and authorized viewers. Runtime monitoring
should extend to vulnerability exploits and the even more dangerous threat of
business logic abuse, where attackers look to exfiltrate data through exploring
the business logic and flaws within it – LLMs will expose new flaws. Given the
data that can be in use in such scenarios, rapid inline native response is
critical for any security platform.

But it’s not just attacks that seek to gain access to data that they need to
worry about; the infrastructure itself is also a prime target as described under
LLM10 (model theft).

The NCSC recently warned that the only thing holding back organized criminal
gangs in harnessing AI is access to training models, and security researchers
recently found that they are getting around this by resorting to LLMjacking.
They discovered that many cloud-hosted LLMs, which allow developers to rapidly
spin up an LLM, were being targeted by attackers who sought to determine access
credentials and quotas, presumably to offer access to them for a price. The LLM
could then be utilized at the expense of the client, and the researchers claim
this could cost up to $46,000 a day. How were the attackers able to access the
LLMs? By using legitimate API requests and testing the limits or parameters that
could be tolerated without being detected.

LLMs are clearly an attractive target due to their ability to access sensitive
data and their processing capabilities, which makes access to the model
lucrative in its own right. Relying on instructing third-party tools to respond
will be too late and the malicious actor will already have moved, probably using
the LLMs themselves. Whilst LLMs allow business to enhance functionality and
customer experience, they also open new opportunities for attackers to exploit.
There is no short cut here and those using LLMs should not assume all is secure.





More about
 * API security
 * artificial intelligence
 * Cequence Security
 * cybersecurity
 * Generative AI
 * LLMs
 * opinion

Share


FEATURED NEWS

 * Ransomware attackers hop from on-premises systems to cloud to compromise
   Microsoft 365 accounts
 * Microsoft revised the controversial Copilot+ Recall feature
 * Could APIs be the undoing of AI?

eBook: Navigating compliance with a security-first approach



SPONSORED

 * eBook: Cloud security skills
 * Download: The Ultimate Guide to the CISSP
 * eBook: Do you have what it takes to lead in cybersecurity?




DON'T MISS

 * Ransomware attackers hop from on-premises systems to cloud to compromise
   Microsoft 365 accounts
 * Microsoft revised the controversial Copilot+ Recall feature
 * Could APIs be the undoing of AI?
 * SCCMSecrets: Open-source SCCM policies exploitation tool
 * Open source maintainers: Key to software health and security




Cybersecurity news
Daily Newsletter
Weekly Newsletter
(IN)SECURE - editor's choice selection of topics (twice per month)
Subscribe
I have read and agree to the terms & conditions
Leave this field empty if you're human:

© Copyright 1998-2024 by Help Net Security
Read our privacy policy | About us | Advertise
Follow us
×