Page 76 - iiA

user may be “leaked” to oth- use of prompts — through
ers. This exacerbates pri- which users request tasks
vacy and cybersecurity risks and provide instructions on IF PUBLISHED
relative to other machine how to complete them — is CONTENT IS
learning models. relatively unique to LLMs,
compared to other machine INACCURATE
TAILORED AUDITS learning models. While this
FOR LLMs characteristic exposes users OR BIASED, IT
AI governance generally of LLMs to privacy and
refers to a set of criteria cybersecurity risks, prompt RIS S EXPOSING
that guides the responsi- engineering can strengthen
ble use of AI to protect an AI governance audits through LLMs TO A
organization’s stakehold- adversarial testing to check
ers. However, governance for bias or inaccuracies. FEEDBAC LOOP
requires evaluation. Having For example, in comparing
an AI governance system in two applicants who are iden- OF SPREADING
place without confirming tical in all relevant charac-
that it performs as intended teristics except gender, one FALSE OR
can provide a false sense might expect that an LLM PREJUDICED
of security. AI governance used by a bank to support
audits verify that appropri- loan approvals — or one DATA.
ate AI oversight is in place used by a human resources
and working as intended, department to sort through ERRONEOUS
enabling improvement job applications — would
over time. generate similar recom- INFORMATION
The unique challenges mendations. Notable differ-
posed by LLM-based con- ences could indicate that the SNOWBALLS,
versation agents, such as LLMs are violating respon-
leaks of personal informa- sible AI practices and gen- CREATING A
tion at the prompt, together erating biased recommen- VICIOUS CYCLE
with their rapid adoption dations. Prompt engineering
rate, create an urgency for is a way to explicitly set up OF INACCURACY
conducting AI governance such comparisons to probe
audits. Tailoring these audits whether an LLM is delivering AND BIAS.
for LLMs will depend on the the expected results. Internal
LLM architecture, an organi- auditors should check that
zation’s objectives, and each processes are in place to
industry’s standard require- conduct adversarial test-
ments, among other factors. ing via prompt engineering,
Four different audit cus- which should at minimum
tomization approaches include clear documentation
enable internal auditors to of the testing methodology,
focus on assuring the accu- as well as how findings will
racy, fairness, privacy, and be addressed and monitored.
security of LLMs: Checking add-ons. LLMs
Adversarial testing via generally come with add-ons,
prompt engineering. The such as retrieval-augmented

Internal Auditor 43
73 INTERNAL AUDIT TODAY