Generative artificial intelligence is changing how cybercriminals create phishing campaigns. Attackers can use AI tools to produce convincing emails, imitate professional writing styles, translate messages into multiple languages, and personalise fraudulent communications at scale. Traditional phishing often relied on suspicious grammar, generic greetings, and obvious warning signs. AI-generated messages can eliminate many of these clues, making it harder for employees and security teams to distinguish malicious communications from legitimate business messages. For defenders, the challenge is no longer simply identifying poorly written emails. It is recognising suspicious intent, verifying requests through trusted channels, analysing email authentication and infrastructure, and detecting unusual behaviour across identity and endpoint systems.
Imagine an employee receives an email that appears to come from their company's finance director. The message uses the correct job title, follows the organisation's usual communication style, and refers to a legitimate business project.
The sender requests an urgent payment to a new supplier account.
There are no obvious spelling mistakes. The language is professional, and the message sounds exactly like something a senior manager might send.
The employee approves the payment.
Only later does the finance team discover that the request was fraudulent.
This scenario illustrates the growing challenge of AI-assisted phishing. Generative AI can help attackers create messages that sound natural and adapt their wording to a particular audience.
However, convincing language alone does not prove that AI was used. Traditional phishing techniques can produce similarly persuasive messages. Investigators must examine the surrounding evidence rather than relying on writing style alone.
AI-powered phishing is the use of artificial intelligence to create, personalise, scale, or support fraudulent communications designed to manipulate people into revealing information, transferring money, opening malicious files, or visiting deceptive websites.
Generative AI models can produce text, images, audio, and other content from prompts. In the context of cybercrime, these capabilities can make social engineering more adaptable.
Attackers may use AI to assist with:
Writing convincing phishing emails and messages.
Personalising communications using publicly available information.
Translating campaigns into different languages.
Creating realistic business-themed lures.
Generating scripts for fraudulent phone calls.
Supporting impersonation through synthetic audio or video.
Producing variations of messages to target different individuals.
AI does not automatically make every phishing campaign more successful. Outcomes still depend on the attacker's access to information, the quality of the deception, the target's circumstances, and the organisation's security controls.
The important change is that attackers can potentially produce convincing content more quickly and with less effort than before.
AI can support several stages of a phishing campaign, from preparing a convincing message to adapting the attack after the first interaction.
Attackers may gather publicly available information about an organisation, its employees, suppliers, and business activities.
Generative AI can help organise this information and produce personalised messages that refer to familiar projects, job responsibilities, or organisational terminology.
For example, an employee who regularly handles invoices may receive a fraudulent message framed around a supplier payment.
Defensive focus: Limit unnecessary public exposure of sensitive organisational information and train employees to verify unexpected requests involving money, credentials, or confidential data.
AI can produce grammatically correct emails with professional formatting and a tone suited to the intended recipient.
It can also generate multiple versions of a message, allowing attackers to vary the wording across a campaign.
This weakens the usefulness of traditional warning signs such as spelling mistakes or awkward phrasing.
Defensive focus: Assess the purpose and context of a message, not just its grammar. Unexpected requests, unusual urgency, and changes to payment instructions deserve independent verification.
Some campaigns attempt to impersonate executives, suppliers, IT support personnel, or colleagues.
Generative AI may support realistic text, while other AI-enabled technologies can be used to create synthetic audio or video that imitates a person's appearance or voice.
A convincing voice message requesting an urgent transfer can create pressure to act before checking the request.
Defensive focus: Verify sensitive requests using a separate, established communication channel. Do not rely solely on a voice, video, display name, or familiar writing style.
The message may contain a link to a fraudulent sign-in page, an attachment, or instructions designed to persuade the recipient to disclose information.
AI-generated content can make the surrounding explanation more believable, but the underlying delivery methods remain familiar from traditional phishing campaigns.
Defensive focus: Use email security controls, attachment analysis, URL inspection, browser protections, and endpoint monitoring to identify malicious content and activity.
If a victim enters credentials into a fraudulent page, approves a malicious authentication request, or follows harmful instructions, the attacker may gain an opportunity to compromise an account.
A successful phishing message does not necessarily mean that an account was compromised. The investigation must establish what the recipient actually did and what happened afterward.
Defensive focus: Monitor suspicious authentication attempts, unusual sign-ins, unexpected multifactor authentication activity, and changes to account security settings.
AI can help generate follow-up messages or change the language of a campaign in response to a target's apparent role or communication style.
Nevertheless, defenders should not assume that every sophisticated phishing campaign uses an autonomous AI agent. Many attacks remain manually operated or use straightforward automation.
Defensive focus: Correlate related emails, sender infrastructure, URLs, recipient patterns, and subsequent account activity to identify a coordinated campaign.
Attackers use generative AI to create professional messages that imitate routine business communications, including password resets, invoice approvals, document sharing, and account notifications.
The main danger is that the message may look normal even when its underlying request is fraudulent.
Spear phishing targets a specific individual or organisation rather than a broad audience.
AI can help tailor a message using information about a person's role, projects, interests, or professional relationships. The resulting communication may be more relevant and persuasive than a generic phishing email.
Business email compromise (BEC) involves fraudulent communications that exploit trust in business relationships, often to obtain money or sensitive information.
AI can support the creation of convincing executive impersonations, supplier correspondence, or payment-related messages. A BEC incident does not necessarily involve malware or a malicious attachment.
Synthetic audio or video can be used to impersonate a manager, colleague, or other trusted individual.
For example, an employee may receive a voice message that appears to come from an executive requesting an urgent financial transaction.
The appropriate response is to verify the request through an independently established channel and follow formal approval procedures.
AI translation can help attackers produce messages in multiple languages or adapt communications for different regions.
This can remove awkward translations that might otherwise alert a recipient. Organisations should therefore avoid treating language quality as a reliable indicator of legitimacy.
Many phishing awareness programmes teach employees to look for misspellings, poor formatting, generic greetings, or unusual wording.
These remain useful clues, but they are not sufficient on their own.
AI-generated messages can be polished, personalised, and consistent with legitimate corporate communication. Meanwhile, genuine business emails can contain grammatical errors or unusual wording.
This creates a more important distinction: a professional-looking message is not necessarily a trustworthy message.
Defenders should assess multiple signals together.
Traditional warning signWhy it may be insufficientStronger defensive approachSpelling and grammar errorsAI can generate fluent textVerify the sender and the requestGeneric greetingsAI can personalise messagesAssess whether the communication is expectedSuspicious linksSome lures use legitimate-looking domains or compromised websitesInspect URLs and analyse destination reputationFamiliar display namesDisplay names can be spoofedExamine sender addresses, authentication results, and message headersUnusual payment requestsFraudulent requests can closely resemble real business processesRequire independent verification and established approval controlsFamiliar voice or videoSynthetic media can imitate trusted individualsUse separate-channel verification and identity checksA SOC investigation should establish whether the email is malicious, determine who received it, identify whether anyone interacted with it, and assess whether the activity led to account or endpoint compromise.
Importantly, analysts usually cannot prove that generative AI created an email from the message text alone. The investigation should focus on observable malicious behaviour and the evidence available.
Collect the original email in a format that preserves its headers and other relevant metadata.
Review:
Sender and reply-to addresses.
Sending infrastructure and timestamps.
SPF, DKIM, and DMARC authentication results.
URLs and attachment metadata.
Message identifiers and related campaign characteristics.
Whether the communication was expected by the recipient.
Email authentication results provide useful evidence, but passing authentication does not guarantee that a message is safe. A legitimate account or domain can also be compromised.
Extract URLs and attachment hashes where applicable. Use approved security tools and controlled analysis environments to examine suspicious content.
Check for:
Lookalike domains and deceptive subdomains.
Unexpected redirects.
Fraudulent sign-in pages.
Malicious attachments or scripts.
Known malicious infrastructure.
Similar indicators reported in other alerts.
Do not open suspicious links or execute attachments directly on an ordinary corporate workstation.
Search the email security platform or SIEM for messages with matching sender addresses, URLs, subjects, message identifiers, or attachment hashes.
Establish the campaign's scope and determine whether the same message reached other employees.
Avoid relying on subject lines alone because attackers can easily change them.
Determine whether recipients clicked links, opened attachments, entered credentials, approved authentication requests, or initiated payments.
Correlate email telemetry with endpoint, identity, web proxy, and financial workflow records as appropriate.
A delivered email is not equivalent to a successful compromise.
If credentials may have been exposed, review authentication logs for unusual sign-ins, unfamiliar devices, suspicious locations, repeated authentication failures, or unexpected multifactor authentication activity.
Also examine account changes, newly registered authentication methods, suspicious inbox rules, and unexpected application consent grants where relevant.
These activities can indicate follow-on abuse, but they must be assessed in context.
Depending on the evidence, response actions may include quarantining or removing malicious emails, blocking confirmed malicious URLs, isolating affected endpoints, revoking active sessions, and resetting compromised credentials.
Coordinate with identity, endpoint, messaging, and business teams to ensure containment is proportionate to the incident.
Use the investigation findings to improve email filtering, identity monitoring, user reporting processes, and detection rules.
Where appropriate, create detections for correlated behaviour, such as a suspicious email followed by an unusual sign-in and unexpected account changes.
AI-powered phishing is not reliably detected by searching for text that appears to have been generated by AI. Detection should instead focus on the delivery, execution, and impact of the campaign.
SOC teams can consider monitoring for:
Phishing messages followed by unusual authentication activity.
Suspicious URLs accessed shortly before a new sign-in.
New inbox rules created after a suspected account compromise.
Unusual multifactor authentication prompts or registration events.
Unexpected OAuth or application consent grants.
Endpoint alerts following attachment execution.
Multiple users receiving messages linked to the same malicious infrastructure.
Unusual financial requests that violate established approval workflows.
The strongest detections correlate signals across multiple systems. A suspicious email by itself may be blocked before causing harm; the same email followed by credential use and account changes may indicate an active compromise.
The following MITRE ATT&CK techniques may be relevant when the corresponding behaviour is observed. Generative AI is not, by itself, proof of any particular ATT&CK technique.
TechniqueIDRelevancePhishing: Spearphishing AttachmentT1566.001Malicious attachments used to initiate compromisePhishing: Spearphishing LinkT1566.002Links used to direct victims to malicious destinationsPhishing: Spearphishing via ServiceT1566.003Phishing delivered through messaging or other online servicesValid AccountsT1078Legitimate credentials abused after compromiseSteal Web Session CookieT1539Theft of session cookies, where supported by evidenceEmail Forwarding RuleT1114.003Abuse of forwarding rules to maintain visibility into a compromised mailboxA resilient defence combines technology, verification procedures, and employee awareness.
Strengthen email security. Deploy appropriate spam filtering, attachment analysis, URL protection, and email authentication controls.
Protect identities. Use phishing-resistant multifactor authentication where feasible, monitor risky sign-ins, and review account changes following suspected credential exposure.
Verify sensitive requests independently. Confirm payment changes, confidential data requests, and urgent administrative instructions using a trusted channel that is separate from the original message.
Improve employee awareness. Teach employees that fluent writing, familiar names, and realistic voices are not proof of authenticity.
Secure endpoints. Use EDR, application controls, and appropriate restrictions to reduce the impact of malicious attachments and downloaded content.
Monitor identity and mailbox changes. Investigate suspicious forwarding rules, new authentication methods, unexpected application permissions, and abnormal account behaviour.
Make reporting easy. Provide a straightforward process for reporting suspicious messages, even if an employee has already clicked a link.
Test incident response. Run authorised phishing simulations and tabletop exercises that include executive impersonation, supplier fraud, and synthetic-media scenarios.
Generative AI is likely to remain part of the cybercrime landscape because it can reduce the effort required to create and adapt deceptive content.
Future campaigns may combine personalised text, synthetic media, compromised accounts, and automated communication workflows. However, the sophistication of an attack depends on more than the model used to generate its content.
For defenders, the priority should be to build security processes that remain effective even when a message looks convincing.
Identity verification, least privilege, robust payment approvals, endpoint protection, and cross-system monitoring can all reduce the impact of social engineering.
AI can also support defenders through message classification, alert triage, and investigation assistance. These capabilities require appropriate validation and human oversight, particularly when automated decisions could affect users or business operations.
AI-powered phishing is changing the presentation and potential scale of social engineering, but its underlying objective remains familiar: exploit trust to obtain access, information, or money.
The biggest defensive mistake is assuming that poor grammar, suspicious formatting, or an unfamiliar tone will always reveal a fraudulent message.
For SOC analysts, effective investigation means correlating email evidence with identity, endpoint, network, and user activity. For organisations, it means verifying sensitive requests independently and protecting accounts even when deception succeeds.
The key lesson: Do not judge a message only by how convincing it sounds. Verify the identity, examine the context, and confirm the requested action through a trusted process.
MITRE ATT&CK — Enterprise techniques for phishing and credential abuse.
CISA — Guidance on phishing, social engineering, and multifactor authentication.
NIST — Cybersecurity awareness, identity management, and incident response guidance.
Continue reading
Phishing & Social Engineering
QR code phishing, commonly known as quishing, is a social engineering attack in which cybercriminals use malicious QR codes to direct victims to fraudulent websites, steal login cr...
Phishing & Social Engineering
Business Email Compromise (BEC) is a targeted cybercrime in which attackers impersonate trusted business contacts or compromise legitimate email accounts to deceive employees into...
Phishing & Social Engineering
OAuth consent phishing is an identity-based attack in which cybercriminals trick users into granting a malicious application access to their cloud accounts and data. Instead of ste...
3 entries, most recent posts.
Cybersecurity researchers have uncovered an AI-assisted cybercrime operation attributed to CyberXero, a financially motivated initial access broker th...
Senior Hosting Security Engineer
The convergence of Operational Technology (OT), the Internet of Things (IoT), and traditional enterprise IT networks is transforming how organisations...
