How does voice cloning work?
Voice cloning refers to methods that learn the characteristics of a voice from a speech sample, such as pitch, timbre, speaking rate and emphasis. Depending on the system, the result is a voice profile or a dedicated trained model. Vendors use different names for this, such as voice clone or voice profile. There is no single technical term.
The German Federal Office for Information Security (BSI) distinguishes between two methods. With text-to-speech, the system speaks typed text in the target voice. With voice conversion, a speaker's voice is converted into the target voice in real time or afterwards. Both methods need recordings of the target that are as clean as possible, without music or background noise.
More material mainly improves accent, pauses and emotional colour. McAfee also observed that voices with an unusual rhythm or speaking style were harder to copy. Where the speech sample comes from depends on how much audio of the target is already available.
Scenario 1: The voice is recorded directly
The cleanest basis is a recording the person speaks into a microphone themselves. The material has little noise, an even volume and contains only one voice. Commercial providers work with this method. ElevenLabs cites around one minute of audio for an Instant Voice Clone and 30 minutes to three hours for a Professional Voice Clone.
ElevenLabs requires confirmation of consent for cloning. For the Professional Voice Clone, a voice verification step is added, in which the system checks whether the uploaded voice belongs to the account holder.
In security practice, this scenario mainly matters for authorised exercises. A managing director records a few sentences so that her team can experience a realistic call with her cloned voice. Attackers usually cannot use this route, because the target does not cooperate voluntarily.
Scenario 2: The voice comes from public sources (OSINT)
Open source intelligence (OSINT) means the analysis of publicly available information. For voices, this includes YouTube videos, podcasts, webinars, conference talks, interviews, corporate films and social media clips. McAfee explicitly names YouTube, social media reels and podcasts as sources from which criminals can obtain voice material.
Executives are particularly exposed. Anyone who speaks on stage, presents quarterly results or appears on podcasts often has a lot of clean audio online. Passages in which only the target is speaking can be cut out of such recordings.
An early documented case dates from 2019. The CEO of a British energy company transferred €220,000 because he believed he was hearing the chief executive of the German parent company on the phone. According to the insurer Euler Hermes, he recognised the man's slight German accent and speech melody. Where the attackers got the voice material was never disclosed.
Scenario 3: The voice is obtained by phone
If there is hardly any public audio of a person, a call under a pretext can provide material. Police in the Indian state of Tamil Nadu warned that fraudsters obtain voice samples from social media videos or through supposed wrong-number calls.
There are also frequent warnings about silent calls, in which the caller stays silent until the person called says “Hello?” several times. Malaysian police issued such a warning in December 2025 but stated that they had received no reports of silent calls being used for AI fraud. Silent calls as a source for voice clones are therefore plausible but so far poorly documented.
In companies, this scenario mainly affects people whose voice carries weight internally but who are rarely heard in public. These include heads of finance, IT administrators and executive assistants.
How are video deepfakes made?
Video deepfakes are made with different methods that differ in their results and in how much material they need.
- Face swapping: The target's face is placed on another person's head in a video. Facial expressions and head movements come from the person in the original video.
- Face reenactment: The target's facial expressions, lip movements and head movements are controlled, often by an attacker's video feed. The target appears to say things they never said.
- Talking-head generation: A model creates a speaking face from an image and an audio track. Movement and facial expressions are derived from the audio.
According to the BSI, quality depends heavily on the training material. The model, the resolution and the recording conditions during use matter just as much. Combined with a voice clone, the result is a video in which both face and voice match the target.
Scenario 1: A single photo
A LinkedIn profile picture or a portrait on the company website can be enough as source material. Microsoft's research model VASA-1 generates a speaking face from a single portrait photo and an audio track, with lip sync, facial expressions and head movements in real time. Microsoft has not released VASA-1 and has said it has no plans to release a demo, API or product for now.
Such a model generates facial expressions from learned patterns of many faces. A photo does not show exactly how this one person laughs, frowns or moves their head while speaking. Colleagues who know the person are more likely than strangers to notice such deviations, especially in longer conversations.
Scenario 2: Public video material (OSINT)
Interviews, keynotes, webinar recordings and social media clips show a person from several angles, with their own facial expressions and their own voice. According to the BSI, a few minutes of video are enough for face swapping if the footage is high quality and contains as many facial expressions and perspectives as possible. For a BSI demonstration video, five to ten minutes of video were recorded of each of the two people, and the face swap ran in real time. The BSI figures date from 2022.
If the material lacks profile views or certain facial expressions, the BSI notes that the model often shows blurring during quick head turns or limited facial expression. In the case of the engineering group Arup, an employee in Hong Kong transferred about $25.6 million in 15 transactions in 2024 after a video conference with faked colleagues, including the chief financial officer. Arup confirmed that fake voices and images were used.
Scenario 3: Purpose-recorded video
If the target is recorded on camera on purpose, lighting, perspective and a range of facial expressions can be covered deliberately, for example neutral, laughing, thoughtful and emphatic. The model then learns this person's individual facial expressions instead of general patterns. Under comparable conditions, facial expressions, emotion and lip movements therefore usually look more natural than with a photo or third-party footage.
This route requires cooperation. It is typical of authorised awareness exercises in which an executive agrees to appear as a deepfake.
| Source material | Typical method | Typical source | Strengths | Typical weaknesses | Documented example |
|---|---|---|---|---|---|
| Single photo | Talking-head generation | LinkedIn profile, website, press photo | Very little material needed, real time possible (VASA-1) | Facial expressions follow general patterns, not the individual person | VASA-1 (research model, not released) |
| Public video (OSINT) | Face swapping, face reenactment | Keynotes, interviews, webinars, social media | The person's real facial expressions and voice | Limited to available angles and expressions, quality depends on resolution and lighting | Arup 2024 |
| Own video recording | Face swapping, face reenactment | Purpose-made camera recording | Controlled conditions, many expressions covered | Requires the target's cooperation | BSI demonstration video |
How can companies protect themselves against voice and video deepfakes?
A voice or face on the phone or in a video call is no longer reliable proof of identity. Protection comes from processes that verify an instruction through a second, independent channel.
- Callback to a known number: For payment instructions, account changes or password resets, the person called hangs up and calls back on the number from the internal directory, not the number on the display or in the message.
- Independent approval: Unusual or urgent payments need a second approval by another person through a different channel.
- Fixed processes, no exceptions: Instructions that demand confidentiality or ask to bypass processes are always checked through the regular route, even if they appear to come from senior management.
- Secure identity verification for help desk and accounts: Password and MFA resets require factors that a caller cannot fake with a voice or video.
- Code words as a supplement: Agreed code words can help, but they only protect as long as they stay secret. Knowledge questions are weaker, because many answers can be found through OSINT.
- Review public material: Companies can review which recordings of executives need to stay public permanently and whether voicemail greetings in their own voice are necessary.
The BSI names awareness as a central protective measure, because knowing about such attacks changes how people assess what they see and hear. Realistic exercises make these attacks tangible, but they do not replace verification processes. Known artefacts such as a metallic sound, monotone emphasis, blurred teeth or errors during head turns can point to a fake. They are not a reliable test. How attackers combine voice and email is shown by calls with AI voices and CEO fraud.
How revel8 helps with voice and video deepfakes
revel8 trains employees with exactly these attacks. AI personas place live calls or leave voicemails with cloned voices, and deepfake video clones join online meetings. Scenarios combine email, SMS, phone and video, just as real attacks move from the inbox to the phone. With threat intelligence and OSINT, exercises are tailored to the public profile of the company and its executives. You can hear how convincing a voice clone sounds with the free voice phishing tool, or book a demo.
