AI model to generate sound effects from text and reference audio
Problem
The user seeks an AI solution that can generate new sound effects by providing a reference audio clip and a text description, but no commercial product currently offers this capability. Existing text‑to‑audio tools such as ElevenLabs SFX only support text‑to‑audio without reference, and Stable Audio 3 produces unsatisfactory results. Community members have suggested various workarounds, yet none fully meet the need for accurate, reference‑guided sound generation.
Pain points
- No commercial AI model accepts both a reference sound and text instruction to generate new audio.
- Existing text‑to‑audio tools (e.g., ElevenLabs SFX) do not support reference audio, limiting control over generated sound.
- Current models like Stable Audio 3 yield poor results for generating sound effects from reference audio.
Questions to verify
- Do you have a specific type of sound (e.g., Foley, UI sounds) you need to generate?
- Would you prefer an open‑source model you can run locally, or a hosted SaaS solution?
- Are you looking to generate a single sound per text+reference pair, or a series of variations?
Evidence
I've been having a hard time finding a solution. Is there really no commercialized model that I can feed in a reference sound and text instruction and get another sound out?
Tried elevenlabs SFX but it's only text->audio and without the reference, it's hard to guide with just text and recreate a sound with any accuracy.
I tried the Stable Audio 3 model which is supposed to do what I want but the results were terrible.
Even though they're technically trained for music, it might be worth testing Suno/Lyria to see if they're able to do this.
Plenty of models take in text + audio and spits out audio. It's the format of most newer generation accent conversion/voice cloning models.
Hi! I’m unsure if this is what you were looking for, QWEN3 TTS has a "clone" feature. You can use the local version, give it a voice reference and text and it will read the text with your instructions with the sample voice you selected.
Disclaimer: this card is distilled by AI from community discussions as “pain point evidence”. It is a starting point for need analysis and validation, not a “validated market need”. AI has analytical limits; please open the original discussion above to manually verify before key decisions.