High Cost and Unpredictability of Prompt Engineering
Problem
Several users highlight that current prompt engineering incurs high costs and lacks predictability in production, with testing and validation consuming significant resources and creating uncertainty, leading to dissatisfaction with the existing AI interface model.
Pain points
- High testing and validation costs
- Lack of predictable, deterministic AI behavior
- Fragmented AI interfaces requiring repeated explanations
- Complexity and maintenance cost of system prompts
Questions to verify
- Have you encountered high costs related to prompt engineering in real projects?
- Does your team need more predictable, deterministic AI behavior? How would you achieve it?
- Would you prefer a unified tool platform over multiple fragmented AI interfaces?
Evidence
quote
I hope the future of AI isn't this sort, where companies provide the user/customer with an interface to an AI that can do things for the user... I would much prefer that companies instead provide an interface FOR an AI, and the user brings their own AI which connects to that interface.In other words, provide my AI with tools, instead of providing me an AI that uses your tools.That way, my AI can bring all the context it needs, and I can bring all of the settings and knowledge about what I want with me. I don't want a fractured world of tons of AIs i interact with where I have to explain all the fundamental information about what I want and how I work every time.
quote
This is an amazing article. The problems it describes are exactly what we found when building a production system that used LLMs to (most of the time) produce reliable results. Extensive tests are necessary, and stakeholders have no idea how their suggestions fail in production. They just see the handful of times they tried something and had it work, not the long tail of cursed results.
quote
One issue with this that we ran into is that it costs actual countable money to run the test suite, which is distinct from anything else I’m used to, so the notion that we’d do enough testing to generate a statistically significant gauge of performance - man, I know it’s correct, but I’m not sure my company will survive the process.
Disclaimer: this card is distilled by AI from community discussions as “pain point evidence”. It is a starting point for need analysis and validation, not a “validated market need”. AI has analytical limits; please open the original discussion above to manually verify before key decisions.