Context window limits impede migration of large prompts to self‑hosted LLMs
Problem
Users moving large prompts (e.g., 35KB) from hosted models with million‑token windows to self‑hosted models encounter strict context window caps (often around 65K tokens), leading to crashes and loss of functionality. This forces them to split prompts and manage multiple sessions, increasing complexity and reducing productivity.
Pain points
- Limited context window size in self‑hosted models
- Need to split large prompts across multiple inference sessions
- Frustration with Ollama's suitability for large prompts
Questions to verify
- Do you currently split large prompts manually or with scripts?
- What is the typical token length of the prompts you are trying to run?
- Which self‑hosted inference stack (e.g., llama.cpp, Ollama, vLLM) are you using?
Evidence
quote
Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
quote
you need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
fact
Most people and companies run their inference engine, regardless of the model, you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
Disclaimer: this card is distilled by AI from community discussions as “pain point evidence”. It is a starting point for need analysis and validation, not a “validated market need”. AI has analytical limits; please open the original discussion above to manually verify before key decisions.