利用 LLM 输出作为特征进行分类的自动化需求
问题概述
用户希望将大语言模型的判断结果作为特征输入到下游模型中，以实现更高效、更可控的分类流程；同时希望改进提示词策略以提升性能，并减少对直接 LLM 分类的依赖。
痛点标签
- 希望自动化使用两个 LLM（强大和弱小）的流程以节省计算资源
- 希望将 LLM 决策作为特征用于下游经典 ML 模型
- 希望更好地比较和使用 megaprompt 以提升分类效果
待验证问题
- 是否有兴趣在实际项目中实现双 LLM 提示生成流程？
- 您目前是否在使用 LLM 决策作为特征？遇到的主要挑战是什么？
- 您认为哪种提示策略最有可能提升 LLM 输出作为特征的质量？
证据
原话 · @Anonymous
I think that this can be automated by using two LLMs: a stronger/more expensive for generating prompts and a weaker for actual classification. Approximate algorithm:1. Give "strong" LLM the task formulation and some labeled examples. Ask it to generate a prompt for the "weak" LLM.2. Run "weak" LLM on the training set with generated prompt from 1, use replies as features for a smaller ML model (logreg, decision tree etc).3. Pick examples from the training set that your small model is most wrong about and ask "strong" LLM to generate one more prompt (like in 1), except this time you are using the misclassified examples instead of random.4. Run "weak" LLM on generated prompt from 3, add results as one more feature for your model.5. Repeat 2 - 4 until your token budget for this task is ex
原话 · @Anonymous
In my work on LLM as a judge, I prefer to use LLM decisions as features in a downstream classic ML model for the final decision.
原话 · @Anonymous
Why does he first ask to label "ironic" or "not", and then answer the feature questions? Wouldn't it be better to reverse the order?
原话 · @Anonymous
Really needs a comparison to the "megaprompt" itself (i.e. "here is a tweet, rate it as ironic or not, considering the following properties; explain your reasoning then output your final answer at the end"). I bet that would get you very far towards the logistic classifier, and would generalize much better out of distribution.
免责声明：本卡片由 AI 从社区讨论中提炼，属于「痛点证据」，是需求分析与验证的起点，不等同于「已验证的市场需求」。AI 存在分析局限，关键决策前请打开上方原始讨论人工复核。