[BUG] Promptflow does not stream LLM response about promptflow HOT 4 CLOSED

Great! Thank you very much for your feedback @liucheng-ms!

from promptflow.

Hi @giannisha

Thank you for bringing up this issue related to streaming behavior in PromptFlow. I appreciate your attention to detail and your desire to improve the system.

Upon reviewing your scenario, it seems that there might be some confusion regarding the streaming functionality within PromptFlow. To help clarify, I'd like to direct you to our comprehensive documentation on streaming mode here: Enable streaming mode in Azure Machine Learning.

As per the current design of PromptFlow, streaming is only supported by the output of the final node in a given flow. The term "last node" specifically refers to a node whose output is not utilized by any subsequent nodes in the workflow.

In the scenario you've described, the output of the llm node is being passed on to a cache Python node. This interconnection implies that the llm node is not the last node in your flow, hence it does not satisfy the criteria for streaming its output.

To achieve the behavior you expect, you would need to adjust your flow so that the streaming node is the final node, thus allowing its output to be streamed directly to your clients. Any subsequent actions, such as caching, would then need to be managed independently outside of the PromptFlow sequence.

If you have any further questions or need additional clarification, feel free to reach out. Feedback is highly appreciated.

from promptflow.

Hi @liucheng-ms

Thank you very much for your detailed response.
Do you think this scenario, to have nodes detached from the main flow, would be good to be supported in the future?
For similar cases, or other cases where you want to fire something and forget it (cache, logs etc.).

from promptflow.

Hi @giannisha ,

I understand your request for a feature that allows nodes to be detached from the main flow, enabling actions like caching or logging to be executed independently. Your suggestion for “fire and forget” functionality could indeed be beneficial for certain use cases.

Currently, we do not have immediate plans to support this feature within PromptFlow’s scope. However, we recognize the potential value it could offer. We will consider the need for such a feature and evaluate whether it aligns with our design principles and priorities.

Thank you for your feedback, and please continue to share your thoughts and ideas with us.

from promptflow.