249 lines
8.7 KiB
Markdown
249 lines
8.7 KiB
Markdown
# Agentflow V2 Setup
|
|
|
|
This is the recommended Flowise Agentflow V2 setup for the Speech Learning App.
|
|
It is designed for the current Go backend adapter.
|
|
|
|
## Runtime facts
|
|
|
|
- Production app: `https://speech.v2202510301049387828.goodsrv.de`
|
|
- Production API health: `https://speech.v2202510301049387828.goodsrv.de/api/health`
|
|
- Existing Flowise instance: `https://flowise.v2202510301049387828.goodsrv.de`
|
|
- Internal Flowise URL for the API container: `http://flowise:3000`
|
|
- Verified Flowise version on the existing instance: `3.0.11`
|
|
- The Go backend calls Flowise with `POST /api/v1/prediction/{FLOWISE_CHATFLOW_ID}`.
|
|
- The request body includes `question` as a JSON string and `form` as the same structured object.
|
|
|
|
Build the first V2 flow with Start Node input type `Chat Input`; it is the simplest path because
|
|
all current tasks arrive as one JSON context. V2 `Form Input` is also possible now, but only if the
|
|
form fields match the backend payload fields for the task you want to handle.
|
|
|
|
The backend parses Flowise responses from `text`, `json`, or a raw JSON response. A plain JSON text
|
|
reply is still the easiest response format to debug in Flowise and in the app.
|
|
|
|
## Current backend tasks
|
|
|
|
The current backend can call the agent for these tasks:
|
|
|
|
1. `practice_feedback`
|
|
- Sent after Azure/mock pronunciation assessment.
|
|
- Context includes `target_text`, `recognized_text`, `scores`, `weak_words`, and rules.
|
|
- The agent must never create or modify pronunciation scores.
|
|
2. `start_conversation`
|
|
- Starts a short A1 roleplay.
|
|
- Context includes `target_language` and `cefr_level`.
|
|
3. `continue_conversation`
|
|
- Continues a short A1 roleplay.
|
|
- Context includes `target_language`, `cefr_level`, and `user_text`.
|
|
|
|
The backend does not currently call Flowise to choose the next DB exercise. The prompt
|
|
`prompts/exercise_generation.md` is useful for a later backend extension, but it is not used now.
|
|
|
|
## Recommended V2 graph
|
|
|
|
Create a new `Agentflow V2` named `Speech A1 English Teacher`.
|
|
|
|
Recommended minimal graph:
|
|
|
|
```txt
|
|
Start Node (Chat Input)
|
|
-> LLM Node or Agent Node
|
|
-> Direct Reply Node
|
|
```
|
|
|
|
Use an `LLM Node` if you want the most stable JSON. Use an `Agent Node` if you want to add tools
|
|
later. For the current backend, the LLM Node is enough because the Go API already owns auth, DB,
|
|
Azure Speech, attempts, progress, and conversation persistence.
|
|
|
|
Start Node:
|
|
|
|
- Input Type: `Chat Input`
|
|
- Ephemeral Memory: `On`
|
|
- Flow State keys, if you use state:
|
|
- `teacher_response`: empty string
|
|
- `validation_error`: empty string
|
|
|
|
LLM/Agent Node:
|
|
|
|
- Input Message: `{{ $flow.input }}`
|
|
- Memory: off for now. The backend does not pass a Flowise `sessionId` yet.
|
|
- Output: JSON text only.
|
|
|
|
Direct Reply Node:
|
|
|
|
- Message: the LLM/Agent output.
|
|
|
|
## System prompt
|
|
|
|
Paste this into the LLM/Agent system message:
|
|
|
|
```txt
|
|
You are an A1 English teacher for German-speaking adult learners in a pronunciation and speaking app.
|
|
|
|
You receive one JSON object as the user message. It contains a field named "task".
|
|
|
|
Global rules:
|
|
- Return valid JSON only. No Markdown. No code fences. No commentary outside JSON.
|
|
- Use exactly this JSON shape:
|
|
{
|
|
"exercise_type": "repeat_sentence",
|
|
"target_text": "English text the learner should say next",
|
|
"teacher_feedback": "Short German feedback",
|
|
"tts_text": "Text for speech synthesis",
|
|
"ssml": "",
|
|
"next_recommendation": "One short next step in German"
|
|
}
|
|
- Allowed exercise_type values: "repeat_sentence", "conversation".
|
|
- Keep the learner at CEFR A1 unless the input explicitly says otherwise.
|
|
- Explain in German. Target sentences and roleplay turns are in English.
|
|
- Use short, everyday A1 English. Prefer 3 to 9 words for target_text when possible.
|
|
- Give at most two corrections per turn.
|
|
- Do not invent pronunciation scores. Scores come only from Azure Speech or the backend mock.
|
|
- Do not change score values. Use scores only to decide feedback.
|
|
- Prefer repetition, minimal pairs, shadowing, short roleplay, and one clear next step.
|
|
- Keep ssml as an empty string unless the backend later asks for SSML.
|
|
|
|
Task rules:
|
|
|
|
If task is "practice_feedback":
|
|
- Use target_text, recognized_text, scores, and weak_words.
|
|
- Mention the weakest word or sound if available.
|
|
- If overall_score is high, praise briefly and recommend a more natural rhythm.
|
|
- If weak_words are present, ask the learner to repeat the original target_text slowly.
|
|
- exercise_type must be "repeat_sentence".
|
|
- target_text and tts_text should usually equal the original target_text.
|
|
|
|
If task is "start_conversation":
|
|
- Start one simple A1 roleplay.
|
|
- Use one short teacher sentence as target_text and tts_text.
|
|
- Give a German next_recommendation that tells the learner how to answer.
|
|
- exercise_type must be "conversation".
|
|
|
|
If task is "continue_conversation":
|
|
- Read user_text and continue with one short teacher sentence.
|
|
- Correct only the most important issue, if any.
|
|
- Keep the roleplay moving with a simple A1 question or prompt.
|
|
- exercise_type must be "conversation".
|
|
|
|
If the task is missing or unknown:
|
|
- Return a safe repeat_sentence response for A1 practice.
|
|
```
|
|
|
|
## JSON contract
|
|
|
|
Use this contract mentally or as a structured-output schema if the UI supports it.
|
|
|
|
```json
|
|
{
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"exercise_type": {
|
|
"type": "string",
|
|
"enum": ["repeat_sentence", "conversation"]
|
|
},
|
|
"target_text": {
|
|
"type": "string"
|
|
},
|
|
"teacher_feedback": {
|
|
"type": "string"
|
|
},
|
|
"tts_text": {
|
|
"type": "string"
|
|
},
|
|
"ssml": {
|
|
"type": "string"
|
|
},
|
|
"next_recommendation": {
|
|
"type": "string"
|
|
}
|
|
},
|
|
"required": [
|
|
"exercise_type",
|
|
"target_text",
|
|
"teacher_feedback",
|
|
"tts_text",
|
|
"ssml",
|
|
"next_recommendation"
|
|
]
|
|
}
|
|
```
|
|
|
|
## Test payloads
|
|
|
|
Paste these as chat input in Flowise.
|
|
|
|
Practice feedback:
|
|
|
|
```json
|
|
{"task":"practice_feedback","target_text":"Can I have a coffee, please?","recognized_text":"Can I have coffee please","scores":{"recognized_text":"Can I have coffee please","overall_score":72,"accuracy_score":70,"fluency_score":78,"completeness_score":82,"prosody_score":69,"word_scores":[{"word":"Can","accuracy":91,"highlight":"good"},{"word":"I","accuracy":88,"highlight":"good"},{"word":"have","accuracy":74,"highlight":"ok"},{"word":"a","accuracy":55,"highlight":"weak"},{"word":"coffee","accuracy":79,"highlight":"ok"},{"word":"please","accuracy":68,"highlight":"weak"}],"provider":"mock"},"weak_words":[{"word":"a","accuracy":55,"highlight":"weak"},{"word":"please","accuracy":68,"highlight":"weak"}],"rules":"Answer in German. A1 learner. Max two corrections. Return valid JSON only."}
|
|
```
|
|
|
|
Start conversation:
|
|
|
|
```json
|
|
{"task":"start_conversation","target_language":"en-US","cefr_level":"A1","rules":"Create a simple A1 roleplay. Return valid JSON only."}
|
|
```
|
|
|
|
Continue conversation:
|
|
|
|
```json
|
|
{"task":"continue_conversation","target_language":"en-US","cefr_level":"A1","user_text":"My name Anna","rules":"Continue a simple A1 conversation. Return valid JSON only."}
|
|
```
|
|
|
|
Expected output example:
|
|
|
|
```json
|
|
{
|
|
"exercise_type": "conversation",
|
|
"target_text": "Nice to meet you, Anna.",
|
|
"teacher_feedback": "Fast richtig: Sage \"My name is Anna.\"",
|
|
"tts_text": "Nice to meet you, Anna.",
|
|
"ssml": "",
|
|
"next_recommendation": "Antworte kurz: I am from ..."
|
|
}
|
|
```
|
|
|
|
## Activation
|
|
|
|
After saving the Agentflow V2, copy its ID into `.env`:
|
|
|
|
```env
|
|
FLOWISE_CHATFLOW_ID=<flow-id>
|
|
```
|
|
|
|
If the flow has API-key protection enabled, also set:
|
|
|
|
```env
|
|
FLOWISE_API_KEY=<flow-api-key>
|
|
```
|
|
|
|
Then recreate the API container on the VPS. Use `--build` when deploying backend code changes:
|
|
|
|
```bash
|
|
docker compose up --build -d speech-api
|
|
```
|
|
|
|
## Validation
|
|
|
|
1. In Flowise, run each test payload and confirm the reply is plain JSON text.
|
|
2. In the app, submit a pronunciation attempt and verify that the `teacher` response is not the local fallback.
|
|
3. Use the Debug page or `GET /api/debug/flowise` after the API is recreated with `DEBUG_ENABLED=true`.
|
|
|
|
If `/api/debug/*` returns `404`, the running API was started without debug routes. Recreate the
|
|
`speech-api` container with the current `.env`.
|
|
|
|
## Later upgrades
|
|
|
|
These backend changes would unlock a more capable V2 agent:
|
|
|
|
- Send `overrideConfig.sessionId` and/or recent turns for conversation continuity.
|
|
- Add typed V2 `Form Input` variants if individual task-specific flows are split out.
|
|
- Add a backend task for dynamic exercise generation using `prompts/exercise_generation.md`.
|
|
- Add a safe internal admin/debug tool only if the backend passes a scoped service credential.
|
|
|
|
References:
|
|
|
|
- Flowise Agentflow V2: https://docs.flowiseai.com/using-flowise/agentflowv2
|
|
- Flowise Prediction API: https://docs.flowiseai.com/api-reference/prediction
|
|
- Flowise Tools and MCP: https://docs.flowiseai.com/tutorials/tools-and-mcp
|