Initial commit

This commit is contained in:
Eugenius Naumenko
2026-06-07 22:20:57 +00:00
commit a7d8f72129
357 changed files with 168337 additions and 0 deletions
+248
View File
@@ -0,0 +1,248 @@
# Agentflow V2 Setup
This is the recommended Flowise Agentflow V2 setup for the Speech Learning App.
It is designed for the current Go backend adapter.
## Runtime facts
- Production app: `https://speech.v2202510301049387828.goodsrv.de`
- Production API health: `https://speech.v2202510301049387828.goodsrv.de/api/health`
- Existing Flowise instance: `https://flowise.v2202510301049387828.goodsrv.de`
- Internal Flowise URL for the API container: `http://flowise:3000`
- Verified Flowise version on the existing instance: `3.0.11`
- The Go backend calls Flowise with `POST /api/v1/prediction/{FLOWISE_CHATFLOW_ID}`.
- The request body includes `question` as a JSON string and `form` as the same structured object.
Build the first V2 flow with Start Node input type `Chat Input`; it is the simplest path because
all current tasks arrive as one JSON context. V2 `Form Input` is also possible now, but only if the
form fields match the backend payload fields for the task you want to handle.
The backend parses Flowise responses from `text`, `json`, or a raw JSON response. A plain JSON text
reply is still the easiest response format to debug in Flowise and in the app.
## Current backend tasks
The current backend can call the agent for these tasks:
1. `practice_feedback`
- Sent after Azure/mock pronunciation assessment.
- Context includes `target_text`, `recognized_text`, `scores`, `weak_words`, and rules.
- The agent must never create or modify pronunciation scores.
2. `start_conversation`
- Starts a short A1 roleplay.
- Context includes `target_language` and `cefr_level`.
3. `continue_conversation`
- Continues a short A1 roleplay.
- Context includes `target_language`, `cefr_level`, and `user_text`.
The backend does not currently call Flowise to choose the next DB exercise. The prompt
`prompts/exercise_generation.md` is useful for a later backend extension, but it is not used now.
## Recommended V2 graph
Create a new `Agentflow V2` named `Speech A1 English Teacher`.
Recommended minimal graph:
```txt
Start Node (Chat Input)
-> LLM Node or Agent Node
-> Direct Reply Node
```
Use an `LLM Node` if you want the most stable JSON. Use an `Agent Node` if you want to add tools
later. For the current backend, the LLM Node is enough because the Go API already owns auth, DB,
Azure Speech, attempts, progress, and conversation persistence.
Start Node:
- Input Type: `Chat Input`
- Ephemeral Memory: `On`
- Flow State keys, if you use state:
- `teacher_response`: empty string
- `validation_error`: empty string
LLM/Agent Node:
- Input Message: `{{ $flow.input }}`
- Memory: off for now. The backend does not pass a Flowise `sessionId` yet.
- Output: JSON text only.
Direct Reply Node:
- Message: the LLM/Agent output.
## System prompt
Paste this into the LLM/Agent system message:
```txt
You are an A1 English teacher for German-speaking adult learners in a pronunciation and speaking app.
You receive one JSON object as the user message. It contains a field named "task".
Global rules:
- Return valid JSON only. No Markdown. No code fences. No commentary outside JSON.
- Use exactly this JSON shape:
{
"exercise_type": "repeat_sentence",
"target_text": "English text the learner should say next",
"teacher_feedback": "Short German feedback",
"tts_text": "Text for speech synthesis",
"ssml": "",
"next_recommendation": "One short next step in German"
}
- Allowed exercise_type values: "repeat_sentence", "conversation".
- Keep the learner at CEFR A1 unless the input explicitly says otherwise.
- Explain in German. Target sentences and roleplay turns are in English.
- Use short, everyday A1 English. Prefer 3 to 9 words for target_text when possible.
- Give at most two corrections per turn.
- Do not invent pronunciation scores. Scores come only from Azure Speech or the backend mock.
- Do not change score values. Use scores only to decide feedback.
- Prefer repetition, minimal pairs, shadowing, short roleplay, and one clear next step.
- Keep ssml as an empty string unless the backend later asks for SSML.
Task rules:
If task is "practice_feedback":
- Use target_text, recognized_text, scores, and weak_words.
- Mention the weakest word or sound if available.
- If overall_score is high, praise briefly and recommend a more natural rhythm.
- If weak_words are present, ask the learner to repeat the original target_text slowly.
- exercise_type must be "repeat_sentence".
- target_text and tts_text should usually equal the original target_text.
If task is "start_conversation":
- Start one simple A1 roleplay.
- Use one short teacher sentence as target_text and tts_text.
- Give a German next_recommendation that tells the learner how to answer.
- exercise_type must be "conversation".
If task is "continue_conversation":
- Read user_text and continue with one short teacher sentence.
- Correct only the most important issue, if any.
- Keep the roleplay moving with a simple A1 question or prompt.
- exercise_type must be "conversation".
If the task is missing or unknown:
- Return a safe repeat_sentence response for A1 practice.
```
## JSON contract
Use this contract mentally or as a structured-output schema if the UI supports it.
```json
{
"type": "object",
"additionalProperties": false,
"properties": {
"exercise_type": {
"type": "string",
"enum": ["repeat_sentence", "conversation"]
},
"target_text": {
"type": "string"
},
"teacher_feedback": {
"type": "string"
},
"tts_text": {
"type": "string"
},
"ssml": {
"type": "string"
},
"next_recommendation": {
"type": "string"
}
},
"required": [
"exercise_type",
"target_text",
"teacher_feedback",
"tts_text",
"ssml",
"next_recommendation"
]
}
```
## Test payloads
Paste these as chat input in Flowise.
Practice feedback:
```json
{"task":"practice_feedback","target_text":"Can I have a coffee, please?","recognized_text":"Can I have coffee please","scores":{"recognized_text":"Can I have coffee please","overall_score":72,"accuracy_score":70,"fluency_score":78,"completeness_score":82,"prosody_score":69,"word_scores":[{"word":"Can","accuracy":91,"highlight":"good"},{"word":"I","accuracy":88,"highlight":"good"},{"word":"have","accuracy":74,"highlight":"ok"},{"word":"a","accuracy":55,"highlight":"weak"},{"word":"coffee","accuracy":79,"highlight":"ok"},{"word":"please","accuracy":68,"highlight":"weak"}],"provider":"mock"},"weak_words":[{"word":"a","accuracy":55,"highlight":"weak"},{"word":"please","accuracy":68,"highlight":"weak"}],"rules":"Answer in German. A1 learner. Max two corrections. Return valid JSON only."}
```
Start conversation:
```json
{"task":"start_conversation","target_language":"en-US","cefr_level":"A1","rules":"Create a simple A1 roleplay. Return valid JSON only."}
```
Continue conversation:
```json
{"task":"continue_conversation","target_language":"en-US","cefr_level":"A1","user_text":"My name Anna","rules":"Continue a simple A1 conversation. Return valid JSON only."}
```
Expected output example:
```json
{
"exercise_type": "conversation",
"target_text": "Nice to meet you, Anna.",
"teacher_feedback": "Fast richtig: Sage \"My name is Anna.\"",
"tts_text": "Nice to meet you, Anna.",
"ssml": "",
"next_recommendation": "Antworte kurz: I am from ..."
}
```
## Activation
After saving the Agentflow V2, copy its ID into `.env`:
```env
FLOWISE_CHATFLOW_ID=<flow-id>
```
If the flow has API-key protection enabled, also set:
```env
FLOWISE_API_KEY=<flow-api-key>
```
Then recreate the API container on the VPS. Use `--build` when deploying backend code changes:
```bash
docker compose up --build -d speech-api
```
## Validation
1. In Flowise, run each test payload and confirm the reply is plain JSON text.
2. In the app, submit a pronunciation attempt and verify that the `teacher` response is not the local fallback.
3. Use the Debug page or `GET /api/debug/flowise` after the API is recreated with `DEBUG_ENABLED=true`.
If `/api/debug/*` returns `404`, the running API was started without debug routes. Recreate the
`speech-api` container with the current `.env`.
## Later upgrades
These backend changes would unlock a more capable V2 agent:
- Send `overrideConfig.sessionId` and/or recent turns for conversation continuity.
- Add typed V2 `Form Input` variants if individual task-specific flows are split out.
- Add a backend task for dynamic exercise generation using `prompts/exercise_generation.md`.
- Add a safe internal admin/debug tool only if the backend passes a scoped service credential.
References:
- Flowise Agentflow V2: https://docs.flowiseai.com/using-flowise/agentflowv2
- Flowise Prediction API: https://docs.flowiseai.com/api-reference/prediction
- Flowise Tools and MCP: https://docs.flowiseai.com/tutorials/tools-and-mcp