Files
Speech/flowise/agentflow-v2-setup.md
T
2026-06-07 22:20:57 +00:00

8.7 KiB

Agentflow V2 Setup

This is the recommended Flowise Agentflow V2 setup for the Speech Learning App. It is designed for the current Go backend adapter.

Runtime facts

  • Production app: https://speech.v2202510301049387828.goodsrv.de
  • Production API health: https://speech.v2202510301049387828.goodsrv.de/api/health
  • Existing Flowise instance: https://flowise.v2202510301049387828.goodsrv.de
  • Internal Flowise URL for the API container: http://flowise:3000
  • Verified Flowise version on the existing instance: 3.0.11
  • The Go backend calls Flowise with POST /api/v1/prediction/{FLOWISE_CHATFLOW_ID}.
  • The request body includes question as a JSON string and form as the same structured object.

Build the first V2 flow with Start Node input type Chat Input; it is the simplest path because all current tasks arrive as one JSON context. V2 Form Input is also possible now, but only if the form fields match the backend payload fields for the task you want to handle.

The backend parses Flowise responses from text, json, or a raw JSON response. A plain JSON text reply is still the easiest response format to debug in Flowise and in the app.

Current backend tasks

The current backend can call the agent for these tasks:

  1. practice_feedback
    • Sent after Azure/mock pronunciation assessment.
    • Context includes target_text, recognized_text, scores, weak_words, and rules.
    • The agent must never create or modify pronunciation scores.
  2. start_conversation
    • Starts a short A1 roleplay.
    • Context includes target_language and cefr_level.
  3. continue_conversation
    • Continues a short A1 roleplay.
    • Context includes target_language, cefr_level, and user_text.

The backend does not currently call Flowise to choose the next DB exercise. The prompt prompts/exercise_generation.md is useful for a later backend extension, but it is not used now.

Create a new Agentflow V2 named Speech A1 English Teacher.

Recommended minimal graph:

Start Node (Chat Input)
  -> LLM Node or Agent Node
  -> Direct Reply Node

Use an LLM Node if you want the most stable JSON. Use an Agent Node if you want to add tools later. For the current backend, the LLM Node is enough because the Go API already owns auth, DB, Azure Speech, attempts, progress, and conversation persistence.

Start Node:

  • Input Type: Chat Input
  • Ephemeral Memory: On
  • Flow State keys, if you use state:
    • teacher_response: empty string
    • validation_error: empty string

LLM/Agent Node:

  • Input Message: {{ $flow.input }}
  • Memory: off for now. The backend does not pass a Flowise sessionId yet.
  • Output: JSON text only.

Direct Reply Node:

  • Message: the LLM/Agent output.

System prompt

Paste this into the LLM/Agent system message:

You are an A1 English teacher for German-speaking adult learners in a pronunciation and speaking app.

You receive one JSON object as the user message. It contains a field named "task".

Global rules:
- Return valid JSON only. No Markdown. No code fences. No commentary outside JSON.
- Use exactly this JSON shape:
  {
    "exercise_type": "repeat_sentence",
    "target_text": "English text the learner should say next",
    "teacher_feedback": "Short German feedback",
    "tts_text": "Text for speech synthesis",
    "ssml": "",
    "next_recommendation": "One short next step in German"
  }
- Allowed exercise_type values: "repeat_sentence", "conversation".
- Keep the learner at CEFR A1 unless the input explicitly says otherwise.
- Explain in German. Target sentences and roleplay turns are in English.
- Use short, everyday A1 English. Prefer 3 to 9 words for target_text when possible.
- Give at most two corrections per turn.
- Do not invent pronunciation scores. Scores come only from Azure Speech or the backend mock.
- Do not change score values. Use scores only to decide feedback.
- Prefer repetition, minimal pairs, shadowing, short roleplay, and one clear next step.
- Keep ssml as an empty string unless the backend later asks for SSML.

Task rules:

If task is "practice_feedback":
- Use target_text, recognized_text, scores, and weak_words.
- Mention the weakest word or sound if available.
- If overall_score is high, praise briefly and recommend a more natural rhythm.
- If weak_words are present, ask the learner to repeat the original target_text slowly.
- exercise_type must be "repeat_sentence".
- target_text and tts_text should usually equal the original target_text.

If task is "start_conversation":
- Start one simple A1 roleplay.
- Use one short teacher sentence as target_text and tts_text.
- Give a German next_recommendation that tells the learner how to answer.
- exercise_type must be "conversation".

If task is "continue_conversation":
- Read user_text and continue with one short teacher sentence.
- Correct only the most important issue, if any.
- Keep the roleplay moving with a simple A1 question or prompt.
- exercise_type must be "conversation".

If the task is missing or unknown:
- Return a safe repeat_sentence response for A1 practice.

JSON contract

Use this contract mentally or as a structured-output schema if the UI supports it.

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "exercise_type": {
      "type": "string",
      "enum": ["repeat_sentence", "conversation"]
    },
    "target_text": {
      "type": "string"
    },
    "teacher_feedback": {
      "type": "string"
    },
    "tts_text": {
      "type": "string"
    },
    "ssml": {
      "type": "string"
    },
    "next_recommendation": {
      "type": "string"
    }
  },
  "required": [
    "exercise_type",
    "target_text",
    "teacher_feedback",
    "tts_text",
    "ssml",
    "next_recommendation"
  ]
}

Test payloads

Paste these as chat input in Flowise.

Practice feedback:

{"task":"practice_feedback","target_text":"Can I have a coffee, please?","recognized_text":"Can I have coffee please","scores":{"recognized_text":"Can I have coffee please","overall_score":72,"accuracy_score":70,"fluency_score":78,"completeness_score":82,"prosody_score":69,"word_scores":[{"word":"Can","accuracy":91,"highlight":"good"},{"word":"I","accuracy":88,"highlight":"good"},{"word":"have","accuracy":74,"highlight":"ok"},{"word":"a","accuracy":55,"highlight":"weak"},{"word":"coffee","accuracy":79,"highlight":"ok"},{"word":"please","accuracy":68,"highlight":"weak"}],"provider":"mock"},"weak_words":[{"word":"a","accuracy":55,"highlight":"weak"},{"word":"please","accuracy":68,"highlight":"weak"}],"rules":"Answer in German. A1 learner. Max two corrections. Return valid JSON only."}

Start conversation:

{"task":"start_conversation","target_language":"en-US","cefr_level":"A1","rules":"Create a simple A1 roleplay. Return valid JSON only."}

Continue conversation:

{"task":"continue_conversation","target_language":"en-US","cefr_level":"A1","user_text":"My name Anna","rules":"Continue a simple A1 conversation. Return valid JSON only."}

Expected output example:

{
  "exercise_type": "conversation",
  "target_text": "Nice to meet you, Anna.",
  "teacher_feedback": "Fast richtig: Sage \"My name is Anna.\"",
  "tts_text": "Nice to meet you, Anna.",
  "ssml": "",
  "next_recommendation": "Antworte kurz: I am from ..."
}

Activation

After saving the Agentflow V2, copy its ID into .env:

FLOWISE_CHATFLOW_ID=<flow-id>

If the flow has API-key protection enabled, also set:

FLOWISE_API_KEY=<flow-api-key>

Then recreate the API container on the VPS. Use --build when deploying backend code changes:

docker compose up --build -d speech-api

Validation

  1. In Flowise, run each test payload and confirm the reply is plain JSON text.
  2. In the app, submit a pronunciation attempt and verify that the teacher response is not the local fallback.
  3. Use the Debug page or GET /api/debug/flowise after the API is recreated with DEBUG_ENABLED=true.

If /api/debug/* returns 404, the running API was started without debug routes. Recreate the speech-api container with the current .env.

Later upgrades

These backend changes would unlock a more capable V2 agent:

  • Send overrideConfig.sessionId and/or recent turns for conversation continuity.
  • Add typed V2 Form Input variants if individual task-specific flows are split out.
  • Add a backend task for dynamic exercise generation using prompts/exercise_generation.md.
  • Add a safe internal admin/debug tool only if the backend passes a scoped service credential.

References: