Overriding User Message

You can enhance or override user messages by adding additional parts, images, or structured data before sending them to the agent. This is useful for browser automation, adding context, or attaching rich content.

Overview

User message overriding allows you to:

• Add images or screenshots to user messages
• Attach structured data and observations
• Include additional context without modifying the original text
• Use session store user parts for automatic attachment

JavaScript / TypeScript

Using Session Store User Parts

The recommended approach is to use session store user parts, which automatically attach to user messages:

import { Distri } from '@distri/core';

const client = new Distri({
  baseUrl: 'http://localhost:8787/api/v1',
});

const threadId = 'thread-123';

// Set user parts that will automatically attach to the next user message
await client.setUserPartText(
  threadId,
  'observation',
  'User clicked the submit button at 14:32:15'
);

await client.setUserPartImage(
  threadId,
  'screenshot',
  {
    bytes: base64ImageString,
    mimeType: 'image/png',
    name: 'screenshot.png',
  }
);

// Send a message - user parts are automatically included
await client.sendMessage({
  agentId: 'my_agent',
  threadId,
  message: {
    role: 'user',
    parts: [{ part_type: 'text', data: 'What do you see in this screenshot?' }],
  },
});

Manually Adding Parts to Messages

You can also manually construct messages with multiple parts:

import { Distri } from '@distri/core';

const client = new Distri({
  baseUrl: 'http://localhost:8787/api/v1',
});

// Send a message with text and image
await client.sendMessage({
  agentId: 'my_agent',
  threadId: 'thread-123',
  message: {
    role: 'user',
    parts: [
      { 
        part_type: 'text', 
        data: 'Analyze this screenshot and tell me what you see' 
      },
      {
        part_type: 'image_url',
        image_url: {
          url: 'data:image/png;base64,iVBORw0KGgoAAAANS...',
        },
      },
      {
        part_type: 'data',
        data: {
          observation: 'User is on the login page',
          timestamp: '2024-01-15T10:30:00Z',
        },
      },
    ],
  },
});

Using React Hooks

import { useChat } from '@distri/react';
import { useAgent } from '@distri/react';

function MyComponent() {
  const { agent } = useAgent({ agentIdOrDef: 'my_agent' });
  const { sendMessage } = useChat({ agent, threadId: 'thread-123' });

  const handleSendWithScreenshot = async (text: string, screenshot: string) => {
    await sendMessage([
      { part_type: 'text', data: text },
      {
        part_type: 'image_url',
        image_url: { url: screenshot },
      },
    ]);
  };

  return (
    <button onClick={() => handleSendWithScreenshot('Analyze this', screenshotData)}>
      Send with Screenshot
    </button>
  );
}

Rust

Using Session Store User Parts

use distri::Distri;
use distri_types::Part;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Distri::from_env();
    let session_id = "thread-123";

    // Set user parts that will automatically attach to the next user message
    client.set_user_part_text(
        session_id,
        "observation",
        "User clicked the submit button at 14:32:15",
    ).await?;

    client.set_user_part_image(
        session_id,
        "screenshot",
        distri_types::FileType::Bytes {
            bytes: base64_image_string,
            mime_type: "image/png".to_string(),
            name: Some("screenshot.png".to_string()),
        },
    ).await?;

    // Send a message - user parts are automatically included
    let messages = vec![
        distri_types::Message::user(
            "What do you see in this screenshot?".into(),
            None,
        ),
    ];

    let replies = client.invoke("my_agent", &messages).await?;
    for reply in replies {
        if let Some(text) = reply.as_text() {
            println!("{text}");
        }
    }

    Ok(())
}

Manually Adding Parts to Messages

use distri::Distri;
use distri_types::{Message, Part};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Distri::from_env();

    // Create a message with multiple parts
    let message = Message {
        role: distri_types::Role::User,
        parts: vec![
            Part::Text("Analyze this screenshot and tell me what you see".to_string()),
            Part::ImageUrl {
                url: "data:image/png;base64,iVBORw0KGgoAAAANS...".to_string(),
            },
            Part::Data(serde_json::json!({
                "observation": "User is on the login page",
                "timestamp": "2024-01-15T10:30:00Z",
            })),
        ],
    };

    let messages = vec![message];
    let replies = client.invoke("my_agent", &messages).await?;

    for reply in replies {
        if let Some(text) = reply.as_text() {
            println!("{text}");
        }
    }

    Ok(())
}

API Endpoints

Send Message with Multiple Parts

POST /api/v1/agents/{agent_id}
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": "1",
  "method": "message/send",
  "params": {
    "thread_id": "thread-123",
    "message": {
      "kind": "message",
      "message_id": "m1",
      "role": "user",
      "parts": [
        {
          "kind": "text",
          "text": "Analyze this screenshot"
        },
        {
          "kind": "image_url",
          "image_url": {
            "url": "data:image/png;base64,..."
          }
        },
        {
          "kind": "data",
          "data": {
            "observation": "User is on the login page",
            "timestamp": "2024-01-15T10:30:00Z"
          }
        }
      ]
    }
  }
}

Set User Part via Session Store

POST /api/v1/threads/{thread_id}/session
Content-Type: application/json

{
  "key": "__user_part_screenshot",
  "value": {
    "part_type": "image_url",
    "image_url": {
      "url": "data:image/png;base64,..."
    }
  }
}

Use Cases

• Browser Automation: Attach screenshots and DOM observations to user messages
• Rich Context: Add structured data, timestamps, or metadata to messages
• Multi-modal Input: Combine text, images, and data in a single message
• State Tracking: Include application state or user actions as part of the message
• Debugging: Attach diagnostic information to help agents understand context

Best Practices

1. Use Session Store for Repeated Context: If the same context applies to multiple messages, use session store user parts
2. Clear Part Types: Use appropriate part types (text, image_url, data) for clarity
3. Compress Images: Use base64 encoding efficiently and consider compression for large images
4. Structured Data: Use the data part type for structured information that agents can parse
5. Clean Up: Remove user parts from session store when they're no longer needed