Prompt Iteration & Debugging

import anthropic

client = anthropic.Anthropic()

# Test cases
test_cases = [
    ("I love this!", "positive"),
    ("Terrible product.", "negative"),
    ("It's okay.", "neutral"),
]

def test_prompt(prompt, test_data):
    """Run prompt on test cases and check accuracy."""
    correct = 0
    for text, expected in test_data:
        response = client.messages.create(
            model="claude-sonnet-4-5-20250929",
            max_tokens=100,
            messages=[{
                "role": "user",
                "content": f"{prompt}\n\nText: {text}"
            }]
        )
        result = response.content[0].text.strip().lower()
        if expected in result:
            correct += 1
    accuracy = (correct / len(test_data)) * 100
    return accuracy

# Iteration cycle
prompt_v1 = "Classify sentiment: positive, negative, or neutral."
acc_v1 = test_prompt(prompt_v1, test_cases)
print(f"V1: {acc_v1}%")

# Analyze and improve
prompt_v2 = """Classify sentiment.
Examples: "love" → positive, "bad" → negative.
Return ONE word: positive, negative, or neutral."""
acc_v2 = test_prompt(prompt_v2, test_cases)
print(f"V2: {acc_v2}%")

# Debug: Is it the format spec or the system prompt?

# Test 1: Just format spec (no system prompt)
response_1 = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    messages=[{
        "role": "user",
        "content": "Return valid JSON: {\"name\": \"...\"}"
    }]
)
print("Test 1 (no system):", response_1.content[0].text)

# Test 2: With system prompt that says to be helpful
response_2 = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    system="Be helpful",
    messages=[{
        "role": "user",
        "content": "Return valid JSON: {\"name\": \"...\"}"
    }]
)
print("Test 2 (vague system):", response_2.content[0].text)

# Test 3: Strong system prompt about JSON only
response_3 = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    system="Return ONLY valid JSON. No markdown. No explanation.",
    messages=[{
        "role": "user",
        "content": "Return valid JSON: {\"name\": \"...\"}"
    }]
)
print("Test 3 (strong system):", response_3.content[0].text)

# Conclusion: Hypothesis was right! System prompt was too weak.

import json
from anthropic import Anthropic

client = Anthropic()

# Test dataset
test_data = [
    {"text": "I love this!", "expected": "positive"},
    {"text": "Awful experience.", "expected": "negative"},
    {"text": "It's okay.", "expected": "neutral"},
]

def test_prompt_version(prompt, dataset):
    """Test a prompt and return accuracy."""
    results = []
    for item in dataset:
        response = client.messages.create(
            model="claude-sonnet-4-5-20250929",
            max_tokens=100,
            messages=[{
                "role": "user",
                "content": f"{prompt}\n\nText: {item['text']}"
            }]
        )
        result = response.content[0].text.strip().lower()
        correct = item["expected"] in result
        results.append({
            "input": item["text"],
            "expected": item["expected"],
            "output": result,
            "correct": correct
        })
    accuracy = (sum(1 for r in results if r["correct"])) / len(results) * 100
    return accuracy, results

# Prompt A (basic)
prompt_a = "Classify sentiment: positive, negative, or neutral."

# Prompt B (improved with examples)
prompt_b = """Classify sentiment: positive, negative, neutral.
Examples:
- "love" → positive
- "hate" → negative
- "ok" → neutral

Return ONE word only."""

# Run A/B test
acc_a, results_a = test_prompt_version(prompt_a, test_data)
acc_b, results_b = test_prompt_version(prompt_b, test_data)

print(f"Prompt A accuracy: {acc_a}%")
print(f"Prompt B accuracy: {acc_b}%")
print(f"Winner: {'B' if acc_b > acc_a else 'A'}")

## Sentiment Classifier Prompt Changelog

### v1.0 (2025-01-15)
- Initial: "Classify sentiment: positive, negative, neutral"
- Accuracy: 72%
- Issue: Too vague, model guesses

### v1.1 (2025-01-16)
- Added few-shot examples
- Added "Return ONE word only" constraint
- Accuracy: 85%
- Improvement: +13%

### v1.2 (2025-01-17)
- Moved examples to system prompt
- Added "If unsure, say 'unclear'"
- Accuracy: 87%
- Improvement: +2%

### v2.0 (2025-01-18)
- Full XML structure with tags
- Explicit JSON output format
- Accuracy: 89%
- Improvement: +2%, better production-grade

Review this code.

You are a code reviewer.
Review this Python code for bugs and improvements.
List issues and suggestions.

Code:
[code here]

You are a senior Python engineer.
Review code focusing on:
1. Security issues (highest priority)
2. Performance problems
3. Code quality improvements

Output format:
## Security Issues
[list with severity]

## Performance
[specific suggestions]

## Code Quality
[improvements with examples]

Code:
[code here]

<system_role>
You are a senior Python engineer at a Fortune 500
company with 15 years of experience. Your code
reviews are known for catching subtle bugs and
suggesting high-impact improvements.
</system_role>

<instructions>
Review this Python code. Flag issues in priority order.
</instructions>

<priorities>
1. Security vulnerabilities (SQL injection, etc.)
2. Race conditions or concurrency issues
3. Memory leaks or performance problems
4. Code quality and style issues
</priorities>

<output_format>
## 🔴 Critical Issues
- Issue 1 (line X): description
- Issue 2 (line Y): description

## 🟡 Warnings
- Warning 1: description

## 💡 Improvements
- Suggestion 1: description

## ✅ Positive Notes
- What's good about the code
</output_format>

<rules>
- Cite specific line numbers
- Provide code examples for fixes
- Be constructive, not dismissive
- Don't comment on style preferences
- If you're unsure, say "needs clarification"
</rules>

Code:
[code here]

<system_role>
You are a senior Python engineer at a Fortune 500
company with 15 years of experience. Your code
reviews are known for catching subtle bugs and
suggesting high-impact improvements.
</system_role>

<examples>
Example 1: SQL injection vulnerability
```python
result = db.execute(f"SELECT * FROM users WHERE id = {user_id}")
```
Review: "🔴 Critical security issue on line 2. This is vulnerable to
SQL injection. Fix: Use parameterized queries."

Example 2: Good code
```python
def calculate_total(items: list[float]) -> float:
    """Calculate sum of items with validation."""
    return sum(item for item in items if isinstance(item, (int, float)))
```
Review: "✅ Good: Type hints, docstring, and defensive code."
</examples>

<instructions>
Review this Python code. Flag issues in priority order.
</instructions>

... [rest of v4 structure] ...

Code:
[code here]