www.promptfoo.dev Open in urlscan Pro
2606:4700:3035::ac43:da14  Public Scan

Form analysis
0 forms found in the DOM

Text Content

Skip to main content

promptfoo
DocsGitHubDiscord

SearchK



ITERATE ON LLMS FASTER

Measure LLM quality and catch regressions

Get Started


USED BY DEVELOPERS AT




... TO RAPIDLY IMPROVE PROMPTS AND EVALUATE MODELS

Simple, declarative config


# Compare prompts...
prompts:
  - "Summarize this in {{language}}: {{document}}"
  - "Summarize this in {{language}}, concisely and professionally: {{document}}"

# And models...
providers:
  - openai:gpt-4-0125-preview
  - anthropic:claude-3-opus
  - mistral:mistral-large-latest

# ... using these tests
tests:
  - vars:
      language: French
      document: "To be or not to be, that is the question..."
    assert:
      - type: contains
        value: "Être ou ne pas être"
      - type: cost
        threshold: 0.01
      - type: latency
        threshold: 1000
      - type: llm-rubric
        value: does not apologize
  - # ...
              




Detailed, actionable results


HOW IT WORKS


CREATE A TEST DATASET

Use a representative sample of user inputs to reduce subjectivity when tuning
prompts.


SET UP EVALUATION METRICS

Use built-in metrics, LLM-graded evals, or define your own custom metrics.


SELECT THE BEST PROMPT & MODEL

Compare prompts and model outputs side-by-side, or integrate the library into
your existing test/CI workflow.

WEB VIEWER

COMMAND LINE


PROMPTFOO IS USED BY LLM APPS SERVING OVER 10 MILLION USERS

Get Started
Docs
 * Intro
 * Command line
 * Node package
 * Privacy policy

Guides
 * Running benchmarks
 * Evaluating factuality
 * Evaluating RAGs
 * Minimizing hallucinations

Community
 * GitHub
 * Discord

© 2024 promptfoo