Pick LangSmith if your agents are built with LangChain. Pick Langfuse if you want open source tracing, flexible analytics, and lower lock-in. Both help you answer the same painful question: why did my AI agent just do that weird thing?
TLDR: LangSmith is polished and great for LangChain teams. Langfuse is more open, cheaper to control, and strong for product analytics. For example, a support team running 10,000 agent chats per week might find that 18% of failures come from one slow CRM tool call. A profiler can turn that mess into a clear fix in one afternoon.
What is an agent profiler?
An agent profiler is like a fitness tracker for your AI agent. It watches every step. It counts tokens. It times tool calls. It records prompts, answers, errors, retries, and costs.
Without it, debugging agents feels like chasing a raccoon through a dark kitchen. You hear noise. You see damage. You have no idea what happened.
With it, you get a trace. A trace shows the full trip from user message to final answer. You can see when the agent called a search tool. You can see if it picked the wrong tool. You can see if it spent 8 seconds thinking about nothing useful. Fun? Sort of. Useful? Very.
Why agent profiling matters
Simple chatbots are easy to inspect. Agents are not. Agents plan. They call tools. They loop. They retry. They sometimes panic in a very expensive way.
A good profiler helps you track:
- Latency: How long each step takes.
- Cost: How many tokens and dollars each run burns.
- Tool behavior: Which tools are called, and when.
- Errors: Failed API calls, bad JSON, timeouts, and odd outputs.
- Quality: Whether the final answer was helpful.
- Versions: Which prompt, model, or agent code caused a change.
Honestly, it feels like every agent works great in a demo. Then real users arrive. Suddenly the “smart assistant” spends $2.40 to answer “Where is my order?” That is when profiling saves your budget.
LangSmith: best for LangChain teams
LangSmith is built by the LangChain team. That matters. If your agent already uses LangChain or LangGraph, setup is smooth. Traces show up with rich detail. Runs are grouped well. Debugging feels clean.
LangSmith is strong at:
- Tracing: See chains, tools, model calls, and nested steps.
- Evaluation: Run tests against datasets.
- Prompt testing: Compare model outputs across versions.
- LangGraph visibility: See agent paths and state changes.
- Team review: Share runs and inspect failures together.
Think of LangSmith as a neat control room for LangChain apps. It is not the cheapest room in the building. But the buttons are where you expect them.
The annoying bit? If your stack is not LangChain-heavy, some parts may feel less natural. You can still use it. But the magic is strongest when you are already inside that ecosystem.
Langfuse: open source and analytics friendly
Langfuse is a popular open source option for LLM observability. It works well across many frameworks. You can self-host it. You can use the cloud version. You can track prompts, sessions, users, traces, scores, and costs.
Langfuse is strong at:
- Open source control: Host it yourself if needed.
- Product analytics: Track sessions, users, and feedback.
- Cost tracking: Watch token spend by model, user, or feature.
- Prompt management: Store and compare prompt versions.
- Evaluations: Add scores from users, rules, or LLM judges.
Langfuse feels more like a flexible dashboard for LLM apps. It is not only for agents. That is good if you run chat, search, summaries, and agents in one product.
The catch is that self-hosting means you own the chores. Updates. Storage. Access control. Tiny config issues that eat 35 minutes and somehow feel personal.
LangSmith vs Langfuse, in plain English
| Feature | LangSmith | Langfuse |
|---|---|---|
| Best fit | LangChain and LangGraph teams | Teams wanting open source observability |
| Setup | Very easy with LangChain | Easy, but more flexible setup paths |
| Tracing | Excellent for chains and agents | Strong and framework friendly |
| Analytics | Good for debugging and evals | Strong for user and product metrics |
| Hosting | Cloud focused | Cloud or self-hosted |
| Cost control | Good | Very good, especially with custom views |
If your agent is a LangGraph beast with branches, state, and retries, LangSmith is usually the clean pick. If your company wants control, custom analytics, and self-hosting, Langfuse is hard to ignore.
Other tools worth knowing
LangSmith and Langfuse are not alone. Several tools can profile agent performance, or at least parts of it.
- Arize Phoenix: Great for open source tracing, embeddings, evals, and experiments. Nice for teams that care about model quality.
- Weights & Biases Weave: Strong for experiment tracking and LLM app inspection. Good if your ML team already uses W&B.
- Helicone: Handy for API logging, cost tracking, caching, and user analytics. Works well as a gateway for LLM calls.
- PromptLayer: Useful for prompt logging, versioning, and review. More prompt focused than full agent focused.
- OpenTelemetry: Great standard for traces and metrics. Best when your engineering team already has observability tools.
- Datadog or Honeycomb: Strong for system monitoring. Pair them with LLM-specific tracing for deeper agent insight.
No single tool wins every fight. That would be too easy. And software rarely lets us have nice things.
A quick user case
Imagine a retail company with an AI returns agent. It handles 5,000 chats each week. The team sees user complaints rise from 6% to 13%. Nobody knows why.
They add profiling. The traces show three ugly facts:
- The agent calls the order API twice in 41% of chats.
- One tool timeout adds 4.8 seconds on average.
- A new prompt version causes refund policy mistakes in 9% of tested runs.
The team fixes the tool retry rule. They roll back the prompt. They cache order lookups. One week later, average response time drops from 11.2 seconds to 5.9 seconds. Support handoffs fall by 22%.
That is the point of an agent profiler. It turns “the bot is bad” into “tool B is slow and prompt v14 is confused.” Much better.
What to measure first
Do not track everything on day one. That path leads to dashboards nobody opens.
Start with these five metrics:
- Success rate: Did the agent solve the task?
- Average latency: How long did the full run take?
- Cost per run: How much did each answer cost?
- Tool failure rate: Which tools break most often?
- Human handoff rate: How often did the agent need help?
Then add finer detail. Track step count. Track retry count. Track model choice. Track user feedback. Keep it useful. Keep it boring. Boring metrics pay bills.
How to choose
Use this simple rule:
- Choose LangSmith if you use LangChain, LangGraph, or want smooth agent tracing fast.
- Choose Langfuse if you want open source control, strong analytics, and flexible hosting.
- Choose Phoenix if evals and model quality are your main worry.
- Choose Helicone if you need clean API logging and cost controls.
- Choose OpenTelemetry plus Datadog if your team already has mature observability systems.
Final take
Agent profiling is not a fancy extra. It is the seatbelt. Agents are too costly, too weird, and too confident to run blind.
LangSmith is the sharper choice for LangChain-native agent work. Langfuse is the stronger pick for open source control and product analytics. The right tool is the one that shows failures clearly, helps your team fix them fast, and keeps your agent from turning one simple task into a tiny, expensive opera.