This package contains the Python client for interacting with the LangSmith platform.
To install:
pip install -U langsmith
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=ls_...Then trace:
import openai
from langsmith.wrappers import wrap_openai
from langsmith import traceable
# Auto-trace LLM calls in-context
client = wrap_openai(openai.Client())
@traceable # Auto-trace this function
def pipeline(user_input: str):
result = client.chat.completions.create(
messages=[{"role": "user", "content": user_input}],
model="gpt-5.4"
)
return result.choices[0].message.content
pipeline("Hello, world!")Every LLM call inside pipeline is nested under a single trace in the LangSmith UI.
LangSmith helps you and your team develop and evaluate language models and intelligent agents. It is compatible with any LLM application.
Cookbook: For tutorials on how to get more value out of LangSmith, check out the Langsmith Cookbook repo.
A typical workflow looks like:
We'll walk through these steps in more detail below.
When sandbox code needs to call AWS services, use the sandbox AWS auth proxy. The proxy keeps the real AWS credentials outside the sandbox and signs supported AWS HTTPS requests with SigV4, so code in the sandbox can use AWS SDKs normally without storing long-lived AWS keys in files, environment variables, shell history, or logs.
Store AWS credentials as LangSmith workspace secrets using names that make sense for your workspace. Then create the sandbox with an AWS auth proxy config:
from langsmith.sandbox import (
SandboxClient,
aws_auth,
proxy_config,
workspace_secret,
)
client = SandboxClient()
auth_config = proxy_config(
rules=[
aws_auth(
access_key_id=workspace_secret("SANDBOX_AWS_ACCESS_KEY_ID"),
secret_access_key=workspace_secret("SANDBOX_AWS_SECRET_ACCESS_KEY"),
)
],
)
with client.sandbox(
name="aws-sandbox",
proxy_config=auth_config,
) as sandbox:
result = sandbox.run("python your_aws_script.py")
print(result.stdout)Use opaque_secret("...") instead of workspace_secret(...) when your
application needs to pass short-lived write-only AWS credentials at sandbox
creation time. Plaintext AWS credential values are not accepted directly; wrap
them as opaque_secret(...) values.
When sandbox code needs to call Google APIs, use the sandbox GCP auth proxy. The proxy keeps the service account JSON outside the sandbox and injects OAuth bearer tokens for Google API hosts matched automatically by the sandbox proxy.
Store the service account JSON as a LangSmith workspace secret. Then create the sandbox with a GCP auth proxy config:
from langsmith.sandbox import (
SandboxClient,
gcp_auth,
proxy_config,
workspace_secret,
)
client = SandboxClient()
auth_config = proxy_config(
rules=[
gcp_auth(
service_account_json=workspace_secret(
"SANDBOX_GCP_SERVICE_ACCOUNT_JSON"
),
scopes=["https://www.googleapis.com/auth/devstorage.read_write"],
)
],
)
with client.sandbox(
name="gcp-sandbox",
proxy_config=auth_config,
) as sandbox:
result = sandbox.run("python your_gcp_script.py")
print(result.stdout)Use opaque_secret("...") for short-lived write-only service account JSON.
Plaintext service account JSON is not accepted directly.
When you create a LangSmith sandbox that needs filesystem access to external
data such as object storage buckets or public Git repositories, pass a
mount_config on sandbox creation. Mount specs contain only the mount target.
Provider credentials stay in mount_config.auth; the backend expands them into
runtime proxy auth rules. You can also pass proxy_config for non-mount proxy
behavior such as custom headers, callbacks, access control, and generic egress
rules. Explicit AWS/GCP proxy auth rules conflict with mount_config auth for
the same provider.
S3 mounts require AWS auth:
from langsmith.sandbox import (
aws_auth,
mount_config,
s3_mount,
workspace_secret,
)
mount_cfg = mount_config(
auth=[
aws_auth(
access_key_id=workspace_secret("SANDBOX_AWS_ACCESS_KEY_ID"),
secret_access_key=workspace_secret("SANDBOX_AWS_SECRET_ACCESS_KEY"),
)
],
mounts=[
s3_mount(
id="customer_data",
mount_path="/mnt/mounts/customer-data",
bucket="example-bucket",
prefix="datasets/customer-data",
region="us-east-1",
endpoint_url="https://s3.amazonaws.com",
path_style=False,
read_only=False,
)
],
)
with client.sandbox(
name="s3-mount-sandbox",
mount_config=mount_cfg,
) as sandbox:
result = sandbox.run("ls /mnt/mounts/customer-data")
print(result.stdout)GCS mounts require GCP auth:
from langsmith.sandbox import (
gcp_auth,
gcs_mount,
mount_config,
workspace_secret,
)
mount_cfg = mount_config(
auth=[
gcp_auth(
service_account_json=workspace_secret(
"SANDBOX_GCP_SERVICE_ACCOUNT_JSON"
)
)
],
mounts=[
gcs_mount(
id="customer_data",
mount_path="/mnt/mounts/customer-data",
bucket="example-bucket",
prefix="datasets/customer-data",
)
],
)
with client.sandbox(
name="gcs-mount-sandbox",
mount_config=mount_cfg,
) as sandbox:
result = sandbox.run("ls /mnt/mounts/customer-data")
print(result.stdout)Public Git mounts do not require AWS or GCP auth:
from langsmith.sandbox import git_mount, mount_config
mount_cfg = mount_config(
mounts=[
git_mount(
id="repo",
mount_path="/mnt/repo",
remote_url="https://github.com/langchain-ai/langsmith-sdk.git",
ref={"type": "branch", "name": "main"},
refresh_interval_seconds=60,
)
],
)
with client.sandbox(
name="git-mount-sandbox",
mount_config=mount_cfg,
) as sandbox:
result = sandbox.run("ls /mnt/repo")
print(result.stdout)Private Git repositories can use low-level proxy_config rules when the remote
requires proxy-managed auth. There is not yet a high-level private Git auth
helper.
Sign up for LangSmith using your GitHub, Discord accounts, or an email address and password. If you sign up with an email, make sure to verify your email address before logging in.
Then, create a unique API key on the Settings Page, which is found in the menu at the top right corner of the page.
[!NOTE] Save the API Key in a secure location. It will not be shown again.
You can log traces natively using the LangSmith SDK or within your LangChain application.
LangSmith seamlessly integrates with the Python LangChain library to record traces from your LLM applications.
Tracing can be activated by setting the following environment variables or by manually specifying the LangChainTracer.
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_ENDPOINT"] = "https://api.smith.langchain.com"
# os.environ["LANGSMITH_ENDPOINT"] = "https://eu.api.smith.langchain.com" # If signed up in the EU region
os.environ["LANGSMITH_API_KEY"] = "<YOUR-LANGSMITH-API-KEY>"
# os.environ["LANGSMITH_PROJECT"] = "My Project Name" # Optional: "default" is used if not set
# os.environ["LANGSMITH_WORKSPACE_ID"] = "<YOUR-WORKSPACE-ID>" # Required for org-scoped API keysTip: Projects are groups of traces. All runs are logged to a project. If not specified, the project is set to
default.
If the environment variables are correctly set, your application will automatically connect to the LangSmith platform.
from langchain_core.runnables import chain
@chain
def add_val(x: dict) -> dict:
return {"val": x["val"] + 1}
add_val({"val": 1})You can still use the LangSmith development platform without depending on any LangChain code.
import os
os.environ["LANGSMITH_ENDPOINT"] = "https://api.smith.langchain.com"
os.environ["LANGSMITH_API_KEY"] = "<YOUR-LANGSMITH-API-KEY>"
# os.environ["LANGSMITH_PROJECT"] = "My Project Name" # Optional: "default" is used if not setThe easiest way to log traces using the SDK is via the @traceable decorator. Below is an example.
from datetime import datetime
from typing import List, Optional, Tuple
import openai
from langsmith import traceable
from langsmith.wrappers import wrap_openai
client = wrap_openai(openai.Client())
@traceable
def argument_generator(query: str, additional_description: str = "") -> str:
return client.chat.completions.create(
model="gpt-5.4",
messages=[
{"role": "system", "content": "You are a debater making an argument on a topic."
f"{additional_description}"
f" The current time is {datetime.now()}"},
{"role": "user", "content": f"The discussion topic is {query}"}
]
).choices[0].message.content
@traceable
def argument_chain(query: str, additional_description: str = "") -> str:
argument = argument_generator(query, additional_description)
# ... Do other processing or call other functions...
return argument
argument_chain("Why is blue better than orange?")Alternatively, you can manually log events using the Client directly or using a RunTree, which is what the traceable decorator is meant to manage for you!
A RunTree tracks your application. Each RunTree object is required to have a name and run_type. These and other important attributes are as follows:
name: str - used to identify the component's purposerun_type: str - Currently one of "llm", "chain" or "tool"; more options will be added in the futureinputs: dict - the inputs to the componentoutputs: Optional[dict] - the (optional) returned values from the componenterror: Optional[str] - Any error messages that may have arisen during the callfrom langsmith.run_trees import RunTree
parent_run = RunTree(
name="My Chat Bot",
run_type="chain",
inputs={"text": "Summarize this morning's meetings."},
# project_name= "Defaults to the LANGSMITH_PROJECT env var"
)
parent_run.post()
# .. My Chat Bot calls an LLM
child_llm_run = parent_run.create_child(
name="My Proprietary LLM",
run_type="llm",
inputs={
"prompts": [
"You are an AI Assistant. The time is XYZ."
" Summarize this morning's meetings."
]
},
)
child_llm_run.post()
child_llm_run.end(
outputs={
"generations": [
"I should use the transcript_loader tool"
" to fetch meeting_transcripts from XYZ"
]
}
)
child_llm_run.patch()
# .. My Chat Bot takes the LLM output and calls
# a tool / function for fetching transcripts ..
child_tool_run = parent_run.create_child(
name="transcript_loader",
run_type="tool",
inputs={"date": "XYZ", "content_type": "meeting_transcripts"},
)
child_tool_run.post()
# The tool returns meeting notes to the chat bot
child_tool_run.end(outputs={"meetings": ["Meeting1 notes.."]})
child_tool_run.patch()
child_chain_run = parent_run.create_child(
name="Unreliable Component",
run_type="tool",
inputs={"input": "Summarize these notes..."},
)
child_chain_run.post()
try:
# .... the component does work
raise ValueError("Something went wrong")
child_chain_run.end(outputs={"output": "foo"})
child_chain_run.patch()
except Exception as e:
child_chain_run.end(error=f"I errored again {e}")
child_chain_run.patch()
pass
# .. The chat agent recovers
parent_run.end(outputs={"output": ["The meeting notes are as follows:..."]})
res = parent_run.patch()
res.result()Once your runs are stored in LangSmith, you can convert them into a dataset. For this example, we will do so using the Client, but you can also do this using the web interface, as explained in the LangSmith docs.
from langsmith import Client
client = Client()
dataset_name = "Example Dataset"
# We will only use examples from the top level AgentExecutor run here,
# and exclude runs that errored.
runs = client.list_runs(
project_name="my_project",
execution_order=1,
error=False,
)
dataset = client.create_dataset(dataset_name, description="An example dataset")
for run in runs:
client.create_example(
inputs=run.inputs,
outputs=run.outputs,
dataset_id=dataset.id,
)Check out the LangSmith Testing & Evaluation dos for up-to-date workflows.
For generating automated feedback on individual runs, you can run evaluations directly using the LangSmith client.
from typing import Optional
from langsmith.evaluation import StringEvaluator
def jaccard_chars(output: str, answer: str) -> float:
"""Naive Jaccard similarity between two strings."""
prediction_chars = set(output.strip().lower())
answer_chars = set(answer.strip().lower())
intersection = prediction_chars.intersection(answer_chars)
union = prediction_chars.union(answer_chars)
return len(intersection) / len(union)
def grader(run_input: str, run_output: str, answer: Optional[str]) -> dict:
"""Compute the score and/or label for this run."""
if answer is None:
value = "AMBIGUOUS"
score = 0.5
else:
score = jaccard_chars(run_output, answer)
value = "CORRECT" if score > 0.9 else "INCORRECT"
return dict(score=score, value=value)
evaluator = StringEvaluator(evaluation_name="Jaccard", grading_function=grader)
runs = client.list_runs(
project_name="my_project",
execution_order=1,
error=False,
)
for run in runs:
client.evaluate_run(run, evaluator)LangSmith easily integrates with your favorite LLM framework.
We provide a convenient wrapper for the OpenAI SDK.
In order to use, you first need to set your LangSmith API key.
export LANGSMITH_API_KEY=<your-api-key>Next, you will need to install the LangSmith SDK:
pip install -U langsmithAfter that, you can wrap the OpenAI client:
from openai import OpenAI
from langsmith import wrappers
client = wrappers.wrap_openai(OpenAI())Now, you can use the OpenAI client as you normally would, but now everything is logged to LangSmith!
client.chat.completions.create(
model="gpt-5.4",
messages=[{"role": "user", "content": "Say this is a test"}],
)Oftentimes, you use the OpenAI client inside of other functions.
You can get nested traces by using this wrapped client and decorating those functions with @traceable.
See this documentation for more documentation how to use this decorator
from langsmith import traceable
@traceable(name="Call OpenAI")
def my_function(text: str):
return client.chat.completions.create(
model="gpt-5.4",
messages=[{"role": "user", "content": f"Say {text}"}],
)
my_function("hello world")We provide a convenient integration with Instructor, largely by virtue of it essentially just using the OpenAI SDK.
In order to use, you first need to set your LangSmith API key.
export LANGSMITH_API_KEY=<your-api-key>Next, you will need to install the LangSmith SDK:
pip install -U langsmithAfter that, you can wrap the OpenAI client:
from openai import OpenAI
from langsmith import wrappers
client = wrappers.wrap_openai(OpenAI())After this, you can patch the OpenAI client using instructor:
import instructor
client = instructor.patch(OpenAI())Now, you can use instructor as you normally would, but now everything is logged to LangSmith!
from pydantic import BaseModel
class UserDetail(BaseModel):
name: str
age: int
user = client.chat.completions.create(
model="gpt-5.4",
response_model=UserDetail,
messages=[
{"role": "user", "content": "Extract Jason is 25 years old"},
]
)Oftentimes, you use instructor inside of other functions.
You can get nested traces by using this wrapped client and decorating those functions with @traceable.
See this documentation for more documentation how to use this decorator
@traceable()
def my_function(text: str) -> UserDetail:
return client.chat.completions.create(
model="gpt-5.4",
response_model=UserDetail,
messages=[
{"role": "user", "content": f"Extract {text}"},
]
)
my_function("Jason is 25 years old")The LangSmith pytest plugin lets Python developers define their datasets and evaluations as pytest test cases. See online docs for more information.
This plugin is installed as part of the LangSmith SDK, and is enabled by default. See also official pytest docs: How to install and use plugins
To learn more about the LangSmith platform, check out the docs.
The LangSmith SDK is licensed under the MIT License.
The copyright information for certain dependencies' are reproduced in their corresponding COPYRIGHT.txt files in this repo, including the following:
API key authentication for write replicas.
Service-to-service JWT authentication for write replicas.
Custom authentication headers for write replicas.
Configuration for a write replica endpoint.
agent_id and agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it reject
Run Schema with back-references for posting runs.
Middleware for propagating distributed tracing context using LangSmith.
This middleware checks for the 'langsmith-trace' header and propagates the
tracing context if present. It does not start new
String node extracted from the data.
Processes a list of string nodes for masking.
Configuration options for replacing sensitive data.
Declarative rule used for replacing sensitive data.
String node processor that uses a list of rules to replace sensitive data.
String node processor that uses a callable function to replace sensitive data.
Annotated type that will be stored as an attachment if used.
Protocol for binary IO-like objects.
Example base model.
Example upload with attachments.
Example create with attachments.
Info for an attachment.
Example model.
Operations to perform on attachments.
Example update with attachments.
Enum for dataset data types.
Dataset base model.
Schema for dataset transformations.
Dataset ORM model.
Class representing a dataset version.
Base Run schema.
A Run is a span representing a single unit of work or operation within your LLM app. This could be a single call to an LLM or chain, to a prompt formatting call, to a runnable lambda
Run schema when loading from the DB.
(Deprecated) Enum for run types. Use string directly.
Run-like dictionary, for type-hinting.
agent_id and agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it rejects the
Run schema with annotation queue info.
Base class for feedback sources.
This represents whether feedback is submitted from the API, model, human labeler, etc.
API feedback source.
Model feedback source.
Feedback source type.
Feedback schema.
agent_id / agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it rejects the feedback,
so it is
Specific value and label pair for feedback.
Represents how a feedback value ought to be interpreted.
Schema used for creating feedback.
Schema for getting feedback.
TracerSession schema for the API.
Sessions are also referred to as "Projects" in the UI.
A project, hydrated with additional information.
Sessions are also referred to as "Projects" in the UI.
A protocol representing objects similar to BaseMessage.
Represents the schema for a dataset share.
Represents a rubric item assigned to an annotation queue.
Links a feedback config to a queue with optional per-queue customization.
Represents an annotation queue.
Represents an annotation queue with details.
A run identified by its full lookup key, for adding to an annotation queue.
Unlike a bare run ID, this carries the partition key (session_id and
start_time) so the run can be located directly
Configuration for batch ingestion.
Information about the LangSmith server.
Settings for the LangSmith tenant.
Represents the schema for a feedback ingest token.
Run event schema.
Timedelta input schema.
Represents the difference information between two datasets.
Represents a comparative experiment.
This information summarizes evaluation results comparing two or more models on a given dataset.
Represents a Prompt with a manifest.
Represents a listed prompt commit with associated metadata.
Represents a Prompt with metadata.
A list of prompts with metadata.
Enum for sorting fields for prompts.
A file with inline content.
A link to another agent repo.
A link to a skill repo.
An agent pulled from hub.
A skill pulled from hub.
Commit details returned from a directory commit.
Response body for POST /directories/commits.
Breakdown of input token counts.
Does not need to sum to full input token count. Does not need to have all keys.
Breakdown of output token counts.
Does not need to sum to full output token count. Does not need to have all keys.
Breakdown of input token costs.
Does not need to sum to full input cost. Does not need to have all keys.
Breakdown of output token costs.
Does not need to sum to full output cost. Does not need to have all keys.
Usage metadata for a message, such as token counts.
This is a standard representation of token usage that is consistent across models.
Usage metadata dictionary extracted from a run.
Should be the same as UsageMetadata, but does not require all keys to be present.
Response object returned from the upsert_examples_multipart method.
Example with runs.
Run statistics for an experiment.
Results container for experiment data with stats and examples.
Breaking change in v0.4.32: The 'stats' field has been split into 'feedback_stats' and 'run_stats'.
An Insights Report created by the Insights Agent over a tracing project.
A trace highlighted in an insights report summary.
High-level summary of an insights job: key points and highlighted traces.
A single cluster of runs in an insights report.
Full result of fetching an Insights report (job + clusters + summary + optional runs).
A feedback key and weight used when calculating feedback formulas.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores vi
Schema used for creating a feedback formula.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores via the LangSmith UI ins
Schema used for updating a feedback formula.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores via the LangSmith UI ins
Schema for getting feedback formulas.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores via the LangSmith UI instead. T
Represents a feedback configuration for a tenant's feedback key.
Feedback configurations define how feedback with a given key should be interpreted, including its type (continuous, categorical, or fr
An error occurred while communicating with the LangSmith API.
Internal server error while communicating with LangSmith.
Client took too long to send request body.
User error caused an exception when communicating with LangSmith.
You have exceeded the rate limit for the LangSmith API.
Couldn't authenticate with the LangSmith API.
Couldn't find the requested resource.
The resource already exists.
Couldn't connect to the LangSmith API.
Port of ExceptionGroup for Py < 3.11.
Base class for warnings.
Warning for missing API key.
Filter urllib3 warnings logged when the connection pool isn't reused.
Filter for retries from this lib.
Wrapper to filter logs with this name.
ThreadPoolExecutor that copies the context to the child thread.
A single cache entry with metadata for TTL tracking.
Cache performance metrics.
Thread-safe LRU cache with background thread refresh.
For use with the synchronous Client.
Features:
Thread-safe LRU cache with asyncio task refresh.
For use with the asynchronous AsyncClient.
Features:
Deprecated alias for PromptCache. Use PromptCache instead.
Deprecated alias for AsyncPromptCache. Use AsyncPromptCache instead.
Overrides for LangSmith runtime behavior.
This class allows overriding default async implementations for environments that don't support certain asyncio features (e.g., Temporal doesn't support ``run
A sentinel singleton class used to distinguish omitted keyword arguments from those passed in with the value None (which may have different behavior).
A class for making assertions on expectation values.
A class for setting expectations on test results.
Any additional info to be injected into the run dynamically.
Implementations of this Protocol accept an optional langsmith_extra parameter.
Manage a LangSmith run in context.
This class can be used as both a synchronous and asynchronous context manager.
agent_id / agent_environment are in beta. Agent a
A string that LangSmith serializes as [LANGSMITH SECRET].
Wrap a credential once, where it is read, and LangSmith masks it wherever it appears in a trace, at any nesting depth::
@traceable
Plugin for rendering LangSmith results.
Introduced in python 3.9.
Used for optional OTEL tracing.
Item returned by :meth:Client.list_threads.
Client for interacting with the LangSmith API.
One write destination: a server plus the credentials to reach it.
Async Client for interacting with the LangSmith API.
Optional per-mount cache configuration supported by bucket mounts.
Optional fields applied per bucket-backed sandbox mount.
Required S3 configuration for a sandbox mount.
S3 configuration for a sandbox mount.
S3-backed sandbox mount specification.
Required GCS configuration for a sandbox mount.
GCS configuration for a sandbox mount.
GCS-backed sandbox mount specification.
Git ref selected for a sandbox mount.
Required Git configuration for a sandbox mount.
Git configuration for a sandbox mount.
Git-backed sandbox mount specification.
Required Context Hub configuration for a sandbox mount.
Context Hub configuration for a sandbox mount.
Read-only Context Hub-backed sandbox mount specification.
AWS credentials used by the backend to authenticate S3 mounts.
IAM role restricted by the backend to this sandbox's S3 mount scopes.
GCP credentials used by the backend to authenticate GCS mounts.
Provider auth blocks for sandbox mounts.
Public mount config sent to the sandbox API.
What a command handle needs from its transport to steer a command.
A one-way transport supplies this too, and raises from the methods it cannot honor.
Async equivalent of :class:StreamControl.
Result of executing a command in a sandbox.
Lightweight provisioning status for any async-created resource.
The user, working directory and environment commands run with.
Mirrors docker run -u / -w / -e: user and work_dir replace the
layer below, env_vars merge into it key by key. It applie
One filesystem entry returned by :meth:Sandbox.glob.
Entries matching a glob pattern.
One matching line found by :meth:Sandbox.grep.
Lines matching a literal search.
What a HEAD on a sandbox file reports, without transferring it.
Bytes returned by a ranged read, and where they sit in the file.
Represents a sandbox snapshot.
Snapshots are built from Docker images or captured from running sandboxes. They are used to create new sandboxes.
One tag published under a snapshot name, and the snapshot it resolves to.
Authenticated URL for accessing an HTTP service running in a sandbox.
Properties auto-refresh the token transparently when it nears expiry.
HTTP helper methods (.get, .post, etc.) inject the
Async variant of :class:ServiceURL with auto-refreshing token.
Properties and HTTP helpers are async. Use with
:meth:AsyncSandboxClient.service or :meth:AsyncSandbox.service.
Example::
sv
A link that downloads one sandbox file with no LangSmith credential.
The link is pinned to the sandbox, the file path, and the response headers, so it cannot be repointed at another file. It is pinne
Service URL gated by LangSmith login rather than a token.
The grant is durable: there is no token to carry and no expiry, so the URL is only usable from a browser signed in to LangSmith. That is also
A single chunk of streaming output from command execution.
Handle to a running command with streaming output and auto-reconnect.
Iterable, yielding OutputChunk objects (stdout and stderr interleaved in arrival order). Access .result after iteration to get th
Async handle to a running command with streaming output and auto-reconnect.
Async iterable, yielding OutputChunk objects (stdout and stderr interleaved in arrival order). Access .result after iterati
Represents an active sandbox for running commands and file operations.
This class is typically obtained from SandboxClient.sandbox() and supports the context manager protocol for automatic cleanup.
TCP tunnel to a port inside a sandbox.
Opens a local TCP listener and forwards each accepted connection through a yamux-multiplexed WebSocket to the daemon, which dials the target port inside the san
Async wrapper around :class:Tunnel.
The underlying tunnel runs in background threads (TCP listener + bridges); async context-manager methods delegate to the sync tunnel via the event loop's executo
Raised when a sandbox user token or callback signature fails verification.
The LangSmith user a service URL request was made by.
The sandbox whose outbound request triggered a proxy callback.
Snapshot of the outbound request, sent for full_request callbacks.
A verified proxy callback payload.
Verifies tokens LangSmith signs for code running in or behind a sandbox.
Keys are fetched from LangSmith's JWKS endpoint and cached.
A single multiplexed stream within a yamux session.
Streams are created via :meth:YamuxSession.open_stream and provide
blocking read/write/close with per-stream flow control.
Client-side yamux session over a byte-stream connection.
The connection must implement read(n) -> bytes, write(data) -> int,
and close() -> None. Typically this is a :class:_WSAdapter
Represents an active sandbox for running commands and file operations async.
This class is typically obtained from AsyncSandboxClient.sandbox() and supports the async context manager protocol for aut
Sync httpx transport that retries on transient errors.
Retries on:
Async httpx transport that retries on transient errors.
Async equivalent of RetryTransport. See RetryTransport for details.
Async client for interacting with the Sandbox Server API.
This client provides an async interface for managing sandboxes and snapshots.
Client for interacting with the Sandbox Server API.
This client provides a simple interface for managing sandboxes and snapshots.
Base exception for sandbox client errors.
Raised when the API endpoint returns an unexpected error.
For example, this is raised for wrong URL or path.
Raised when authentication fails (invalid or missing API key).
Raised when connection to the sandbox server fails.
Raised when a transient failure occurs before a command can start.
run() retries this error with the same command ID, so the server can
deduplicate an attempt whose outcome is unknown.
Raised when the socket fails or times out before the WebSocket handshake.
The execute frame was never sent, so re-issuing the same command ID cannot double-run a command.
Raised when the server sends a 1001 Going Away close frame.
This indicates a server hot-reload, not a true connection failure. The command is still running on the server.
This is a subclass of Sandb
Raised when a resource is not found.
Raised when an operation times out.
Raised when deleting a resource that is still in use.
Raised when creating a resource that already exists.
Raised when updating a resource name to one that already exists.
Raised when request validation fails.
This includes:
Raised when organization quota limits are exceeded.
Users should contact technical support via our Support Portal (https://support.langchain.com) to increase quotas.
Raised when resource provisioning fails.
Raised when dataplane_url is not available for the sandbox.
This occurs when the sandbox-router URL is not configured for the cluster.
Raised when attempting to interact with a sandbox that is not ready.
Raised when a sandbox operation fails (run, read, write).
Raised when a command exceeds its timeout.
Base exception for TCP tunnel errors.
The daemon rejected the port as not allowed.
Nothing is listening on the target port inside the sandbox.
Protocol version mismatch between the tunnel client and the daemon.
A secret value that can be used by sandbox proxy rules.
Grant letting code inside a sandbox call the LangSmith API as the creator.
mode is "INHERIT" for everything the creator can do, or
"EXPLICIT" for the subset named in permissions. Perm
Represents the results of an evaluate() call.
This class provides an iterator interface to iterate over the experiment results as they become available. It also provides methods to access the experim
Represents the results of an evaluate_comparative() call.
This class provides an iterator interface to iterate over the comparison results, indexed access by example ID, and properties to access the
Configuration for a categorical score.
Configuration for a continuous score.
A class for building LLM-as-a-judge evaluators.
.. deprecated:: 0.5.0
LLMEvaluator is deprecated. Use openevals instead: https://github.com/langchain-ai/openevals
A category for categorical feedback.
Configuration to define a type of feedback.
Applied on on the first creation of a feedback_key.
Evaluation result.
Batch evaluation results.
This makes it easy for your evaluator to return multiple metrics at once.
Evaluator interface class.
Feedback scores for the results of comparative evaluations.
These are generated by functions that compare two or more runs, returning a ranking or other feedback.
A dynamic evaluator that wraps a function and transforms it into a RunEvaluator.
This class is designed to be used with the @run_evaluator decorator, allowing
functions that take a Run and an o
Compare predictions (as traces) from 2 or more runs.
Grades the run's string input, output, and optional answer.
.. deprecated:: 0.5.0
StringEvaluator is deprecated. Use openevals instead: https://github.com/langchain-ai/openevals
For parameters with a meaningful None value, we need to distinguish between the user explicitly passing None, and the user not passing the parameter at all.
User code shouldn't need to use not_given
To explicitly omit something from being sent in a request, use omit.
# as the default `Content-Type` header is `application/json` that will be sent
client.post("/upload/files", files={"file":
Represents a type that has inherited from Generic
The __orig_bases__ property can be used to determine the resolved
type variable for a given base class.
Compute the run ID used for a secondary tracing replica.