An agent needs to read a web page. The implementation exposes fetch_url(url) from a worker that can also reach internal services. A user, a retrieved document, or a model-generated link now influences where that worker sends requests. The apparent feature is browsing; the actual capability is network access from your infrastructure.
An AI URL-fetching tool needs deterministic destination authorization outside the model and enforcement at the network boundary. A prompt telling the model to avoid private addresses is not an access-control mechanism.
This article focuses on server-side request forgery, or SSRF: inducing a server to make requests it should not make. It complements agent control-plane design rather than replacing authorization, sandboxing, or prompt-injection defenses.
Reduce the tool’s authority first #
Start with a product question: must the agent fetch arbitrary URLs at all? A documentation assistant might only need a small set of approved public documentation origins. An internal support assistant might need a typed read_ticket(ticket_id) tool instead of general network access.
These are different products with different security costs. Prefer a constrained connector when it solves the problem. Arbitrary browsing needs a more capable isolation boundary, ongoing policy maintenance, and a response-processing budget.
OWASP distinguishes allowlisted destination scenarios from unrestricted destination scenarios. It recommends defense in depth, including network controls, careful address validation, and redirect handling. DNS resolution introduces further validation concerns. OWASP SSRF guidance.
Separate the gates #
untrusted model proposal
-> tool authorization and argument validation
-> approved destination policy
-> isolated fetcher / enforced egress boundary
-> bounded response reader
-> untrusted document contentEach gate answers a different question. Tool authorization decides whether this user and agent may perform the operation. Destination policy decides where it may go. The egress layer controls the actual connection. Response handling bounds resource consumption and prevents downloaded instructions from silently gaining tool authority.
Do not let a model call select proxy settings, attach arbitrary authorization headers, or forward incoming cookies. Those expand the capability beyond “read this public page.” If a connector needs credentials, scope them to that connector and its verified destination.
A small input policy, deliberately not a complete defense #
For a documentation-only tool, a strict origin check is easy to review:
const allowedOrigins = new Set([
'https://www.postgresql.org',
'https://kubernetes.io',
])
function approveDocumentationUrl(input) {
if (typeof input !== 'string' || input.length > 2048)
throw new Error('Invalid URL input')
const url = new URL(input)
if (url.protocol !== 'https:' || url.username || url.password)
throw new Error('Only credential-free HTTPS URLs are allowed')
if (!allowedOrigins.has(url.origin))
throw new Error('Destination not approved')
url.hash = ''
return url.href
}The exact-origin comparison rejects lookalike suffixes and unapproved ports. It is intentionally narrower than checking whether a string contains a trusted hostname. It still permits every path and query on the approved origin; tighten those if the connector needs only a specific documentation subtree.
This function is only the input gate. It does not resolve addresses, pin a connection to a validated address, constrain a proxy, read a body, or stop an approved server from redirecting elsewhere. Do not advertise it as a secure standalone fetcher.
Authorize the connection, not an earlier DNS answer #
The dangerous gap is between checking a name and connecting to it. If those actions use different resolutions or transport paths, a previously approved answer may not describe the destination actually contacted. Your fetch service must enforce its destination rules where the connection is established.
For arbitrary public browsing, explicitly handle IPv4 and IPv6, loopback, private and link-local destinations, reserved ranges, and infrastructure-specific metadata endpoints. Use a maintained policy implementation rather than a few string prefixes. The fetch worker should have no route to sensitive private services even when application validation contains a mistake.
TLS verification must still use the intended hostname. “Pin the IP” is not permission to disable certificate checks. Proxies also need to participate in the policy; otherwise the application validates one address while a proxy resolves and connects to another.
These are architectural requirements, not features implemented by the small function above. A security review should inspect the actual HTTP client, resolver, proxy, and network rules together.
Redirects and bodies need budgets #
For the narrow connector, reject redirects unless the product needs them. Fetch exposes an error redirect mode; checking the final response URL after an automatic redirect is too late to prevent the redirected request. Fetch redirect behavior.
If redirects are required, process a bounded number manually and apply the full policy at each hop. Bound connection time, total request time, and streamed response bytes. A Content-Length header is not proof that the body will stay within your limit. Account for decompression and parser expansion too.
A fetched page remains untrusted input. It can contain instructions that attempt to alter the agent’s next action. Keep retrieved content separate from trusted instructions and require deterministic checks before subsequent tool execution. OWASP prompt-injection guidance.
Failure policy worth testing #
| Condition | Policy |
|---|---|
| Unapproved origin or credentials in URL | Reject before network access |
| DNS or connection destination violates policy | Reject; never fall back to an unrestricted fetch |
| Redirect without permission | Stop; report an unsupported destination change |
| Oversized or slow response | Abort within the configured resource budget |
| Retrieved page requests another privileged action | Treat it as data, not authority |
Test lookalike domains, alternate ports, redirects, IPv6 destinations, slow bodies, and proxy behavior in an isolated environment you control. Unit tests of URL parsing are useful evidence for the first gate only; they are not a network penetration test.
For observability, record a policy decision and sanitized destination identifier. Avoid logging full signed URLs, embedded secrets, or entire downloaded documents by default.
The trade-off is coverage versus authority. A narrow connector may reject legitimate links but has a smaller policy to defend. A general browser is more useful and materially harder to secure. Make that cost visible before choosing the tool interface. More on the surrounding design is in production AI and why valid structured output is not authorization.