Skip to main content
The HTTP provider fetches content from web URLs. It handles both http:// and https:// schemes, managing connection pooling and timeouts automatically. The HTTP provider is registered by default—no configuration required.

​
Configuration

Configure the HTTP provider in colin.toml to customize timeout behavior:
colin.toml
[[providers.http]]
timeout = 60.0
FieldDescriptionDefault
timeoutRequest timeout in seconds30.0
Without explicit configuration, Colin registers the HTTP provider with default settings.

​
Template Functions

The HTTP provider exposes one function:
FunctionPurpose
get()Fetch content from a URL

​
Get

Use colin.http.get() to fetch content from any URL:
models/sources/api-data.md
---
name: API Data
---

{{ colin.http.get('https://api.example.com/data.json') }}
URLs without a scheme default to https://: 
{{ colin.http.get('example.com/api/users') }}
The function returns an HTTPResource object with content and metadata:
PropertyDescription
.contentResponse body as a string
.uriThe requested URL
.content_typeContent-Type header value
.updatedLast-Modified timestamp, if available
Access properties directly in templates: 
{% set response = colin.http.get('https://api.example.com/status') %}

Content-Type: {{ response.content_type }}
Last Modified: {{ response.updated }}

{{ response.content }}

​
Objects

HTTPResource - Returned by get():
PropertyTypeDescription
uristrRequested URL
contentstrResponse body
content_typestr | NoneContent-Type header
updateddatetime | NoneLast-Modified time
The object supports conversion to RefResult via the to_ref_result() method.

​
URI Support

URLs starting with http:// or https:// work directly with ref(): 
{{ ref('https://api.example.com/config.json') }}
This is equivalent to using colin.http.get() but automatically tracks the dependency.

​
Dependency Tracking

HTTP resources can be tracked as dependencies using ref(): 
{# Fetched but not tracked #}
{{ colin.http.get('https://api.example.com/config').content }}

{# Tracked in manifest #}
{{ ref(colin.http.get('https://api.example.com/config')).content }}
Tracked dependencies appear in the manifest: 
{
  "documents": {
    "sources/config": {
      "refs_evaluated": ["https://api.example.com/config"]
    }
  }
}

​
Staleness Detection

The HTTP provider supports efficient staleness checking via HEAD requests. When Colin evaluates whether a document needs recompilation, it checks Last-Modified headers without downloading full content. If the server provides a Last-Modified header, Colin compares it against the cached timestamp. Resources without this header are treated as potentially stale.

​
Working with APIs

The HTTP provider works well with JSON APIs. Combine with LLM functions to process API responses:
models/sources/github-issues.md
---
name: GitHub Issues
---

{% set issues = colin.http.get('https://api.github.com/repos/owner/repo/issues') %}

{{ issues.content }}
models/context/issue-summary.md
---
name: Issue Summary
---

{{ ref('sources/github-issues') | llm_extract('open issues with their priorities and assignees') }}

​
Authenticated Requests

The HTTP provider supports unauthenticated requests only. For APIs requiring authentication, use an MCP server that handles credentials:
colin.toml
[[providers.mcp]]
name = "github"
command = "uvx"
args = ["mcp-server-github"]
env = { GITHUB_TOKEN = "${GITHUB_TOKEN}" }
Then access authenticated endpoints through MCP: 
{{ colin.mcp.github.resource('repo://owner/repo/issues') }}

​
Error Handling

The provider raises specific exceptions for HTTP errors:
StatusException
404FileNotFoundError
Other 4xx/5xxhttpx.HTTPStatusError
Template rendering stops on HTTP errors. Ensure URLs are valid and accessible before compilation.

​
Lifecycle

The HTTP provider manages an httpx.AsyncClient through its lifespan context. Colin creates the client when starting compilation and closes it on shutdown. This connection pooling improves performance when fetching multiple URLs from the same host.