HTTP

omniload reads a file addressed by an HTTP or HTTPS URL, using the same reader stack as every other filesystem source. See that page for the supported formats, format and reader hints, and type normalization.

URI format

The URI is the URL of the file, exactly as you would open it.

https://example.org/path/to/data.csv
http://example.org/path/to/data.parquet

The query string is part of the address

Unlike the other schemes in this family, an HTTP URL’s query string is not connection configuration: it is sent to the server. That is what makes a presigned URL work, since the signature lives in the query and is computed over its encoded form, and an escape that carries meaning (%2F) is preserved rather than decoded. An escape of an unreserved character is normalized (%7E becomes ~), which every canonical signing scheme treats as the same value:

omniload ingest \
    --source-uri   'https://bucket.s3.amazonaws.com/exports/orders.csv?X-Amz-Signature=...&X-Amz-Expires=900' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.orders'

Quote the URI in your shell: & would otherwise background the command.

Note

Because the query is addressing information, connection options cannot ride in it, the way they do for s3:// and the other schemes. They are available when you build the source yourself, where a keyword argument reaches the underlying fsspec HTTP filesystem:

from dlt_filesystem.source.fsspec.http import HttpFilesystemSource

source = HttpFilesystemSource().dlt_source(
    "https://example.org/data.jsonl", "", block_size=65536
)

omniload ingest and run_ingest have no channel for them.

Authentication

Credentials in the URL’s userinfo become an HTTP basic-authentication header. Percent-encode any character that a URI cannot carry literally, @ and / included:

https://<USERNAME>:<PASSWORD>@example.org/private/data.csv

Environment proxy settings and .netrc are honoured.

Extended type support

Type	Support	Remarks
Formats	✅	Every format the filesystem page lists.
Ranges	✅	A file is read in ranges where the server serves them.
Globs	❌	One concrete URL per source; see Limitations.

Examples

Load a public CSV file into DuckDB

omniload ingest \
    --source-uri   'https://example.org/path/to/data.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Name the format for a URL that has no useful extension

An API endpoint rarely ends in .csv. Append a #format fragment:

omniload ingest \
    --source-uri   'https://api.example.org/exports/latest#csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

How much is transferred

Where the server honours byte ranges, the file is read in ranges rather than downloaded whole, so a line-delimited document starts producing rows before all of it has arrived. Two cases are read whole instead, because they cannot be read any other way:

• a server that ignores Range and answers with the entire body;

• a response with no Content-Length (ordinary Transfer-Encoding: chunked), which leaves the file with no known size and therefore no seekable form.

Parquet is read whole in every case: pyarrow asks for a file’s entire data section in one read, independently of the transport.

Limitations

• One concrete URL per source. Wildcards are not supported: HTTP has no listing operation, only whatever links a server happens to render, so a glob cannot be resolved reliably.

• No incremental file selection. --filesystem-incremental is refused for HTTP. A response need not carry a Last-Modified header, and a missing one would read as “just now”, so every file would be reloaded on every run while the run reported that it had filtered.

• Read only. There is no portable way to write a file over plain HTTP.

Both of the first two are the subject of ongoing work; see the WebDAV source for an HTTP-based transport that does support listing.