roboto.storage.http_range_reader
Seekable HTTP byte-range reader for efficient partial access to remote files.
This module provides a seekable file-like object backed by HTTP Range requests, enabling efficient random access to remote files (e.g., MCAP files on S3) without downloading the entire file.
Module Contents
HttpRangeReader
A seekable, buffered byte-range reader backed by an HTTP URL.
Uses HTTP range requests so only the requested byte ranges are fetched, allowing efficient partial access to remote files (e.g., reading just the MCAP summary/index section at the end of a file without downloading the full data payload).
Reads are satisfied from an in-memory sparse cache. HTTP requests are only issued on a cache miss, fetching READ_AHEAD_SIZE bytes at a time. Unlike a simple single-buffer approach, this cache retains all fetched regions, so seeking back to previously-read data doesn’t trigger re-fetches.
This class implements the IO[bytes] protocol methods needed by mcap.reader.
Uses urllib3 connection pooling to reuse HTTP connections across requests, reducing TCP handshake and TLS negotiation overhead.
Parameters
url strread_ahead_size intHttpRangeReader.close()
Close the reader and release resources.
Return type
HttpRangeReader.prefetch_range()
Prefetch a byte range using parallel HTTP requests.
Byte spans already in the cache (e.g., placed there by the footer read-behind at open, which covers the whole file when it is small) are not re-fetched; only the uncovered gaps are requested.
Parameters
start intStart byte offset (inclusive)
end intEnd byte offset (inclusive)
Return type
HttpRangeReader.read()
Parameters
size intReturn type
HttpRangeReader.readable()
Return type
HttpRangeReader.seek()
Parameters
offset intwhence intReturn type
HttpRangeReader.seekable()
Return type
Properties
HttpRangeReader.tell()
Return type
HttpRangeReader.writable()
Return type
as_io_bytes()
Cast an HttpRangeReader to typing.IO[bytes] for type-checking purposes.
Parameters
reader HttpRangeReaderReturn type