roboto.formats.parquet.fetch
Module Contents
logger
open_parquet_file()
Open a remote Parquet file under a cache policy, from the cheapest available source.
Dispatches on choose_fetch_mode(): an already-cached copy is reused, a download is performed (concurrency-safe, atomic) when the policy calls for one, and otherwise the file is streamed over HTTP range requests without touching disk.
Parameters
url_provider Callable[[], str]Resolves the file’s signed download URL. Called at most once, and only when the chosen mode actually needs the URL.
cache_outfile Optional[pathlib.The file’s stable local cache path, or None when no cache location is configured (forces streaming).
The caller’s cache policy.
estimated_column_count intHow many columns the read is expected to project; informs the ADAPTIVE download-vs-stream choice.
size_bytes Optional[int]The backing object’s size in bytes when the server reports it; None when unknown. It gates cache reuse: a present cached file whose size does not match is treated as stale and re-fetched. On the STREAM path it lets a known-large file skip the whole-file head probe; on the DOWNLOAD path it verifies the downloaded file is complete before it is promoted to the cache.
Returns
An open pyarrow.parquet.ParquetFile.
parquet_file_from_url()
Open a Parquet file over HTTP via a signed URL (no local download).
A single ranged GET over fsspec’s shared HTTP session probes the first _STREAM_WHOLE_FILE_PROBE_BYTES of the file. A file smaller than the probe arrives whole in that one request and is read from an in-memory buffer; a larger file (or a failed probe) falls back to HTTP range-request streaming through pyarrow’s filesystem layer.
When size_bytes is known and at least _STREAM_WHOLE_FILE_PROBE_BYTES, the file is known-large up front: the whole-file probe could never win (a BufferReader read is taken only for sub-threshold files), so it is skipped and the file is range-streamed directly, avoiding a wasted 16 MiB GET. size_bytes of None (an older server omits the size) preserves the probe-then-decide behavior.
Parameters
signed_url strThe file’s signed download URL.
size_bytes Optional[int]The backing object’s size in bytes when the server reports it; None when unknown.
Raises
ValueErrorThe probe succeeds but the object is empty (0 bytes), which is not a readable Parquet file.
Return type