roboto.storage.cache

Column count at or above which an ADAPTIVE read downloads the whole file instead of streaming.

Streaming projects only the requested columns but pays a per-column HTTP cost, so reading many columns over the network grows slow; downloading fetches the whole file once. 10 is the crossover: below it, streaming’s wasted-bytes savings win; at or above it, the per-column overhead dominates and the local copy is reliably faster.

Bases: str, enum.Enum

Governs whether a fetched data file is cached to local disk before reading.

The policy applies to formats with a disk-cache path (Parquet today); a format that always streams (MCAP) ignores it.

Bases: enum.Enum

How a single remote file will be opened, as chosen by choose_fetch_mode().

Report whether the cached file can be reused instead of downloaded again.

Returns True when the file exists and either its size matches expected_size or expected_size is None. A size mismatch means the file’s content changed since the cache was written, so the cached bytes are stale and must not be reused; the file is then treated as absent and re-downloaded. expected_size of None (no size reported) reuses any existing file.

Pick the cheapest way to open a remote columnar file under a cache policy.

A previously-downloaded file is always cheaper to open than streaming over HTTP, regardless of how many columns the current read projects, so an existing cached copy wins under every policy except NEVER (which never touches the disk cache at all, not even to read it).

Download a remote file to outfile safely under concurrency.

Acquires a per-path lock to dedupe in-process downloads, double-checks existence (another thread may have completed the download while we were waiting), creates the cache directory lazily, and writes via a uniquely named .part file followed by os.replace(). The rename is atomic on POSIX and Windows, so:

• Readers never see a partial file at outfile.
• Two processes racing both produce a complete file; the second os.replace simply overwrites the first.
• If the download raises, the .part file is removed and outfile is left untouched.

When expected_size is supplied, the written .part file’s size is verified against it before the atomic rename. A truncated body — which urllib.request.urlretrieve() does not flag when the response carries no Content-Length — fails this check, so the .part is discarded and nothing is promoted to the cache: a partial download is never made sticky.

Return the in-process download lock for a cache path, creating it on first use.