roboto.storage.sparse_buffer
Seekable read-only buffer backed by sparse in-memory byte regions.
Provides a file-like interface (IO[bytes]) over non-contiguous byte data, allowing efficient representation of partially-fetched remote files. Only the regions that have been explicitly loaded are stored in memory.
Module Contents
SparseBuffer
A seekable, read-only file-like object backed by sparse in-memory byte regions.
Stores fetched byte regions and provides a standard IO[bytes] interface for reading from them. Regions are automatically merged when they overlap or are adjacent, keeping the internal representation compact.
This is intended to be used as: 1. The cache backend for HttpRangeReader (sparse storage with smart fetching) 2. The stream for mcap.reader.SeekingReader after bulk-fetching byte ranges
Usage
buf = SparseBuffer(file_size=1000)
buf.add_region(0, b"MCAP_MAGIC") # header
buf.add_region(900, b"footer_data") # footer
buf.seek(0)
# 0
buf.read(10)
# b'MCAP_MAGIC'Parameters
file_size intSparseBuffer.add_region()
Store a byte region at the given file offset.
Merges with any overlapping or adjacent existing regions.
Parameters
offset intByte offset within the virtual file.
data bytesRaw bytes to store at that offset.
Return type
SparseBuffer.clear()
Remove all cached regions.
Return type
SparseBuffer.find_region()
Check if [start, start+size) is fully contained in a cached region.
Parameters
start intStart byte offset.
size intNumber of bytes.
Returns
The requested bytes if fully cached, None otherwise.
SparseBuffer.read()
Read up to size bytes from the current position.
If the current position is within a cached region, returns available bytes (may be fewer than requested if the region ends before size bytes). If the current position is not in any cached region (a gap), returns b”“.
This allows callers to detect partial hits and fetch missing data: - len(result) == size: fully satisfied - 0 < len(result) < size: partial hit, more data may be needed - len(result) == 0: gap at current position, caller should fetch
Parameters
size intMaximum number of bytes to read. -1 means read to end of file.
Returns
Bytes read from cached regions, or b”” if at a gap or past EOF.
SparseBuffer.readable()
Return True - this buffer supports reading.
Return type
Properties
SparseBuffer.regions
List of (start, end) byte ranges currently cached.
End is exclusive. Useful for fetch planning and debugging.
SparseBuffer.seek()
Move the read position.
Parameters
offset intByte offset relative to the position indicated by whence.
whence int0=SEEK_SET (start), 1=SEEK_CUR (current), 2=SEEK_END (end).
Returns
The new absolute position.
SparseBuffer.seekable()
Return True - this buffer supports seeking.
Return type
Properties
SparseBuffer.tell()
Return the current read position.
Return type
SparseBuffer.writable()
Return False - this buffer is read-only.
Return type