roboto.experimental.topics.decode.common
What reading one file of a read plan’s partition takes and gives.
Module Contents
FileDecodeParams
What decoding a file takes beyond the read plan: how to reach the file and whether to cache it.
Caching applies to Parquet files only; MCAP files always stream.
Attributes
FileDecodeParams.cache_policy
Whether fetched Parquet files are cached to local disk.
FileDecodeParams.signed_url_resolver
Mints a signed download URL for a file.
FileDecoder
Bases: abc.ABC
Decodes the fields one file supplies to a partition into RecordBatches.
Opening a decoder opens its file, so its fields are known before the first batch. Close it when done, or use it as a context manager.
FileDecoder.batches()
The window’s rows, in the file’s stored row order; iterate it once.
A partition that declares a data_range gets only the window’s rows inside that slice of the file. Each batch has the columns of topic_data_schema() over value_fields: the row number, the timestamp, then the value columns. A row’s number is its 0-based position among the file’s rows of the topic, counting every stored row, including rows outside the window or the data_range and rows with a null timestamp, so a row has the same number in every file of its partition. The timestamp is absolute: the stored value in nanoseconds plus the partition’s time_offset_ns. Batch boundaries carry no meaning.
Return type
FileDecoder.close()
Release the file. Safe to call more than once.
Return type
FileDecoder.struct_field_names()
The names of the fields of the struct at path in the file, in the file’s order.
None when the file has no struct at path.
Parameters
Return type
Properties
FileDecoder.value_fields
The value columns, one per top-level field the file supplies, sorted by name, comparing Unicode code points.
Each struct keeps the fields the file supplies, in the order the file stores them.
FileDecoderOpener
Opens a FileDecoder of a group’s file for its partition and the partition’s window.
The window is absolute and includes both ends. A decoder keeps the rows whose absolute timestamp lies in it.
ScanTaskGroup
The scan tasks of a partition that read one file, and the fields they supply.
Every scan task in the group agrees on the file’s format, transformations and topic name.
Attributes
ScanTaskGroup.format
ScanTaskGroup.object
ScanTaskGroup.supplies
In the order of the leaf-most paths they come from.
For one path, the supplied field at the path itself comes before the subtrees inside it, which come in plan order. Empty when the file is read only for its row numbers and timestamps.
ScanTaskGroup.topic_name
The topic the scan tasks read from the file; see topic_name.
SignedUrlResolver
Resolves a file id (fs_node_id) to a signed download URL.
SuppliedField
The field at path, which one file supplies to the read, less the fields at excluded, which other scan tasks supply (they may read the same file).
Attributes
SuppliedField.excluded
Paths strictly inside path, none inside another.