roboto.domain.topics.parquet
Submodules
Package Contents
ParquetParser
Parameters
source pathlib.min_required_row_group_size intsmall_row_group_count_threshold intProperties
ParquetParser.extract_timestamp_info()
Parameters
timestamp_column_name Optional[str]timestamp_unit Optional[Union[str, roboto.Properties
ParquetParser.fields
ParquetParser.find_timestamp_field_by_type()
Return type
ParquetParser.get_data_for_column()
Parameters
column_name strReturn type
ParquetParser.get_timestamp_field_by_name()
Parameters
column_name strReturn type
ParquetParser.is_parquet_file()
Parameters
path pathlib.Return type
ParquetParser.requires_rewrite()
Parameters
Return type
ParquetParser.rewrite()
Parameters
outfile pathlib.target_row_group_size_bytes intReturn type
ParquetTopicReader
Bases: roboto.domain.topics.topic_reader.TopicReader
Private interface for retrieving topic data stored in Parquet files.
Parameters
roboto_client roboto.cache_dir Optional[pathlib.ParquetTopicReader.accepts()
Parameters
message_paths_to_representations collections.Return type
ParquetTopicReader.get_data()
Parameters
message_paths_to_representations collections.start_time Optional[int]end_time Optional[int]timestamp_message_path_representation_mapping Optional[roboto.Return type
ParquetTopicReader.get_data_as_df()
Parameters
message_paths_to_representations collections.start_time Optional[int]end_time Optional[int]timestamp_message_path_representation_mapping Optional[roboto.Return type
generate_message_path_requests()
Generate AddMessagePathRequest objects for all fields in a Parquet schema.
Traverses the schema recursively to generate message paths for nested types (structs, lists) in addition to top-level fields.
Parameters
ParquetParser instance containing the schema and data.
Timestamp information for the topic.
max_depth intMaximum recursion depth for nested types (default: 10).
Yields
AddMessagePathRequest objects for each field and nested field in the schema.
Return type
Usage
For a schema with a struct column `position: struct<x: float, y: float>`: - Yields position (Object) - Yields position.x (Number) - Yields position.y (Number)
For a schema with `values: list<float64>`: - Yields values (NumberArray)
For a schema with `points: list<struct<x: float, y: float>>`: - Yields points (Array) - Yields points.x (Number) - Yields points.y (Number)
make_topic_filename_safe()
Parameters
name strreplacement_char strReturn type
upload_representation_file()
Parameters
file_path pathlib.association roboto.caller_org_id Optional[str]roboto_client Optional[roboto.Return type