---
sidebar:
  hidden: true
title: roboto.experimental.topics.batch_transforms
---
Representation conversion for topic-data RecordBatches.

Topic data moves through the read path as Arrow RecordBatches in its public shape: one column per top-level projected field, with struct/list types mirroring the schema tree, plus one dedicated timestamp column of absolute Unix-epoch nanoseconds (`int64`) marked by field metadata ([`TIMESTAMP_FIELD_METADATA_KEY`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.TIMESTAMP_FIELD_METADATA_KEY)).

Inside the read path, a decoded batch also leads with a row number column ([`topic_data_schema()`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.topic_data_schema)): each row's `uint64` position among its file's rows of the topic, marked by [`ROW_NUMBER_FIELD_METADATA_KEY`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.ROW_NUMBER_FIELD_METADATA_KEY). When a partition's fields are stored across several files, merging those files compares this column to confirm every file holds the same rows. Batches returned to a caller leave it out ([`drop_row_number_column()`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.drop_row_number_column)).

[`flatten_table()`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.flatten_table) expands a table's struct columns into dot-delimited leaf columns, which `Topic.get_data_as_df(flatten=True)` returns as the value columns of its DataFrame.

It also exposes the helpers that construct and locate the timestamp and row number columns ([`topic_data_schema()`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.topic_data_schema), [`timestamp_field()`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.timestamp_field), [`timestamp_column_index()`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.timestamp_column_index), [`row_number_column_index()`](/docs/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.row_number_column_index)), which the read path uses to mark and find those columns by metadata rather than name.

## Module Contents

### MARKER_FIELD_METADATA_VALUE

```python
roboto.experimental.topics.batch_transforms.MARKER_FIELD_METADATA_VALUE = b'true'
```

`from roboto.experimental.topics.batch_transforms import MARKER_FIELD_METADATA_VALUE`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L42-L42)

Value of the metadata key on the column it marks, for both [`TIMESTAMP_FIELD_METADATA_KEY`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.TIMESTAMP_FIELD_METADATA_KEY) and [`ROW_NUMBER_FIELD_METADATA_KEY`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.ROW_NUMBER_FIELD_METADATA_KEY).

### ROW_NUMBER_FIELD_METADATA_KEY

```python
roboto.experimental.topics.batch_transforms.ROW_NUMBER_FIELD_METADATA_KEY = b'roboto.topic_data.row_number'
```

`from roboto.experimental.topics.batch_transforms import ROW_NUMBER_FIELD_METADATA_KEY`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L62-L62)

Arrow field-metadata key marking the row number column of a decoded topic-data batch.

The row number column holds this key with the value `b"true"` ([`MARKER_FIELD_METADATA_VALUE`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.MARKER_FIELD_METADATA_VALUE)); a column holding the key with any other value is a value column.

### ROW_NUMBER_FIELD_NAME

```python
roboto.experimental.topics.batch_transforms.ROW_NUMBER_FIELD_NAME = '_row'
```

`from roboto.experimental.topics.batch_transforms import ROW_NUMBER_FIELD_NAME`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L68-L68)

Requested name of the row number column; `_` is appended while another column of the batch has it.

The column's identity is its metadata marker ([`ROW_NUMBER_FIELD_METADATA_KEY`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.ROW_NUMBER_FIELD_METADATA_KEY)), never this name.

### TIMESTAMP_FIELD_METADATA_KEY

```python
roboto.experimental.topics.batch_transforms.TIMESTAMP_FIELD_METADATA_KEY = b'roboto.topic_data.timestamp'
```

`from roboto.experimental.topics import TIMESTAMP_FIELD_METADATA_KEY`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L46-L46)

Arrow field-metadata key marking the per-row timestamp column of a topic-data batch.

The timestamp column holds this key with the value `b"true"` ([`MARKER_FIELD_METADATA_VALUE`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.MARKER_FIELD_METADATA_VALUE)); a column holding the key with any other value is a value column.

### TIMESTAMP_FIELD_NAME

```python
roboto.experimental.topics.batch_transforms.TIMESTAMP_FIELD_NAME = '_index'
```

`from roboto.experimental.topics.batch_transforms import TIMESTAMP_FIELD_NAME`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L52-L52)

Name of the emitted per-row timestamp column.

Source-neutral by design: the column always carries the resolved timeline's absolute Unix-epoch nanoseconds, whatever that source is (message log time, publish time, or a schema field), so the name asserts no particular origin. It matches the `_index` index that [`Topic.get_data_as_df()`](/reference/python-sdk/roboto/experimental/topics/topic#roboto.experimental.topics.topic.Topic.get_data_as_df) labels its rows with. The column's real identity is its metadata marker ([`TIMESTAMP_FIELD_METADATA_KEY`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.TIMESTAMP_FIELD_METADATA_KEY)), never this name, which is uniquified by suffixing when a projected field already claims it.

### drop_row_number_column()

```python
def roboto.experimental.topics.batch_transforms.drop_row_number_column(
    batch: pyarrow.RecordBatch,
) -> pyarrow.RecordBatch
```

`from roboto.experimental.topics.batch_transforms import drop_row_number_column`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L84-L91)

Return `batch` without its row number column, found by its metadata marker.

**Parameters**

- **batch** (`pyarrow.RecordBatch`)

**Raises**

- [`RobotoInternalException`](/reference/python-sdk/roboto/exceptions/domain#roboto.exceptions.domain.RobotoInternalException): The batch does not contain exactly one marked row number column.

**Returns**

- `pyarrow.RecordBatch`

### flatten_table()

```python
def roboto.experimental.topics.batch_transforms.flatten_table(
    table: pyarrow.Table,
) -> pyarrow.Table
```

`from roboto.experimental.topics.batch_transforms import flatten_table`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L137-L174)

Expand struct columns into dot-delimited leaf columns, recursively.

A null at any struct level propagates to nulls in every leaf column beneath it. List-typed columns stay whole. This is the DataFrame packing shape: dotted leaf columns over the projected tree.

**Parameters**

- **table** (`pyarrow.Table`)

**Raises**

- [`RobotoInvalidRequestException`](/reference/python-sdk/roboto/exceptions/domain#roboto.exceptions.domain.RobotoInvalidRequestException): Two columns resolve to the same dotted name — e.g. a top-level field literally named `pose.x` alongside a struct `pose` with child `x`. A plain dict would silently drop one (last write wins); the ambiguity is rejected instead. Rename the offending field or disable `flatten=True` to recover the column.

**Returns**

- `pyarrow.Table`

### row_number_column_index()

```python
def roboto.experimental.topics.batch_transforms.row_number_column_index(
    schema: pyarrow.Schema,
) -> int
```

`from roboto.experimental.topics.batch_transforms import row_number_column_index`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L74-L81)

Locate the row number column: the field whose [`ROW_NUMBER_FIELD_METADATA_KEY`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.ROW_NUMBER_FIELD_METADATA_KEY) metadata is `b"true"`.

**Parameters**

- **schema** (`pyarrow.Schema`)

**Raises**

- [`RobotoInternalException`](/reference/python-sdk/roboto/exceptions/domain#roboto.exceptions.domain.RobotoInternalException): The schema does not contain exactly one marked column.

**Returns**

- `int`

### timestamp_column_index()

```python
def roboto.experimental.topics.batch_transforms.timestamp_column_index(
    schema: pyarrow.Schema,
) -> int
```

`from roboto.experimental.topics import timestamp_column_index`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L123-L134)

Locate the timestamp column: the field whose [`TIMESTAMP_FIELD_METADATA_KEY`](/reference/python-sdk/roboto/experimental/topics/batch_transforms#roboto.experimental.topics.batch_transforms.TIMESTAMP_FIELD_METADATA_KEY) metadata is `b"true"`.

The column is identified by metadata, never by name: a projected root field can legitimately carry any name, including the timestamp column's conventional one.

**Parameters**

- **schema** (`pyarrow.Schema`)

**Raises**

- [`RobotoInternalException`](/reference/python-sdk/roboto/exceptions/domain#roboto.exceptions.domain.RobotoInternalException): The schema does not contain exactly one marked column.

**Returns**

- `int`

### timestamp_field()

```python
def roboto.experimental.topics.batch_transforms.timestamp_field(
    name: str = TIMESTAMP_FIELD_NAME,
) -> pyarrow.Field
```

`from roboto.experimental.topics.batch_transforms import timestamp_field`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L117-L120)

The timestamp column's Arrow field: int64 epoch nanoseconds, metadata-marked.

**Parameters**

- **name** (`str`)

**Returns**

- `pyarrow.Field`

### topic_data_schema()

```python
def roboto.experimental.topics.batch_transforms.topic_data_schema(
    value_fields: collections.abc.Sequence[pyarrow.Field],
) -> pyarrow.Schema
```

`from roboto.experimental.topics.batch_transforms import topic_data_schema`

[Source](https://github.com/roboto-ai/roboto-python-sdk/blob/main/src/roboto/experimental/topics/batch_transforms.py#L94-L114)

The schema of a decoded batch whose value columns are `value_fields`: row number, timestamp, then values.

The row number column is non-null `uint64`. `_` is appended to the timestamp column's name until no value column has it, then to the row number column's name until neither a value column nor the timestamp column has it.

**Parameters**

- **value_fields** (`collections.abc.Sequence[pyarrow.Field]`)

**Returns**

- `pyarrow.Schema`