Support writing to a table with sort-order

[Fokko]

Feature Request / Improvement

We fail when we see a sort order, it would be great if we could sort+write the data based on the sort-order.

[vinjai]

Hi @Fokko
I would like to give a shot at this if no one has already taken it.

[Fokko]

@vinjai Great, I've assigned it to you!

[vinjai]

Hey @Fokko

Question around transformation defined in SortOrder:
Our input for sorting is a pyarrow table in the append and override methods. Here we have two options:

1. Sort the pyarrow table using the pyarrow transform. A lot of transforms have not been implemented or supported in pyarrow. This introduces two scenarios:
   • Breaking Change: If the transformation is not supported, we don't write ahead raising an appropriate error.
   • Silently moving ahead: Add unsorted order id and ignore the sort functionality altogether
2. Convert the pyarrow table to a python object and then sort within python.

I am more in favor of the first one.

[vinjai]

For instance (in BucketTransform):

iceberg-python/pyiceberg/transforms.py

Lines 303 to 304 in 0bf175d

	def pyarrow_transform(self, source: IcebergType) -> "Callable[[pa.Array], pa.Array]":
	raise NotImplementedError()

[Fokko]

@vinjai Since we ignore the write-order today, I think proceeding is fine. Maybe raise a warning so the user knows the data isn't being sorted. Sorting in Python would be very costly.

[Fokko]

Let's pass this to the next release when we have all the transforms implemented using the Rust extension. cc @sungwy

[vinjai]

Hey @Fokko

I'm not completely sure about the details of the Rust Extension implementation.
Could you share any documents/proposals/issues that explain what we're aiming to achieve?
I’d love to help out in any way I can to move things forward 🙂