POC: New UDF methods option #43678
POC: New UDF methods option #43678
Conversation
| {"A": [12, 12, 12], "B": [27, 27, 27]}, index=["sum", "<lambda>", "<lambda>"] | ||
| ) | ||
|
|
||
| result = pdf.apply([np.square, lambda x: x, lambda x: x]) |
There was a problem hiding this comment.
Changed from sum to square because lambda x: x.sum() would fail for apply; this is now consistent with
pdf.apply(lambda x: x.sum())
which currently fails master.
| } | ||
| ) | ||
| if get_option("new_udf_methods"): | ||
| result = string_series.agg( |
There was a problem hiding this comment.
Changed from apply to agg as now apply with a dictionary of ops will act the same way as apply with a single op - acting on each row of the Series individually.
| if how == "apply": | ||
| expected = DataFrame({name: string_series for name, op in zip(names, ops)}) | ||
| else: | ||
| expected = Series( | ||
| {name: op(string_series) for name, op in zip(names, ops)}, name="series" | ||
| ) |
There was a problem hiding this comment.
Series.apply with a list of ops now acts in the same way as Series.apply with a single op, applying the UDF to each row of the Series. This is a no-op for sum/mean.
| if get_option("new_udf_methods"): | ||
| expected = Series([1, 1, 1, 3], index=index) | ||
| expected.index.name = None | ||
| tm.assert_series_equal(table, expected) | ||
| else: | ||
| expected = DataFrame(index=index) | ||
| tm.assert_frame_equal(table, expected) |
There was a problem hiding this comment.
See the bottom of the PDF attached in the OP for some more details on this.
rhshadrach
force-pushed
the
udfs_poc
branch
from
6bd8488 to
4acdb55
Compare
| @@ -98,6 +102,7 @@ def frame_apply( | |||
| result_type=result_type, | |||
| args=args, | |||
| kwargs=kwargs, | |||
| renamer=renamer, | |||
| for a in arg: | ||
| name = None | ||
| try: | ||
| if isinstance(a, (tuple, list)): |
There was a problem hiding this comment.
prob worth having a function to compute a single arg (and then just call in the list)
| @@ -408,6 +421,70 @@ def agg_list_like(self) -> DataFrame | Series: | |||
| ) | |||
| return concatenated.reindex(full_ordered_index, copy=False) | |||
|
|
|||
| def new_list_like(self, method: str) -> DataFrame | Series: | |||
There was a problem hiding this comment.
can this be something other than new? i.e. a name that will still be helpful/accurate in 12 months
There was a problem hiding this comment.
Agreed - splitting this up, some discussion in the first part: #43736. "future" has been suggested as a replacement for "new".
|
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this. |
Start of #41112
The goal is to provide an implementation for apply on lists. See #41112 (comment) for why this necessitates a change to agg behavior for lists. It is possible to break this up into several PRs, but wanted to give a view as to what it would appear in totality.
Attached is a notebook demonstrating the differences in behavior (github doesn't allow attaching the notebook directly).
udfs.pdf