Whether the `apply` function is implemented using the pandas_udf function? about koalas HOT 1 OPEN

RainFung commented on May 25, 2024

Whether the `apply` function is implemented using the pandas_udf function?

from koalas.

Comments (1)

itholic commented on May 25, 2024

Yeah, for some case we use mapInPandas which leveraging the pandas_udf internally.

koalas/databricks/koalas/frame.py

Lines 2557 to 2563 in 07d8462

    
           if should_use_map_in_pandas: 
        
               output_func = GroupBy._make_pandas_df_builder_func( 
        
                   self_applied, apply_func, return_schema, retain_index=True 
        
               ) 
        
               sdf = self_applied._internal.to_internal_spark_frame.mapInPandas( 
        
                   lambda iterator: map(output_func, iterator), schema=return_schema 
        
               )

from koalas.

Related Issues (20)

convert_dtypes support HOT 2
Koalas vs Pandas HOT 3
Series.to_json(orient='records') does not return records-based JSON HOT 3
Write custom metadata to output files with dataframe.to_parquet? HOT 1
read_excel's parameter - mangle_dupe_cols is used to handle duplicate columns but fails if the duplicate columns are case sensitive. HOT 2
Predicate Pushdown not Working HOT 3
AttributeError: module 'databricks.koalas' has no attribute 'DateOffset' HOT 1
missing function `koalas.series.apply` HOT 1
Joining koalas frame with spark HOT 2
fillna does not work with decimals HOT 1
pyspark is not required when install koalas
data type conversion error HOT 1
pyspark dataframe coverting to koalas dataframe have different elements HOT 5
Spammed with FutureWarnings that are unfilterable
Koalas.idxmin() is not picking the minimum value from a dataframe, but pandas.idxmin() gives HOT 1
Attribute Error: module 'numpy' has no attribute 'bool' HOT 3
Is koalas still being worked on? or is the project on pause at the moment? HOT 2
gotImport Error
Erro XVPL formula!

Whether the `apply` function is implemented using the pandas_udf function? about koalas HOT 1 OPEN

Comments (1)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	if should_use_map_in_pandas:
	output_func = GroupBy._make_pandas_df_builder_func(
	self_applied, apply_func, return_schema, retain_index=True
	)
	sdf = self_applied._internal.to_internal_spark_frame.mapInPandas(
	lambda iterator: map(output_func, iterator), schema=return_schema
	)