I have a trained model, and test input from mnist
import mlflow
from pyspark.sql.functions import struct, col
logged_model = 'runs:/myid/myModel'
loaded_model = mlflow.pyfunc.spark_udf(spark, model_uri=logged_model,
result_type='double')
(x_train, y_train), (x_test, y_test) = mnist.load_data()
However I am having trouble converting ndarray from mnist to spark dataframe so that I can do prediction for my model
mydf = spark.sparkContext.parallelize(x_test).map(lambda x:
[x.tolist()]).toDF(["Doc2Vec"])
output = mydf.withColumn('predictions', loaded_model(struct(*map(col, mydf.columns))))
when I try to output the result
output.show(10)
I got
An exception was thrown from a UDF: 'ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type numpy.ndarray).'.
What am i missing here?
