• Home
• com.johnsnowlabs.nlp
• spark-nlp-m1_2.12
• 4.2.8
• source code
• SparkUtil.scala

×

Project download

Many resources are needed to download a project. Please understand that we have to compensate our server costs. Thank you in advance.
Project price only 1 $

You can buy this project and download/modify it how often you want.

com.johnsnowlabs.util.spark.SparkUtil.scala Maven / Gradle / Ivy

package com.johnsnowlabs.util.spark

import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.functions.udf

object SparkUtil {

  // Helper UDF function to flatten arrays for Spark < 2.4.0
  def flattenArrays: UserDefinedFunction = udf { (arrayColumn: Seq[Seq[String]]) =>
    arrayColumn.flatten.distinct
  }

}