Packages

  • package root
    Definition Classes
    root
  • package ai
    Definition Classes
    root
  • package catboost
    Definition Classes
    ai
  • package spark

    CatBoost is a machine learning algorithm that uses gradient boosting on decision trees.

    CatBoost is a machine learning algorithm that uses gradient boosting on decision trees.

    Overview

    This package provides classes that implement interfaces from Apache Spark Machine Learning Library (MLLib).

    For binary and multi- classification problems use CatBoostClassifier, for regression use CatBoostRegressor.

    These classes implement usual fit method of org.apache.spark.ml.Predictor that accept a single org.apache.spark.sql.DataFrame for training, but you can also use other fit method that accepts additional datasets for computing evaluation metrics and overfitting detection similarily to CatBoost's other APIs.

    This package also contains Pool class that is CatBoost's abstraction of a dataset. It contains additional information compared to simple org.apache.spark.sql.DataFrame.

    It is also possible to create Pool with quantized features before training by calling quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

    Detailed documentation is available on https://catboost.ai/docs/

    Definition Classes
    catboost
  • package params
    Definition Classes
    spark
  • CatBoostClassificationModel
  • CatBoostClassifier
  • CatBoostPredictorTrait
  • CatBoostRegressionModel
  • CatBoostRegressor
  • Pool

class Pool extends Params with HasLabelCol with HasFeaturesCol with HasWeightCol

CatBoost's abstraction of a dataset.

Features data can be stored in raw (features column has org.apache.spark.ml.linalg.Vector type) or quantized (float feature values are quantized into integer bin values, features column has Array[Byte] type) form.

Raw Pool can be transformed to quantized form using quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

Linear Supertypes
HasWeightCol, HasFeaturesCol, HasLabelCol, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Pool
  2. HasWeightCol
  3. HasFeaturesCol
  4. HasLabelCol
  5. Params
  6. Serializable
  7. Serializable
  8. Identifiable
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Pool(data: DataFrame)

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns.

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns. Only features and label columns with "features" and "label" names are assumed by default.

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f)
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")

pool.data.show()
  2. new Pool(uid: String, data: DataFrame = null, featuresLayout: TFeaturesLayout = null, quantizedFeaturesInfo: QuantizedFeaturesInfoPtr = null)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. final val baselineCol: Param[String]
  7. final def clear(param: Param[_]): Pool.this.type
    Definition Classes
    Params
  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  9. def copy(extra: ParamMap): Pool
    Definition Classes
    Pool → Params
  10. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  11. def count: Long

    returns

    Number of objects in the dataset, similar to the same method of org.apache.spark.sql.Dataset

  12. def createQuantizationSchema(quantizationParams: QuantizationParamsTrait): QuantizedFeaturesInfoPtr
    Attributes
    protected
  13. def createQuantized(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool
    Attributes
    protected
  14. val data: DataFrame
  15. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  16. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  18. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  19. def explainParams(): String
    Definition Classes
    Params
  20. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  21. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  22. final val featuresCol: Param[String]
    Definition Classes
    HasFeaturesCol
  23. var featuresLayout: TFeaturesLayout
    Attributes
    protected
  24. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  25. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  26. final def getBaselineCol: String
  27. def getBaselineCount: Int

    returns

    dimension of formula baseline, 0 if no baseline specified

  28. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  29. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  30. def getFeatureCount: Int
  31. def getFeatureNames: Array[String]
  32. final def getFeaturesCol: String
    Definition Classes
    HasFeaturesCol
  33. def getFeaturesLayout: TFeaturesLayout
  34. final def getGroupIdCol: String
  35. final def getGroupWeightCol: String
  36. final def getLabelCol: String
    Definition Classes
    HasLabelCol
  37. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  38. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  39. final def getSampleIdCol: String
  40. final def getSubgroupIdCol: String
  41. def getTargetType: ERawTargetType
  42. final def getTimestampCol: String
  43. final def getWeightCol: String
    Definition Classes
    HasWeightCol
  44. final val groupIdCol: Param[String]
  45. final val groupWeightCol: Param[String]
  46. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  47. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  48. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  49. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  50. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  51. def isQuantized: Boolean
  52. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  53. final val labelCol: Param[String]
    Definition Classes
    HasLabelCol
  54. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  55. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  56. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  57. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  58. def quantize(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool

    Create Pool with quantized features from Pool with raw features.

    Create Pool with quantized features from Pool with raw features. This variant of the method is useful if QuantizedFeaturesInfo with data for quantization (borders and nan modes) has already been computed. Used, for example, to quantize evaluation datasets after the training dataset has been quantized.

  59. def quantize(quantizationParams: QuantizationParamsTrait): Pool

    Create Pool with quantized features from Pool with raw features

    Create Pool with quantized features from Pool with raw features

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[*]")
  .appName("QuantizationTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12"),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22"),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34")
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)

val quantizedPool = pool.quantize(new QuantizationParams)
val quantizedPoolWithTwoBinsPerFeature = pool.quantize(new QuantizationParams().setBorderCount(1))
quantizedPool.data.show()
quantizedPoolWithTwoBinsPerFeature.data.show()
  60. val quantizedFeaturesInfo: QuantizedFeaturesInfoPtr
  61. def repartition(partitionCount: Int, byGroupColumnsIfPresent: Boolean = true): Pool

    Repartion data to the specified number of partitions.

    Repartion data to the specified number of partitions. Useful to repartition data to create one partition per executor for training (where each executor gets its' own CatBoost worker with a part of the training data).

  62. final val sampleIdCol: Param[String]
  63. final def set(paramPair: ParamPair[_]): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  64. final def set(param: String, value: Any): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  65. final def set[T](param: Param[T], value: T): Pool.this.type
    Definition Classes
    Params
  66. final def setBaselineCol(value: String): Pool.this.type
  67. final def setDefault(paramPairs: ParamPair[_]*): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  68. final def setDefault[T](param: Param[T], value: T): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  69. def setFeaturesCol(value: String): Pool

  70. final def setGroupIdCol(value: String): Pool.this.type
  71. final def setGroupWeightCol(value: String): Pool.this.type
  72. def setLabelCol(value: String): Pool

  73. final def setSampleIdCol(value: String): Pool.this.type
  74. final def setSubgroupIdCol(value: String): Pool.this.type
  75. final def setTimestampCol(value: String): Pool.this.type
  76. def setWeightCol(value: String): Pool

  77. final val subgroupIdCol: Param[String]
  78. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  79. final val timestampCol: Param[String]
  80. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  81. val uid: String
    Definition Classes
    Pool → Identifiable
  82. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  83. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  84. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  85. final val weightCol: Param[String]
    Definition Classes
    HasWeightCol

Inherited from HasWeightCol

Inherited from HasFeaturesCol

Inherited from HasLabelCol

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

setParam

Ungrouped