Packages

  • package root
    Definition Classes
    root
  • package ai
    Definition Classes
    root
  • package catboost
    Definition Classes
    ai
  • package spark

    CatBoost is a machine learning algorithm that uses gradient boosting on decision trees.

    CatBoost is a machine learning algorithm that uses gradient boosting on decision trees.

    Overview

    This package provides classes that implement interfaces from Apache Spark Machine Learning Library (MLLib).

    For binary and multi- classification problems use CatBoostClassifier, for regression use CatBoostRegressor.

    These classes implement usual fit method of org.apache.spark.ml.Predictor that accept a single org.apache.spark.sql.DataFrame for training, but you can also use other fit method that accepts additional datasets for computing evaluation metrics and overfitting detection similarily to CatBoost's other APIs.

    This package also contains Pool class that is CatBoost's abstraction of a dataset. It contains additional information compared to simple org.apache.spark.sql.DataFrame.

    It is also possible to create Pool with quantized features before training by calling quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

    Detailed documentation is available on https://catboost.ai/docs/

    Definition Classes
    catboost
  • package params
    Definition Classes
    spark
  • CatBoostClassificationModel
  • CatBoostClassifier
  • CatBoostPredictorTrait
  • CatBoostRegressionModel
  • CatBoostRegressor
  • Pool

class CatBoostClassificationModel extends ProbabilisticClassificationModel[Vector, CatBoostClassificationModel] with CatBoostModelTrait[CatBoostClassificationModel]

Classification model trained by CatBoost. Use CatBoostClassifier to train it

Serialization

Supports standard Spark MLLib serialization. Data can be saved to distributed filesystem like HDFS or local files. When saved to path two files are created: -<path>/metadata which contains Spark-specific metadata in JSON format -<path>/model which contains model in usual CatBoost format which can be read using other local CatBoost APIs (if stored in a distributed filesystem it has to be copied to the local filesystem first).

Examples:

  1. Save model

    val trainPool : Pool = ... init Pool ...
val classifier = new CatBoostClassifier
val model = classifier.fit(trainPool)
val path = "/home/user/catboost_spark_models/model0"
model.write.save(path)
    2. ,

  2. Load model

    val dataFrameForPrediction : DataFrame = ... init DataFrame ...
val path = "/home/user/catboost_spark_models/model0"
val model = CatBoostClassificationModel.load(path)
val predictions = model.transform(dataFrameForPrediction)
predictions.show()
Linear Supertypes
CatBoostModelTrait[CatBoostClassificationModel], MLWritable, ProbabilisticClassificationModel[Vector, CatBoostClassificationModel], ProbabilisticClassifierParams, HasThresholds, HasProbabilityCol, ClassificationModel[Vector, CatBoostClassificationModel], ClassifierParams, HasRawPredictionCol, PredictionModel[Vector, CatBoostClassificationModel], PredictorParams, HasPredictionCol, HasFeaturesCol, HasLabelCol, Model[CatBoostClassificationModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CatBoostClassificationModel
  2. CatBoostModelTrait
  3. MLWritable
  4. ProbabilisticClassificationModel
  5. ProbabilisticClassifierParams
  6. HasThresholds
  7. HasProbabilityCol
  8. ClassificationModel
  9. ClassifierParams
  10. HasRawPredictionCol
  11. PredictionModel
  12. PredictorParams
  13. HasPredictionCol
  14. HasFeaturesCol
  15. HasLabelCol
  16. Model
  17. Transformer
  18. PipelineStage
  19. Logging
  20. Params
  21. Serializable
  22. Serializable
  23. Identifiable
  24. AnyRef
  25. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CatBoostClassificationModel(nativeModel: TFullModel)
  2. new CatBoostClassificationModel(uid: String, nativeModel: TFullModel = null, nativeDimension: Int)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. final def clear(param: Param[_]): CatBoostClassificationModel.this.type
    Definition Classes
    Params
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. def copy(extra: ParamMap): CatBoostClassificationModel
    Definition Classes
    CatBoostClassificationModel → Model → Transformer → PipelineStage → Params
  9. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  10. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  14. def explainParams(): String
    Definition Classes
    Params
  15. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  16. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  17. final val featuresCol: Param[String]
    Definition Classes
    HasFeaturesCol
  18. def featuresDataType: DataType
    Attributes
    protected
    Definition Classes
    PredictionModel
  19. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  20. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  21. def getAdditionalColumnsForApply: Seq[StructField]
    Attributes
    protected
    Definition Classes
    CatBoostClassificationModel → CatBoostModelTrait
  22. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  23. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  24. final def getFeaturesCol: String
    Definition Classes
    HasFeaturesCol
  25. final def getLabelCol: String
    Definition Classes
    HasLabelCol
  26. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  27. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  28. final def getPredictionCol: String
    Definition Classes
    HasPredictionCol
  29. final def getProbabilityCol: String
    Definition Classes
    HasProbabilityCol
  30. final def getRawPredictionCol: String
    Definition Classes
    HasRawPredictionCol
  31. def getResultIteratorForApply(rawObjectsDataProvider: SWIGTYPE_p_NCB__TRawObjectsDataProviderPtr, dstRows: ArrayBuffer[Array[Any]], threadCountForTask: Int): Iterator[Row]
    Attributes
    protected
    Definition Classes
    CatBoostClassificationModel → CatBoostModelTrait
  32. def getThresholds: Array[Double]
    Definition Classes
    HasThresholds
  33. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  34. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  35. def hasParent: Boolean
    Definition Classes
    Model
  36. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  37. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  38. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  40. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  41. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  42. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  43. final val labelCol: Param[String]
    Definition Classes
    HasLabelCol
  44. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  45. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  52. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. var nativeDimension: Int
    Attributes
    protected
    Definition Classes
    CatBoostClassificationModel → CatBoostModelTrait
  57. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  58. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  59. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  60. def numClasses: Int
    Definition Classes
    CatBoostClassificationModel → ClassificationModel
  61. def numFeatures: Int
    Definition Classes
    PredictionModel
    Annotations
    @Since( "1.6.0" )
  62. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  63. var parent: Estimator[CatBoostClassificationModel]
    Definition Classes
    Model
  64. def predict(features: Vector): Double
    Definition Classes
    ClassificationModel → PredictionModel
  65. def predictProbability(features: Vector): Vector
    Attributes
    protected
    Definition Classes
    ProbabilisticClassificationModel
  66. def predictRaw(features: Vector): Vector

    Prefer batch computations operating on datasets as a whole for efficiency

    Prefer batch computations operating on datasets as a whole for efficiency

    Attributes
    protected
    Definition Classes
    CatBoostClassificationModel → ClassificationModel
  67. final def predictRawImpl(features: Vector): Array[Double]

    Prefer batch computations operating on datasets as a whole for efficiency

    Prefer batch computations operating on datasets as a whole for efficiency

    Definition Classes
    CatBoostModelTrait
  68. final val predictionCol: Param[String]
    Definition Classes
    HasPredictionCol
  69. def probability2prediction(probability: Vector): Double
    Attributes
    protected
    Definition Classes
    ProbabilisticClassificationModel
  70. final val probabilityCol: Param[String]
    Definition Classes
    HasProbabilityCol
  71. def raw2prediction(rawPrediction: Vector): Double
    Attributes
    protected
    Definition Classes
    ProbabilisticClassificationModel → ClassificationModel
  72. def raw2probability(rawPrediction: Vector): Vector
    Attributes
    protected
    Definition Classes
    ProbabilisticClassificationModel
  73. def raw2probabilityInPlace(rawPrediction: Vector): Vector

    Prefer batch computations operating on datasets as a whole for efficiency

    Prefer batch computations operating on datasets as a whole for efficiency

    Attributes
    protected
    Definition Classes
    CatBoostClassificationModel → ProbabilisticClassificationModel
  74. final val rawPredictionCol: Param[String]
    Definition Classes
    HasRawPredictionCol
  75. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  76. final def set(paramPair: ParamPair[_]): CatBoostClassificationModel.this.type
    Attributes
    protected
    Definition Classes
    Params
  77. final def set(param: String, value: Any): CatBoostClassificationModel.this.type
    Attributes
    protected
    Definition Classes
    Params
  78. final def set[T](param: Param[T], value: T): CatBoostClassificationModel.this.type
    Definition Classes
    Params
  79. final def setDefault(paramPairs: ParamPair[_]*): CatBoostClassificationModel.this.type
    Attributes
    protected
    Definition Classes
    Params
  80. final def setDefault[T](param: Param[T], value: T): CatBoostClassificationModel.this.type
    Attributes
    protected
    Definition Classes
    Params
  81. def setFeaturesCol(value: String): CatBoostClassificationModel
    Definition Classes
    PredictionModel
  82. def setParent(parent: Estimator[CatBoostClassificationModel]): CatBoostClassificationModel
    Definition Classes
    Model
  83. def setPredictionCol(value: String): CatBoostClassificationModel
    Definition Classes
    PredictionModel
  84. def setProbabilityCol(value: String): CatBoostClassificationModel
    Definition Classes
    ProbabilisticClassificationModel
  85. def setRawPredictionCol(value: String): CatBoostClassificationModel
    Definition Classes
    ClassificationModel
  86. def setThresholds(value: Array[Double]): CatBoostClassificationModel
    Definition Classes
    ProbabilisticClassificationModel
  87. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  88. final val thresholds: DoubleArrayParam
    Definition Classes
    HasThresholds
  89. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  90. def transform(dataset: Dataset[_]): DataFrame
    Definition Classes
    CatBoostClassificationModel → ProbabilisticClassificationModel → ClassificationModel → PredictionModel → Transformer
  91. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  92. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  93. def transformImpl(dataset: Dataset[_]): DataFrame
    Definition Classes
    CatBoostModelTrait → PredictionModel
  94. def transformSchema(schema: StructType): StructType
    Definition Classes
    PredictionModel → PipelineStage
  95. def transformSchema(schema: StructType, logging: Boolean): StructType
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  96. val uid: String
    Definition Classes
    CatBoostClassificationModel → Identifiable
  97. def validateAndTransformSchema(schema: StructType, fitting: Boolean, featuresDataType: DataType): StructType
    Attributes
    protected
    Definition Classes
    ProbabilisticClassifierParams → ClassifierParams → PredictorParams
  98. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  99. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  100. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  101. def write: MLWriter
    Definition Classes
    CatBoostModelTrait → MLWritable

Inherited from CatBoostModelTrait[CatBoostClassificationModel]

Inherited from MLWritable

Inherited from ProbabilisticClassificationModel[Vector, CatBoostClassificationModel]

Inherited from ProbabilisticClassifierParams

Inherited from HasThresholds

Inherited from HasProbabilityCol

Inherited from ClassificationModel[Vector, CatBoostClassificationModel]

Inherited from ClassifierParams

Inherited from HasRawPredictionCol

Inherited from PredictionModel[Vector, CatBoostClassificationModel]

Inherited from PredictorParams

Inherited from HasPredictionCol

Inherited from HasFeaturesCol

Inherited from HasLabelCol

Inherited from Model[CatBoostClassificationModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Ungrouped