Package

org.apache.predictionio

controller

Permalink

package controller

Provides building blocks for writing a complete prediction engine consisting of DataSource, Preparator, Algorithm, Serving, and Evaluation.

Start Building an Engine

The starting point of a prediction engine is the Engine class.

The DASE Paradigm

The building blocks together form the DASE paradigm. Learn more about DASE here.

Types of Building Blocks

Depending on the problem you are solving, you would need to pick appropriate flavors of building blocks.

Engines

There are 3 typical engine configurations:

  1. PDataSource, PPreparator, P2LAlgorithm, LServing 2. PDataSource, PPreparator, PAlgorithm, LServing 3. LDataSource, LPreparator, LAlgorithm, LServing

In both configurations 1 and 2, data is sourced and prepared in a parallelized fashion, with data type as RDD.

The difference between configurations 1 and 2 come at the algorithm stage. In configuration 1, the algorithm operates on potentially large data as RDDs in the Spark cluster, and eventually outputs a model that is small enough to fit in a single machine.

On the other hand, configuration 2 outputs a model that is potentially too large to fit in a single machine, and must reside in the Spark cluster as RDD(s).

With configuration 1 (P2LAlgorithm), PredictionIO will automatically try to persist the model to local disk or HDFS if the model is serializable.

With configuration 2 (PAlgorithm), PredictionIO will not automatically try to persist the model, unless the model implements the PersistentModel trait.

In special circumstances where both the data and the model are small, configuration 3 may be used. Beware that RDDs cannot be used with configuration 3.

Data Source

PDataSource is probably the most used data source base class with the ability to process RDD-based data. LDataSource cannot handle RDD-based data. Use only when you have a special requirement.

Preparator

With PDataSource, you must pick PPreparator. The same applies to LDataSource and LPreparator.

Algorithm

The workhorse of the engine comes in 3 different flavors.

P2LAlgorithm

Produces a model that is small enough to fit in a single machine from PDataSource and PPreparator. The model cannot contain any RDD. If the produced model is serializable, PredictionIO will try to automatically persist it. In addition, P2LAlgorithm.batchPredict is already implemented for Evaluation purpose.

PAlgorithm

Produces a model that could contain RDDs from PDataSource and PPreparator. PredictionIO will not try to persist it automatically unless the model implements PersistentModel. PAlgorithm.batchPredict must be implemented for Evaluation.

LAlgorithm

Produces a model that is small enough to fit in a single machine from LDataSource and LPreparator. The model cannot contain any RDD. If the produced model is serializable, PredictionIO will try to automatically persist it. In addition, LAlgorithm.batchPredict is already implemented for Evaluation purpose.

Serving

The serving component comes with only 1 flavor--LServing. At the serving stage, it is assumed that the result being served is already at a human- consumable size.

Model Persistence

PredictionIO tries its best to persist trained models automatically. Please refer to LAlgorithm.makePersistentModel, P2LAlgorithm.makePersistentModel, and PAlgorithm.makePersistentModel for descriptions on different strategies.

Linear Supertypes
AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. controller
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. abstract class AverageMetric[EI, Q, P, A] extends Metric[EI, Q, P, A, Double] with StatsMetricHelper[EI, Q, P, A] with QPAMetric[Q, P, A, Double]

    Permalink

    Returns the global average of the score returned by the calculate method.

    Returns the global average of the score returned by the calculate method.

    EI

    Evaluation information

    Q

    Query

    P

    Predicted result

    A

    Actual result

  2. trait CustomQuerySerializer extends BaseQuerySerializer

    Permalink

    If your query class cannot be automatically serialized/deserialized to/from JSON, implement a trait by extending this trait, and overriding the querySerializer member with your custom JSON4S serializer.

    If your query class cannot be automatically serialized/deserialized to/from JSON, implement a trait by extending this trait, and overriding the querySerializer member with your custom JSON4S serializer. Algorithm and serving classes using your query class would only need to mix in the trait to enable the custom serializer.

  3. trait Deployment extends EngineFactory

    Permalink

    Defines a deployment that contains an Engine

  4. type EmptyActualResult = SerializableClass

    Permalink

    Empty actual result.

  5. type EmptyAlgorithmParams = EmptyParams

    Permalink

    Empty algorithm parameters.

  6. type EmptyDataParams = EmptyParams

    Permalink

    Empty data parameters.

  7. type EmptyDataSourceParams = EmptyParams

    Permalink

    Empty data source parameters.

  8. type EmptyEvaluationInfo = SerializableClass

    Permalink

    Empty evaluation info.

  9. type EmptyMetricsParams = EmptyParams

    Permalink

    Empty metrics parameters.

  10. type EmptyModel = SerializableClass

    Permalink

    Empty model.

  11. case class EmptyParams() extends Params with Product with Serializable

    Permalink

    A concrete implementation of Params representing empty parameters.

  12. type EmptyPreparatorParams = EmptyParams

    Permalink

    Empty preparator parameters.

  13. type EmptyPreparedData = SerializableClass

    Permalink

    Empty prepared data.

  14. type EmptyServingParams = EmptyParams

    Permalink

    Empty serving parameters.

  15. type EmptyTrainingData = SerializableClass

    Permalink

    Empty training data.

  16. class Engine[TD, EI, PD, Q, P, A] extends BaseEngine[EI, Q, P, A]

    Permalink

    This class chains up the entire data process.

    This class chains up the entire data process. PredictionIO uses this information to create workflows and deployments. In Scala, you should implement an object that extends the EngineFactory trait similar to the following example.

    object ItemRankEngine extends EngineFactory {
      def apply() = {
        new Engine(
          classOf[ItemRankDataSource],
          classOf[ItemRankPreparator],
          Map(
            "knn" -> classOf[KNNAlgorithm],
            "rand" -> classOf[RandomAlgorithm],
            "mahoutItemBased" -> classOf[MahoutItemBasedAlgorithm]),
          classOf[ItemRankServing])
      }
    }
    TD

    Training data class.

    EI

    Evaluation info class.

    PD

    Prepared data class.

    Q

    Input query class.

    P

    Output prediction class.

    A

    Actual value class.

    See also

    EngineFactory

  17. abstract class EngineFactory extends AnyRef

    Permalink

    If you intend to let PredictionIO create workflow and deploy serving automatically, you will need to implement an object that extends this class and return an Engine.

  18. class EngineParams extends Serializable

    Permalink

    This class serves as a logical grouping of all required engine's parameters.

  19. trait EngineParamsGenerator extends AnyRef

    Permalink

    Defines an engine parameters generator.

    Defines an engine parameters generator.

    Implementations of this trait can be supplied to "pio eval" as the second command line argument.

  20. trait Evaluation extends EngineFactory with Deployment

    Permalink

    Defines an evaluation that contains an engine and a metric.

    Defines an evaluation that contains an engine and a metric.

    Implementations of this trait can be supplied to "pio eval" as the first argument.

  21. class FastEvalEngine[TD, EI, PD, Q, P, A] extends Engine[TD, EI, PD, Q, P, A]

    Permalink

    :: Experimental :: FastEvalEngine is a subclass of Engine that exploits the immutability of controllers to optimize the evaluation process

    :: Experimental :: FastEvalEngine is a subclass of Engine that exploits the immutability of controllers to optimize the evaluation process

    Annotations
    @Experimental()
  22. class FastEvalEngineWorkflow[TD, EI, PD, Q, P, A] extends Serializable

    Permalink

    :: Experimental :: Workflow based on FastEvalEngine

    :: Experimental :: Workflow based on FastEvalEngine

    Annotations
    @Experimental()
  23. class IdentityPreparator[TD] extends BasePreparator[TD, TD]

    Permalink

    A helper concrete implementation of org.apache.predictionio.core.BasePreparator that passes training data through without any special preparation.

    A helper concrete implementation of org.apache.predictionio.core.BasePreparator that passes training data through without any special preparation. This can be used in place for both PPreparator and LPreparator.

    TD

    Training data class.

  24. abstract class LAlgorithm[PD, M, Q, P] extends BaseAlgorithm[RDD[PD], RDD[M], Q, P]

    Permalink

    Base class of a local algorithm.

    Base class of a local algorithm.

    A local algorithm runs locally within a single machine and produces a model that can fit within a single machine.

    If your input query class requires custom JSON4S serialization, the most idiomatic way is to implement a trait that extends CustomQuerySerializer, and mix that into your algorithm class, instead of overriding querySerializer directly.

    PD

    Prepared data class.

    M

    Trained model class.

    Q

    Input query class.

    P

    Output prediction class.

  25. class LAverageServing[Q] extends LServing[Q, Double]

    Permalink

    A concrete implementation of LServing returning the average of all algorithms' predictions, where their classes are expected to be all Double.

  26. abstract class LDataSource[TD, EI, Q, A] extends BaseDataSource[RDD[TD], EI, Q, A]

    Permalink

    Base class of a local data source.

    Base class of a local data source.

    A local data source runs locally within a single machine and return data that can fit within a single machine.

    TD

    Training data class.

    EI

    Evaluation Info class.

    Q

    Input query class.

    A

    Actual value class.

  27. class LFirstServing[Q, P] extends LServing[Q, P]

    Permalink

    A concrete implementation of LServing returning the first algorithm's prediction result directly without any modification.

  28. class LIdentityPreparator[TD] extends IdentityPreparator[TD]

    Permalink

    DEPRECATED.

    DEPRECATED. Use IdentityPreparator instead.

    TD

    Training data class.

  29. abstract class LPreparator[TD, PD] extends BasePreparator[RDD[TD], RDD[PD]]

    Permalink

    Base class of a local preparator.

    Base class of a local preparator.

    A local preparator runs locally within a single machine and produces prepared data that can fit within a single machine.

    TD

    Training data class.

    PD

    Prepared data class.

  30. abstract class LServing[Q, P] extends BaseServing[Q, P]

    Permalink

    Base class of serving.

    Base class of serving.

    Q

    Input query class.

    P

    Output prediction class.

  31. trait LocalFileSystemPersistentModel[AP <: Params] extends PersistentModel[AP]

    Permalink

    This trait is a convenience helper for persisting your model to the local filesystem.

    This trait is a convenience helper for persisting your model to the local filesystem. This trait and LocalFileSystemPersistentModelLoader contain concrete implementation and need not be implemented.

    The underlying implementation is Utils.save.

    class MyModel extends LocalFileSystemPersistentModel[MyParams] {
      ...
    }
    
    object MyModel extends LocalFileSystemPersistentModelLoader[MyParams, MyModel] {
      ...
    }
    AP

    Algorithm parameters class.

    See also

    LocalFileSystemPersistentModelLoader

  32. trait LocalFileSystemPersistentModelLoader[AP <: Params, M] extends PersistentModelLoader[AP, M]

    Permalink

    Implement an object that extends this trait for PredictionIO to support loading a persisted model from local filesystem during serving deployment.

    Implement an object that extends this trait for PredictionIO to support loading a persisted model from local filesystem during serving deployment.

    The underlying implementation is Utils.load.

    AP

    Algorithm parameters class.

    M

    Model class.

    See also

    LocalFileSystemPersistentModel

  33. abstract class Metric[EI, Q, P, A, R] extends Serializable

    Permalink

    Base class of a Metric.

    Base class of a Metric.

    EI

    Evaluation information

    Q

    Query

    P

    Predicted result

    A

    Actual result

    R

    Metric result

  34. class MetricEvaluator[EI, Q, P, A, R] extends BaseEvaluator[EI, Q, P, A, MetricEvaluatorResult[R]]

    Permalink

    :: DeveloperApi :: Do no use this directly.

    :: DeveloperApi :: Do no use this directly. Use MetricEvaluator$ instead. This is an implementation of org.apache.predictionio.core.BaseEvaluator that evaluates prediction performance based on metric scores.

    EI

    Evaluation information type

    Q

    Query class

    P

    Predicted result class

    A

    Actual result class

    R

    Metric result class

    Annotations
    @DeveloperApi()
  35. case class MetricEvaluatorResult[R](bestScore: MetricScores[R], bestEngineParams: EngineParams, bestIdx: Int, metricHeader: String, otherMetricHeaders: Seq[String], engineParamsScores: Seq[(EngineParams, MetricScores[R])], outputPath: Option[String]) extends BaseEvaluatorResult with Product with Serializable

    Permalink

    Contains all results of a MetricEvaluator

    Contains all results of a MetricEvaluator

    R

    Type of the primary metric score

    bestScore

    The best score among all iterations

    bestEngineParams

    The set of engine parameters that yielded the best score

    bestIdx

    The index of iteration that yielded the best score

    metricHeader

    Brief description of the primary metric score

    otherMetricHeaders

    Brief descriptions of other metric scores

    engineParamsScores

    All sets of engine parameters and corresponding metric scores

    outputPath

    An optional output path where scores are saved

  36. case class MetricScores[R](score: R, otherScores: Seq[Any]) extends Product with Serializable

    Permalink

    Case class storing a primary score, and other scores

    Case class storing a primary score, and other scores

    R

    Type of the primary metric score

    score

    Primary metric score

    otherScores

    Other scores this metric might have

  37. abstract class OptionAverageMetric[EI, Q, P, A] extends Metric[EI, Q, P, A, Double] with StatsOptionMetricHelper[EI, Q, P, A] with QPAMetric[Q, P, A, Option[Double]]

    Permalink

    Returns the global average of the non-None score returned by the calculate method.

    Returns the global average of the non-None score returned by the calculate method.

    EI

    Evaluation information

    Q

    Query

    P

    Predicted result

    A

    Actual result

  38. abstract class OptionStdevMetric[EI, Q, P, A] extends Metric[EI, Q, P, A, Double] with StatsOptionMetricHelper[EI, Q, P, A] with QPAMetric[Q, P, A, Option[Double]]

    Permalink

    Returns the global standard deviation of the non-None score returned by the calculate method

    Returns the global standard deviation of the non-None score returned by the calculate method

    This method uses org.apache.spark.util.StatCounter library, a one pass method is used for calculation

    EI

    Evaluation information

    Q

    Query

    P

    Predicted result

    A

    Actual result

  39. abstract class P2LAlgorithm[PD, M, Q, P] extends BaseAlgorithm[PD, M, Q, P]

    Permalink

    Base class of a parallel-to-local algorithm.

    Base class of a parallel-to-local algorithm.

    A parallel-to-local algorithm can be run in parallel on a cluster and produces a model that can fit within a single machine.

    If your input query class requires custom JSON4S serialization, the most idiomatic way is to implement a trait that extends CustomQuerySerializer, and mix that into your algorithm class, instead of overriding querySerializer directly.

    PD

    Prepared data class.

    M

    Trained model class.

    Q

    Input query class.

    P

    Output prediction class.

  40. abstract class PAlgorithm[PD, M, Q, P] extends BaseAlgorithm[PD, M, Q, P]

    Permalink

    Base class of a parallel algorithm.

    Base class of a parallel algorithm.

    A parallel algorithm can be run in parallel on a cluster and produces a model that can also be distributed across a cluster.

    If your input query class requires custom JSON4S serialization, the most idiomatic way is to implement a trait that extends CustomQuerySerializer, and mix that into your algorithm class, instead of overriding querySerializer directly.

    To provide evaluation feature, one must override and implement the batchPredict method. Otherwise, an exception will be thrown when pio eval is used.

    PD

    Prepared data class.

    M

    Trained model class.

    Q

    Input query class.

    P

    Output prediction class.

  41. abstract class PDataSource[TD, EI, Q, A] extends BaseDataSource[TD, EI, Q, A]

    Permalink

    Base class of a parallel data source.

    Base class of a parallel data source.

    A parallel data source runs locally within a single machine, or in parallel on a cluster, to return data that is distributed across a cluster.

    TD

    Training data class.

    EI

    Evaluation Info class.

    Q

    Input query class.

    A

    Actual value class.

  42. class PIdentityPreparator[TD] extends IdentityPreparator[TD]

    Permalink

    DEPRECATED.

    DEPRECATED. Use IdentityPreparator instead.

    TD

    Training data class.

  43. abstract class PPreparator[TD, PD] extends BasePreparator[TD, PD]

    Permalink

    Base class of a parallel preparator.

    Base class of a parallel preparator.

    A parallel preparator can be run in parallel on a cluster and produces a prepared data that is distributed across a cluster.

    TD

    Training data class.

    PD

    Prepared data class.

  44. trait Params extends Serializable

    Permalink

    Base trait for all kinds of parameters that will be passed to constructors of different controller classes.

  45. trait PersistentModel[AP <: Params] extends AnyRef

    Permalink

    Mix in and implement this trait if your model cannot be persisted by PredictionIO automatically.

    Mix in and implement this trait if your model cannot be persisted by PredictionIO automatically. A companion object extending IPersistentModelLoader is required for PredictionIO to load the persisted model automatically during deployment.

    Notice that models generated by PAlgorithm cannot be persisted automatically by nature and must implement these traits if model persistence is desired.

    class MyModel extends PersistentModel[MyParams] {
      def save(id: String, params: MyParams, sc: SparkContext): Boolean = {
        ...
      }
    }
    
    object MyModel extends PersistentModelLoader[MyParams, MyModel] {
      def apply(id: String, params: MyParams, sc: Option[SparkContext]): MyModel = {
        ...
      }
    }

    In Java, all you need to do is to implement this interface, and add a static method with 3 arguments of type String, Params, and SparkContext.

    public class MyModel implements PersistentModel<MyParams>, Serializable {
      ...
      public boolean save(String id, MyParams params, SparkContext sc) {
        ...
      }
    
      public static MyModel load(String id, Params params, SparkContext sc) {
        ...
      }
      ...
    }
    AP

    Algorithm parameters class.

    See also

    PersistentModelLoader

  46. trait PersistentModelLoader[AP <: Params, M] extends AnyRef

    Permalink

    Implement an object that extends this trait for PredictionIO to support loading a persisted model during serving deployment.

    Implement an object that extends this trait for PredictionIO to support loading a persisted model during serving deployment.

    AP

    Algorithm parameters class.

    M

    Model class.

    See also

    PersistentModel

  47. trait QPAMetric[Q, P, A, R] extends AnyRef

    Permalink

    Trait for metric which returns a score based on Query, PredictedResult, and ActualResult

    Trait for metric which returns a score based on Query, PredictedResult, and ActualResult

    Q

    Query class

    P

    Predicted result class

    A

    Actual result class

    R

    Metric result class

  48. trait SanityCheck extends AnyRef

    Permalink

    Extends a data class with this trait if you want PredictionIO to automatically perform sanity check on your data classes during training.

    Extends a data class with this trait if you want PredictionIO to automatically perform sanity check on your data classes during training. This is very useful when you need to debug your engine.

  49. class SerializableClass extends Serializable

    Permalink

    Base class of several helper types that represent emptiness

  50. class SimpleEngine[TD, EI, Q, P, A] extends Engine[TD, EI, TD, Q, P, A]

    Permalink

    SimpleEngine has only one algorithm, and uses default preparator and serving layer.

    SimpleEngine has only one algorithm, and uses default preparator and serving layer. Current default preparator is IdentityPreparator and serving is FirstServing.

    TD

    Training data class.

    EI

    Evaluation info class.

    Q

    Input query class.

    P

    Output prediction class.

    A

    Actual value class.

  51. class SimpleEngineParams extends EngineParams

    Permalink

    This shorthand class serves the SimpleEngine class.

  52. abstract class StdevMetric[EI, Q, P, A] extends Metric[EI, Q, P, A, Double] with StatsMetricHelper[EI, Q, P, A] with QPAMetric[Q, P, A, Double]

    Permalink

    Returns the global standard deviation of the score returned by the calculate method

    Returns the global standard deviation of the score returned by the calculate method

    This method uses org.apache.spark.util.StatCounter library, a one pass method is used for calculation

    EI

    Evaluation information

    Q

    Query

    P

    Predicted result

    A

    Actual result

  53. abstract class SumMetric[EI, Q, P, A, R] extends Metric[EI, Q, P, A, R] with QPAMetric[Q, P, A, R]

    Permalink

    Returns the sum of the score returned by the calculate method.

    Returns the sum of the score returned by the calculate method.

    EI

    Evaluation information

    Q

    Query

    P

    Predicted result

    A

    Actual result

    R

    Result, output of the function calculate, must be Numeric

  54. class ZeroMetric[EI, Q, P, A] extends Metric[EI, Q, P, A, Double]

    Permalink

    Returns zero.

    Returns zero. Useful as a placeholder during evaluation development when not all components are implemented.

    EI

    Evaluation information

    Q

    Query

    P

    Predicted result

    A

    Actual result

  55. trait IEngineFactory extends EngineFactory

    Permalink

    DEPRECATED.

    DEPRECATED. Use EngineFactory instead.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) Use EngineFactory instead.

  56. trait IFSPersistentModel[AP <: Params] extends LocalFileSystemPersistentModel[AP]

    Permalink

    DEPRECATED.

    DEPRECATED. Use LocalFileSystemPersistentModel instead.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) Use LocalFileSystemPersistentModel instead.

  57. trait IFSPersistentModelLoader[AP <: Params, M] extends LocalFileSystemPersistentModelLoader[AP, M]

    Permalink

    DEPRECATED.

    DEPRECATED. Use LocalFileSystemPersistentModelLoader instead.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) Use LocalFileSystemPersistentModelLoader instead.

  58. trait IPersistentModel[AP <: Params] extends PersistentModel[AP]

    Permalink

    DEPRECATED.

    DEPRECATED. Use PersistentModel instead.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) Use PersistentModel instead.

  59. trait IPersistentModelLoader[AP <: Params, M] extends PersistentModelLoader[AP, M]

    Permalink

    DEPRECATED.

    DEPRECATED. Use PersistentModelLoader instead.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) Use PersistentModelLoader instead.

  60. trait WithPrId extends AnyRef

    Permalink

    Mix in this trait for queries that contain prId (PredictedResultId).

    Mix in this trait for queries that contain prId (PredictedResultId). This is useful when your engine expects queries to also be associated with prId keys when feedback loop is enabled.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) To be removed in future releases.

  61. trait WithQuerySerializer extends CustomQuerySerializer

    Permalink

    DEPRECATED.

    DEPRECATED. Use CustomQuerySerializer instead.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) Use CustomQuerySerializer instead.

Value Members

  1. object Engine extends Serializable

    Permalink

    This object contains concrete implementation for some methods of the Engine class.

  2. object EngineParams extends Serializable

    Permalink

    Companion object for creating EngineParams instances.

  3. object FastEvalEngineWorkflow extends Serializable

    Permalink

    :: Experimental :: Workflow based on FastEvalEngine

    :: Experimental :: Workflow based on FastEvalEngine

    Annotations
    @Experimental()
  4. object IdentityPreparator extends Serializable

    Permalink

    Companion object of IdentityPreparator that conveniently returns an instance of the class of IdentityPreparator for use with EngineFactory.

  5. object LAverageServing extends Serializable

    Permalink

    A concrete implementation of LServing returning the average of all algorithms' predictions, where their classes are expected to be all Double.

  6. object LFirstServing extends Serializable

    Permalink

    A concrete implementation of LServing returning the first algorithm's prediction result directly without any modification.

  7. object LIdentityPreparator extends Serializable

    Permalink

    DEPRECATED.

    DEPRECATED. Use IdentityPreparator instead.

  8. object MetricEvaluator extends Serializable

    Permalink

    Companion object of MetricEvaluator

  9. object PIdentityPreparator extends Serializable

    Permalink

    DEPRECATED.

    DEPRECATED. Use IdentityPreparator instead.

  10. object Utils

    Permalink

    Controller utilities.

  11. object ZeroMetric extends Serializable

    Permalink

    Companion object of ZeroMetric

Inherited from AnyRef

Inherited from Any

Algorithm

Data Source

Engine

Evaluation

Helper

Preparator

Serving