Classification (classification)

Logistic Regression

class Orange.classification.LogisticRegressionLearner(penalty=’l2’, dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight=None, random_state=None, solver=’liblinear’, max_iter=100, multi_class=’ovr’, verbose=0, n_jobs=1, preprocessors=None)[source]

A wrapper for sklearn.linear_model.logistic.LogisticRegression. The following is its documentation:

Logistic Regression (aka logit, MaxEnt) classifier.

In the multiclass case, the training algorithm uses the one-vs-rest (OvR) scheme if the ‘multi_class’ option is set to ‘ovr’, and uses the cross- entropy loss if the ‘multi_class’ option is set to ‘multinomial’. (Currently the ‘multinomial’ option is supported only by the ‘lbfgs’, ‘sag’ and ‘newton-cg’ solvers.)

This class implements regularized logistic regression using the ‘liblinear’ library, ‘newton-cg’, ‘sag’ and ‘lbfgs’ solvers. It can handle both dense and sparse input. Use C-ordered arrays or CSR matrices containing 64-bit floats for optimal performance; any other input format will be converted (and copied).

The ‘newton-cg’, ‘sag’, and ‘lbfgs’ solvers support only L2 regularization with primal formulation. The ‘liblinear’ solver supports both L1 and L2 regularization, with a dual formulation only for the L2 penalty.

Read more in the User Guide.

Random Forest

class Orange.classification.RandomForestLearner(n_estimators=10, criterion=’gini’, max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=’auto’, max_leaf_nodes=None, bootstrap=True, oob_score=False, n_jobs=1, random_state=None, verbose=0, class_weight=None, preprocessors=None)[source]

A wrapper for sklearn.ensemble.forest.RandomForestClassifier. The following is its documentation:

A random forest classifier.

A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is always the same as the original input sample size but the samples are drawn with replacement if bootstrap=True (default).

Read more in the User Guide.

Simple Random Forest

class Orange.classification.SimpleRandomForestLearner(n_estimators=10, min_instances=2, max_depth=1024, max_majority=1.0, skip_prob=’sqrt’, seed=42)[source]

A random forest classifier, optimized for speed. Trees in the forest are constructed with SimpleTreeLearner classification trees.

Parameters:

n_estimators : int, optional (default = 10)

Number of trees in the forest.

min_instances : int, optional (default = 2)

Minimal number of data instances in leaves. When growing the three, new nodes are not introduced if they would result in leaves with fewer instances than min_instances. Instance count is weighed.

max_depth : int, optional (default = 1024)

Maximal depth of tree.

max_majority : float, optional (default = 1.0)

Maximal proportion of majority class. When this is exceeded, induction stops (only used for classification).

skip_prob : string, optional (default = “sqrt”)

Data attribute will be skipped with probability skip_prob.

  • if float, then skip attribute with this probability.
  • if “sqrt”, then skip_prob = 1 - sqrt(n_features) / n_features
  • if “log2”, then skip_prob = 1 - log2(n_features) / n_features

seed : int, optional (default = 42)

Random seed.

Softmax Regression

class Orange.classification.SoftmaxRegressionLearner(lambda_=1.0, preprocessors=None, **fmin_args)[source]

L2 regularized softmax regression classifier. Uses the L-BFGS algorithm to minimize the categorical cross entropy cost with L2 regularization. This model is suitable when dealing with a multi-class classification problem.

When using this learner you should:

  • choose a suitable regularization parameter lambda_,
  • consider using many logistic regression models (one for each value of the class variable) instead of softmax regression.
Parameters:

lambda_ : float, optional (default=1.0)

Regularization parameter. It controls trade-off between fitting the data and keeping parameters small. Higher values of lambda_ force parameters to be smaller.

preprocessors : list, optional (default=[RemoveNaNClasses(), RemoveNaNColumns(), Impute(), Continuize(), Normalize()])

Preprocessors are applied to data before training or testing. Default preprocessors:

  • remove columns with all values as NaN
  • replace NaN values with suitable values
  • continuize all discrete attributes,
  • transform the dataset so that the columns are on a similar scale,

fmin_args : dict, optional

Parameters for L-BFGS algorithm.

k-Nearest Neighbors

class Orange.classification.KNNLearner(n_neighbors=5, metric=’euclidean’, weights=’uniform’, algorithm=’auto’, metric_params=None, preprocessors=None)[source]

A wrapper for sklearn.neighbors.classification.KNeighborsClassifier. The following is its documentation:

Classifier implementing the k-nearest neighbors vote.

Read more in the User Guide.

Naive Bayes

class Orange.classification.NaiveBayesLearner(preprocessors=None)[source]

Naive Bayes classifier. Works only with discrete attributes. By default, continuous attributes are discretized.

Parameters:

preprocessors : list, optional (default=”[Orange.preprocess.Discretize]”)

An ordered list of preprocessors applied to data before training or testing.

The following code loads lenses data set (four discrete attributes and discrete class), constructs naive Bayesian learner, uses it on the entire data set to construct a classifier, and then applies classifier to the first three data instances:

>>> import Orange
>>> lenses = Orange.data.Table('lenses')
>>> nb = Orange.classification.NaiveBayesLearner()
>>> classifier = nb(lenses)
>>> classifier(lenses[0:3], True)
array([[ 0.04358755,  0.82671726,  0.12969519],
       [ 0.17428279,  0.20342097,  0.62229625],
       [ 0.18633359,  0.79518516,  0.01848125]])

For data sets that include continuous attributes,

Support Vector Machines

class Orange.classification.SVMLearner(C=1.0, kernel=’rbf’, degree=3, gamma=’auto’, coef0=0.0, shrinking=True, probability=False, tol=0.001, cache_size=200, max_iter=-1, preprocessors=None)[source]

A wrapper for sklearn.svm.classes.SVC. The following is its documentation:

C-Support Vector Classification.

The implementation is based on libsvm. The fit time complexity is more than quadratic with the number of samples which makes it hard to scale to dataset with more than a couple of 10000 samples.

The multiclass support is handled according to a one-vs-one scheme.

For details on the precise mathematical formulation of the provided kernel functions and how gamma, coef0 and degree affect each other, see the corresponding section in the narrative documentation: svm_kernels.

Read more in the User Guide.

Linear Support Vector Machines

class Orange.classification.LinearSVMLearner(penalty=’l2’, loss=’squared_hinge’, dual=True, tol=0.0001, C=1.0, multi_class=’ovr’, fit_intercept=True, intercept_scaling=True, random_state=None, preprocessors=None)[source]

A wrapper for sklearn.svm.classes.LinearSVC. The following is its documentation:

Linear Support Vector Classification.

Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.

This class supports both dense and sparse input and the multiclass support is handled according to a one-vs-the-rest scheme.

Read more in the User Guide.

Nu-Support Vector Machines

class Orange.classification.NuSVMLearner(nu=0.5, kernel=’rbf’, degree=3, gamma=’auto’, coef0=0.0, shrinking=True, probability=False, tol=0.001, cache_size=200, max_iter=-1, preprocessors=None)[source]

A wrapper for sklearn.svm.classes.NuSVC. The following is its documentation:

Nu-Support Vector Classification.

Similar to SVC but uses a parameter to control the number of support vectors.

The implementation is based on libsvm.

Read more in the User Guide.

One Class Support Vector Machines

class Orange.classification.OneClassSVMLearner(kernel=’rbf’, degree=3, gamma=’auto’, coef0=0.0, tol=0.001, nu=0.5, shrinking=True, cache_size=200, max_iter=-1, preprocessors=None)[source]

A wrapper for sklearn.svm.classes.OneClassSVM. The following is its documentation:

Unsupervised Outlier Detection.

Estimate the support of a high-dimensional distribution.

The implementation is based on libsvm.

Read more in the User Guide.

Classification Tree

Orange includes three implemenations of classification trees. TreeLearner is home-grown and properly handles multinominal and missing values. The one from scikit-learn, SklTreeLearner, is faster. Another home-grown, SimpleTreeLearner, is simpler and stil faster.

class Orange.classification.TreeLearner(*args, binarize=False, max_depth=None, min_samples_leaf=1, min_samples_split=2, sufficient_majority=0.95, **kwargs)[source]

Tree inducer with proper handling of nominal attributes and binarization.

The inducer can handle missing values of attributes and target. For discrete attributes with more than two possible values, each value can get a separate branch (binarize=False), or values can be grouped into two groups (binarize=True, default).

The tree growth can be limited by the required number of instances for internal nodes and for leafs, the sufficient proportion of majority class, and by the maximal depth of the tree.

If the tree is not binary, it can contain zero-branches.

Args:
binarize (bool):
if True the inducer will find optimal split into two subsets for values of discrete attributes. If False (default), each value gets its branch.
min_samples_leaf (float):
the minimal number of data instances in a leaf
min_samples_split (float):
the minimal nubmer of data instances that is split into subgroups

max_depth (int): the maximal depth of the tree

sufficient_majority (float):
a majority at which the data is not split further
Returns:
instance of OrangeTreeModel
build_tree(data, active_inst, level=1)[source]

Induce a tree from the given data

Returns:
root node (Node)
class Orange.classification.SklTreeLearner(criterion=’gini’, splitter=’best’, max_depth=None, min_samples_split=2, min_samples_leaf=1, max_features=None, random_state=None, max_leaf_nodes=None, preprocessors=None)[source]

Wrapper for SKL’s tree inducer

Simple Tree

class Orange.classification.SimpleTreeLearner(min_instances=2, max_depth=1024, max_majority=1.0, skip_prob=0.0, bootstrap=False, seed=42)[source]

Classification or regression tree learner. Uses gain ratio for classification and mean square error for regression. This learner was developed to speed-up random forest construction, but can also be used as a standalone tree learner.

min_instances : int, optional (default = 2)
Minimal number of data instances in leaves. When growing the three, new nodes are not introduced if they would result in leaves with fewer instances than min_instances. Instance count is weighed.
max_depth : int, optional (default = 1024)
Maximal depth of tree.
max_majority : float, optional (default = 1.0)
Maximal proportion of majority class. When this is exceeded, induction stops (only used for classification).
skip_prob : string, optional (default = 0.0)

Data attribute will be skipped with probability skip_prob.

  • if float, then skip attribute with this probability.
  • if “sqrt”, then skip_prob = 1 - sqrt(n_features) / n_features
  • if “log2”, then skip_prob = 1 - log2(n_features) / n_features
bootstrap : data table, optional (default = False)
A bootstrap data set.
seed : int, optional (default = 42)
Random seed.

Majority Classifier

class Orange.classification.MajorityLearner(preprocessors=None)[source]

A majority classifier. Always returns most frequent class from the training set, regardless of the attribute values from the test data instance. Returns class value distribution if class probabilities are requested. Can be used as a baseline when comparing classifiers.

In the special case of uniform class distribution within the training data, class value is selected randomly. In order to produce consistent results on the same data set, this value is selected based on hash of the class vector.

Elliptic Envelope

class Orange.classification.EllipticEnvelopeLearner(store_precision=True, assume_centered=False, support_fraction=None, contamination=0.1, random_state=None, preprocessors=None)[source]

A wrapper for sklearn.covariance.outlier_detection.EllipticEnvelope. The following is its documentation:

An object for detecting outliers in a Gaussian distributed dataset.

Read more in the User Guide.

Neural Network

class Orange.classification.NNClassificationLearner(hidden_layer_sizes=(100, ), activation=’relu’, solver=’adam’, alpha=0.0001, batch_size=’auto’, learning_rate=’constant’, learning_rate_init=0.001, power_t=0.5, max_iter=200, shuffle=True, random_state=None, tol=0.0001, verbose=False, warm_start=False, momentum=0.9, nesterovs_momentum=True, early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08, preprocessors=None)[source]

A wrapper for sklearn.neural_network.multilayer_perceptron.MLPClassifier. The following is its documentation:

Multi-layer Perceptron classifier.

This model optimizes the log-loss function using LBFGS or stochastic gradient descent.

New in version 0.18.

CN2 Rule Induction

Induction of rules works by finding a rule that covers some learning instances, removing these instances, and repeating this until all instances are covered. Rules are scored by heuristics such as impurity of class distribution of covered instances. The module includes common rule-learning algorithms, and allows for replacing rule search strategies, scoring and other components.

class Orange.classification.rules.CN2Learner(preprocessors=None, base_rules=None)[source]

Classic CN2 inducer that constructs a list of ordered rules. To evaluate found hypotheses, entropy measure is used. Returns a CN2Classifier if called with data.

References

[R1]“The CN2 Induction Algorithm”, Peter Clark and Tim Niblett, Machine Learning Journal, 3 (4), pp261-283, (1989)
class Orange.classification.rules.CN2UnorderedLearner(preprocessors=None, base_rules=None)[source]

Construct a set of unordered rules.

Rules are learnt for each class individually and scored by the relative frequency of the class corrected by the Laplace correction. After adding a rule, only the covered examples of that class are removed.

The code below loads the iris data set (four continuous attributes and a discrete class) and fits the learner.

import Orange
data = Orange.data.Table('iris')
learner = Orange.classification.CN2UnorderedLearner()

# consider up to 10 solution streams at one time
learner.rule_finder.search_algorithm.beam_width = 10

# continuous value space is constrained to reduce computation time
learner.rule_finder.search_strategy.constrain_continuous = True

# found rules must cover at least 15 examples
learner.rule_finder.general_validator.min_covered_examples = 15

# found rules may combine at most 2 selectors (conditions)
learner.rule_finder.general_validator.max_rule_length = 2

classifier = learner(data)

References

[R2]“Rule Induction with CN2: Some Recent Improvements”, Peter Clark and Robin Boswell, Machine Learning - Proceedings of the 5th European Conference (EWSL-91), pp151-163, 1991
class Orange.classification.rules.CN2SDLearner(preprocessors=None, base_rules=None)[source]

Ordered CN2SD inducer that constructs a list of ordered rules. To evaluate found hypotheses, Weighted relative accuracy measure is used. Returns a CN2SDClassifier if called with data.

In this setting, ordered rule induction refers exclusively to finding best rule conditions and assigning the majority class in the rule head (target class is set to None). To later predict instances, rules will be regarded as unordered.

Notes

A weighted covering algorithm is applied, in which subsequently induced rules also represent interesting and sufficiently large subgroups of the population. Covered positive examples are not deleted from the learning set, rather their weight is reduced.

The algorithm demonstrates how classification rule learning (predictive induction) can be adapted to subgroup discovery, a task at the intersection of predictive and descriptive induction.

References

[R3]“Subgroup Discovery with CN2-SD”, Nada Lavrač et al., Journal of Machine Learning Research 5 (2004), 153-188, 2004
class Orange.classification.rules.CN2SDUnorderedLearner(preprocessors=None, base_rules=None)[source]

Unordered CN2SD inducer that constructs a set of unordered rules. To evaluate found hypotheses, Weighted relative accuracy measure is used. Returns a CN2SDUnorderedClassifier if called with data.

Notes

A weighted covering algorithm is applied, in which subsequently induced rules also represent interesting and sufficiently large subgroups of the population. Covered positive examples are not deleted from the learning set, rather their weight is reduced.

The algorithm demonstrates how classification rule learning (predictive induction) can be adapted to subgroup discovery, a task at the intersection of predictive and descriptive induction.

References

[R4]“Subgroup Discovery with CN2-SD”, Nada Lavrač et al., Journal of Machine Learning Research 5 (2004), 153-188, 2004