scikit-learn/scikit-learn
Papers Referenced in This Repository
Gaussian processes for machine learning.
AbstractMachine learning clustering techniques are used to characterize and, after the training phase, to identify phases within an ignition process. For the ethanol mechanism used in this paper, four physically identifiable phases were found and characterized: the initiation phase, preignition phas...
Introduction to information retrieval.
Class-tested and coherent, this textbook teaches classical and web information retrieval, including web search and the related areas of text classification and text clustering from basic concepts. It gives an up-to-date treatment of all aspects of the design and implementation of systems for gatheri...
Greedy function approximation: A gradient boosting machine.
Function estimation/approximation is viewed from the perspective\nof numerical optimization in function space, rather than parameter space. A\nconnection is made between stagewise additive expansions and steepest-descent\nminimization. A general gradient descent “boosting” paradigm is\ndeveloped for...
Automatic model construction with Gaussian processes
This thesis develops a method for automatically constructing, visualizing and describing a large class of models, useful for forecasting and finding structure in domains such as time series, geological formations, and physical dynamics. These models, based on Gaussian processes, can capture many typ...
Making and Evaluating Point Forecasts
Typically, point forecasting methods are compared and assessed by means of an error measure or scoring function, with the absolute error and the squared error being key examples. The individual scores are averaged over forecast cases, to result in a summary measure of the predictive performance, suc...
Adam: A Method for Stochastic Optimization
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescal...
LIBLINEAR: A Library for Large Linear Classification
AbstractWe present a generalization of the classical mathematical homogenization theory aimed at accounting for finite unit cell distortions, which gives rise to a nonperiodic asymptotic expansion. We introduce an auxiliary macro‐deformed configuration, where the overall Cauchy stress is defined, an...
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
BBMRI-NL, X-omics, VOILA, Medical Delta and the Dutch Research Council (NWO-VENI).
A Comparative Analysis of Community Detection Algorithms on Artificial Networks
Many community detection algorithms have been developed to uncover the mesoscopic properties of complex networks. However how good an algorithm is, in terms of accuracy and computing time, remains still open. Testing algorithms on real-world network has certain restrictions which made their insights...
Random Features for Large-Scale Kernel Machines
In this paper, we contributed a stereo face recognition formulation which combines appearance and disparity/depth at feature level. We showed that the present-day passive stereovision in combination with 2D appearance images can match up to other methods which rely on active depth data. A Reduced Mu...
Commonly used software tools produce conflicting and overly-optimistic AUPRC values
The precision-recall curve (PRC) and the area under the precision-recall curve (AUPRC) are useful for quantifying classification performance. They are commonly used in situations with imbalanced classes, such as cancer diagnosis and cell type annotation. We evaluate 10 popular tools for plotting PRC...
Calibration of Machine Learning Models
The evaluation of machine learning models is a crucial step before their application because it is essential to assess how well a model will behave for every single case. In many real applications, not only is it important to know the “total” or the “average” error of the model, it is also important...
On classification, ranking, and probability estimation.
The aim of this thesis was the study of respiration in ocean margin \nsediments and the assessments of tools needed for this purpose.\n \n \nThe first study was on the biological pump and global respiration \npatterns in the deep ocean using an empirical model based \non sediment oxygen consumption ...
On the “degrees of freedom” of the lasso
We study the effective degrees of freedom of the lasso in the framework of Stein’s unbiased risk estimation (SURE). We show that the number of nonzero coefficients is an unbiased estimate for the degrees of freedom of the lasso—a conclusion that requires no special assumption on the predictors. In a...
Asymptotics for the minimum covariance determinant estimator
Consistency is shown for the minimum covariance determinant (MCD) estimators of multivariate location and scale and asymptotic normality is shown for the former. The proofs are made possible by showing a separating ellipsoid property for the MCD subset of observations. An analogous property is shown...
Training linear SVMs in linear time.
Linear Support Vector Machines (SVMs) have become one of the most prominent machine learning techniques for high-dimensional sparse data commonly encountered in applications like text classification, word-sense disambiguation, and drug design. These applications involve a large number of examples n ...
Self-Organizing Feature Maps Identify Proteins Critical to Learning in a Mouse Model of Down Syndrome
Down syndrome (DS) is a chromosomal abnormality (trisomy of human chromosome 21) associated with intellectual disability and affecting approximately one in 1000 live births worldwide. The overexpression of genes encoded by the extra copy of a normal chromosome in DS is believed to be sufficient to p...
Accelerated Hierarchical Density Based Clustering
We present an accelerated algorithm for hierarchical density based clustering. Our new algorithm improves upon HDBSCAN*, which itself provided a significant qualitative improvement over the popular DBSCAN algorithm. The accelerated HDBSCAN* algorithm provides comparable performance to DBSCAN, while ...
Information theoretic measures for clusterings comparison: is a correction for chance necessary?
Information theoretic based measures form a fundamental class of similarity measures for comparing clusterings, beside the class of pair-counting based and set-matching based measures. In this paper, we discuss the necessity of correction for chance for information theoretic based measures for clust...
Learning to Find Pre-Images.
Travel arrangements and flight ticket booking via internet is widely used nowadays and follow already certain standards. Although increasing activity for multimedia/web education components can be observed, we are far away from standards in this important area. Statistics can possibly profit the mos...
An implementation of a randomized algorithm for principal component analysis.
This thesis addresses the language recognition problem with a special focus on phonotactic language recognition. A full description of different steps in a language recognition system is provided. We study state-of-the-art speech modeling techniques in language recognition that comprise phonotactic,...
On Grouping for Maximum Homogeneity
Abstract Given a set of arbitrary numbers, what is a practical procedure for grouping them so that the variance within groups is minimized? An answer to this question, including a description of an automatic computer program, is given for problems up to the size where 200 numbers are to be placed in...
MICE: Multivariate Imputation by Chained Equations in R
The R package mice imputes incomplete multivariate data by chained equations. The software mice 1.0 appeared in the year 2000 as an S-PLUS library, and in 2001 as an R package. mice 1.0 introduced predictor selection, passive imputation and automatic pooling. This article documents mice, which exten...
Missing value estimation methods for DNA microarrays.
Abstract Motivation: Gene expression microarray experiments can generate data sets with multiple missing expression values. Unfortunately, many algorithms for gene expression analysis require a complete matrix of gene array values as input. For example, methods such as hierarchical clustering and K-...
Efficient additive kernels via explicit feature maps
Maji and Berg have recently introduced an explicit feature map approximating the intersection kernel. This enables efficient learning methods for linear kernels to be applied to the non-linear intersection kernel, expanding the applicability of this model to much larger problems. In this paper we ge...
Fast and scalable polynomial kernels via explicit feature maps
Approximation of non-linear kernels using random feature mapping has been successfully employed in large-scale data analysis applications, accelerating the training of kernel machines. While previous random feature mappings run in O(ndD) time for $n$ training samples in d-dimensional space and D ran...
Least angle regression
The purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to s...
Strictly Proper Scoring Rules, Prediction, and Estimation
Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distributionF if he ...
Design of the 2015 ChaLearn AutoML challenge.
ChaLearn is organizing the Automatic Machine Learning (AutoML) contest for IJCNN 2015, which challenges participants to solve classification and regression problems without any human intervention. Participants' code is automatically run on the contest servers to train and test learning machines. How...
The relationship between Precision-Recall and ROC curves.
Receiver Operator Characteristic (ROC) curves are commonly used to present results for binary decision problems in machine learning. However, when dealing with highly skewed datasets, Precision-Recall (PR) curves give a more informative picture of an algorithm's performance. We show that a deep conn...
Macro F1 and Macro F1
The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalen...
The DET curve in assessment of detection task performance.
Abstract : We introduce the DET Curve as a means of representing performance on detection tasks that involve a tradeoff of error types. We discuss why we prefer it to the traditional ROC Curve and offer several examples of its use in speaker recognition and language recognition. We explain why it is...
On Linear DETs.
This paper investigates the properties of a popular ROC variant - the detection error trade-off plot (DET). In particular, we derive a set of conditions on the underlying probability distributions to produce linear DET plots in a generalized setting. We show that the linear DETs on a normal deviate ...
A comparison of event models for naive bayes text classification
Article Free Access Share on Distributional clustering of words for text classification Authors: L. Douglas Baker School of Computer Science, Carnegie Mellon University, Pittsburgh, PA and Just Research 4616 Henry Street, Pittsburgh, PA School of Computer Science, Carnegie Mellon University, Pittsbu...
Nonlinear Component Analysis as a Kernel Eigenvalue Problem.
A new method for performing a nonlinear form of principal component analysis is proposed. By the use of integral operator kernel functions, one can efficiently compute principal components in high-dimensional feature spaces, related to input space by some nonlinear map—for instance, the space of all...
Label Propagation and Quadratic Criterion.
Abstract This chapter shows how the different graph-based algorithms for semi-supervised learning can be cast into a common framework where one minimizes a quadratic cost criterion whose closed-form solution is found by solving a linear system of size n (total number of data points). The cost criter...
Least Median of Squares Regression
Abstract Classical least squares regression consists of minimizing the sum of the squared residuals. Many authors have produced more robust versions of this estimator by replacing the square by something else, such as the absolute value. In this article a different approach is introduced in which th...
A Fast Algorithm for the Minimum Covariance Determinant Estimator
The minimum covariance determinant (MCD) method of Rousseeuw is a highly robust estimator of multivariate location and scatter. Its objective is to find h observations (out of n) whose covariance matrix has the lowest determinant. Until now, applications of the MCD were hampered by the computation t...
Small sample corrections for LTS and MCD
The least trimmed squares estimator and the minimum covariance determinant estimator Rousseeuw (1984) are frequently used robust estimators of regression and of location and scatter. Consistency factors can be computed for both methods to make the estimators consistent at the normal model. However, ...
Algorithms for Nonnegative Matrix Factorization with the β-Divergence.
This letter describes algorithms for nonnegative matrix factorization (NMF) with the β-divergence (β-NMF). The β-divergence is a family of cost functions parameterized by a single shape parameter β that takes the Euclidean distance, the Kullback-Leibler divergence, and the Itakura-Saito divergence a...
Modern Information Retrieval - the concepts and technology behind search, Second edition
Intelligent interaction between humans and computers has been a dream of artificial intelligence since the beginning of digital era and one of the original motivations behind the creation of artificial intelligence. A key step towards the achievement of such an ambitious goal is to enable the Questi...
Special Invited Paper-Additive logistic regression: A statistical view of boosting
Boosting is one of the most important recent developments in\nclassification methodology. Boosting works by sequentially applying a\nclassification algorithm to reweighted versions of the training data and then\ntaking a weighted majority vote of the sequence of classifiers thus produced.\nFor many ...
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU ...
A New Vector Partition of the Probability Score
A new vector partition of the probability, or Brier, score (PS) is formulated and the nature and properties of this partition are described. The relationships between the terms in this partition and the terms in the original vector partition of the PS are indicated. The new partition consists of thr...
Statistical Foundations of Actuarial Learning and its Applications
The aim of this manuscript is to provide the mathematical and statistical foundations of actuarial learning. This is key to most actuarial tasks like insurance pricing, product development, claims reserving and risk management. The basic approach to these tasks is regression modeling. This manuscrip...
V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure
As it is not known a priori which size of the context region around the object yields to most useful information, we pose a second research question.Research question 2 (RQ2): What size of the context region is best suited to lower the false-detection rate?We aim at presenting and evaluating novel a...
Sparse inverse covariance estimation with the graphical lasso.
Abstract We consider the problem of estimating sparse graphs by a lasso penalty applied to the inverse covariance matrix. Using a coordinate descent procedure for the lasso, we develop a simple algorithm—the graphical lasso—that is remarkably fast: It solves a 1000-node problem (∼500000 parameters) ...
Stochastic variational inference.
AbstractTools for estimating population structure from genetic data are now used in a wide variety of applications in population genetics. However, inferring population structure in large modern data sets imposes severe computational challenges. Here, we develop efficient algorithms for approximate ...
Machine Learning Applications to Land and Structure Valuation
In some applications of supervised machine learning, it is desirable to trade model complexity with greater interpretability for some covariates while letting other covariates remain a “black box”. An important example is hedonic property valuation modeling, where machine learning techniques typical...
Generalized Boosted Models: A guide to the gbm package
This article provides an introduction to ensemble statistical procedures as a special case of algorithmic methods. The discussion begins with classification and regression trees (CART) as a didactic device to introduce many of the key issues. Following the material on CART is a consideration of cros...
Feature hashing for large scale multitask learning.
Empirical evidence suggests that hashing is an effective strategy for dimensionality reduction and practical nonparametric estimation. In this paper we provide exponential tail bounds for feature hashing and show that the interaction between random subspaces is negligible with high probability. We d...
Stop Word Lists in Free Open-source Software Packages
Open-source software packages for language processing often include stop word lists. Users may apply them without awareness of their surprising omissions (e.g. “hasn’t” but not “hadn’t”) and inclusions (“computer”), or their incompatibility with a particular tokenizer. Motivated by issues raised abo...
Generalized RBF feature maps for Efficient Detection
Kernel methods yield state-of-the-art performance in certain applications such as image classification and object detection. However, large scale problems require machine learning techniques of at most linear complexity and these are usually limited to linear kernels. This unfortunately rules out go...
Notes on Regularized Least Squares
This is a collection of information about regularized least squares (RLS). The facts here are not “new results”, but we have not seen them usefully collected together before. A key goal of this work is to demonstrate that with RLS, we get certain things “for free”: if we can solve a single supervise...
Performance Evaluation of RANSAC Family.
RANSAC (Random Sample Consensus) has been popular in regression problem with samples contaminated with outliers. It has been a milestone of many researches on robust estimators, but there are a few survey and performance analysis on them. This paper categorizes them on their objectives: being accura...
More on Multidimensional Scaling and Unfolding in R: smacof Version 2.
The smacof package offers a comprehensive implementation of multidimensional scaling (MDS) techniques in R. Since its first publication (De Leeuw and Mair 2009b) the functionality of the package has been enhanced, and several additional methods, features and utilities were added. Major updates inclu...
Automated optimized parameters for T-distributed stochastic neighbor embedding improve visualization and analysis of large datasets
Accurate and comprehensive extraction of information from high-dimensional single cell datasets necessitates faithful visualizations to assess biological populations. A state-of-the-art algorithm for non-linear dimension reduction, t-SNE, requires multiple heuristics and fails to produce clear repre...
Visualizing Data using t-SNE
Tese de mestrado integrado. Engenharia Electrotécnica e de Computadores. Faculdade de Engenharia. Universidade do Porto. 2008
Probabilistic Forecasting
A probabilistic forecast takes the form of a predictive probability distribution over future quantities or events of interest. Probabilistic forecasting aims to maximize the sharpness of the predictive distributions, subject to calibration, on the basis of the available information set. We formalize...
Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies
Humanities majors often find themselves in jobs where they either manage programmers or work with them in close collaboration. These interactions often pose difficulties because specialists in literature, history, philosophy, and so on are not usually "code literate." They do not understand what tas...
Precision-Recall-Gain Curves: PR Analysis Done Right.
Woodworth’s Two-Component model (1899) partitioned speeded limb movements into two distinct phases: (1) a central ballistic open-loop mechanism and (2) a closed-loop feedback component. The present study investigated the implementation of multi-gain control configurations that utilized separate gain...
Multidimensional Binary Search Trees Used for Associative Searching.
This paper develops the multidimensional binary search tree (or k -d tree, where k is the dimensionality of the search space) as a data structure for storage of information to be retrieved by associative searches. The k -d tree is defined and examples are given. It is shown to be quite efficient in ...
LOF: Identifying Density-Based Local Outliers.
For many KDD applications, such as detecting criminal activities in E-commerce, finding the rare instances or the outliers, can be more interesting than finding the common patterns. Existing work in outlier detection regards being an outlier as a binary property. In this paper, we contend that for m...
Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
For large scale learning problems, it is desirable if we can obtain the optimal model parameters by going through the data in only one pass. Polyak and Juditsky (1992) showed that asymptotically the test performance of the simple average of the parameters obtained by stochastic gradient descent (SGD...
In Defense of One-Vs-All Classification
Nested dichotomies are a standard statistical technique for tackling certain polytomous classification problems with logistic regression. They can be represented as binary trees that recursively split a multi-class classification task into a system of dichotomies and provide a statistically sound wa...
Unsupervised Word Sense Disambiguation Rivaling Supervised Methods
This paper presents an unsupervised learning algorithm for sense disambiguation that, when trained on unannotated English text, rivals the performance of supervised techniques that require time-consuming hand annotations. The algorithm is based on two powerful constraints---that words tend to have o...
Predicting good probabilities with supervised learning.
We examine the relationship between the predictions made by different learning algorithms and true posterior probabilities. We show that maximum margin methods such as boosted trees and boosted stumps push probability mass away from 0 and 1 yielding a characteristic sigmoid shaped distortion in the ...
Shrinkage Algorithms for MMSE Covariance Estimation
We address covariance estimation in the sense of minimum mean-squared error (MMSE) for Gaussian samples. Specifically, we consider shrinkage methods which are suitable for high dimensional problems with a small number of samples (large p small n). First, we improve on the Ledoit-Wolf (LW) method by ...
Sequential Karhunen-Loeve basis extraction and its application to images
The Karhunen-Loeve (KL) transform is an optimal method for approximating a set of vectors or images, which was used in image processing and computer vision for several tasks such as face and object recognition. Its computational demands and its batch calculation nature have limited its application. ...
Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions
Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for pe...
A Blockwise Descent Algorithm for Group-penalized Multiresponse and Multinomial Regression
In this paper we purpose a blockwise descent algorithm for group-penalized multiresponse regression. Using a quasi-newton framework we extend this to group-penalized multinomial regression. We give a publicly available implementation for these in R, and compare the speed of this algorithm to a compe...
Transforming classifier scores into accurate multiclass probability estimates
Class membership probability estimates are important for many applications of data mining in which classification outputs are combined with other sources of information for decision-making, such as example-dependent misclassification costs, the outputs of other classifiers, or domain knowledge. Prev...
Convergence Theory for Preconditioned Eigenvalue Solvers in a Nutshell
Preconditioned iterative methods for numerical solution of large matrix eigenvalue problems are increasingly gaining importance in various application areas, ranging from material sciences to data mining. Some of them, e.g., those using multilevel preconditioning for elliptic differential operators ...
Inter-Coder Agreement for Computational Linguistics
This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; ...
Pattern Recognition and Machine Learning
The Journal of Electronic Imaging (JEI), copublished bimonthly with the Society for Imaging Science and Technology, publishes peer-reviewed papers that cover research and applications in all areas of electronic imaging science and technology.
Properties of the Hubert-Arabie adjusted Rand index.
This article provides an investigation of cluster validation indices that relates 4 of the indices to the L. Hubert and P. Arabie (1985) adjusted Rand index--the cluster validation measure of choice (G. W. Milligan & M. C. Cooper, 1986). It is shown how these other indices can be "roughly" transform...
Goodness of Fit and Related Inference Processes for Quantile Regression
Abstract We introduce a goodness-of-fit process for quantile regression analogous to the conventional R2 statistic of least squares regression. Several related inference processes designed to test composite hypotheses about the combined effect of several covariates over an entire range of conditiona...
Permutation Tests for Studying Classifier Performance
We explore the framework of permutation-based p-values for assessing the performance of classifiers. In this paper we study two simple permutation tests. The first test assess whether the classifier has found a real class structure in the data; the corresponding null distribution is estimated by per...
Flexible smoothing with B-splines and penalties
B-splines are attractive for nonparametric modelling, but choosing the optimal number and positions of knots is a complex task. Equidistant knots can be used, but their small and discrete number allows only limited control over smoothness and fit. We propose to use a relatively large number of knots...
A Newton-CG Algorithm with Complexity Guarantees for Smooth Unconstrained Optimization
We consider minimization of a smooth nonconvex objective function using an iterative algorithm based on Newton's method and the linear conjugate gradient algorithm, with explicit detection and use of negative curvature directions for the Hessian of the objective function. The algorithm tracks Newton...