ECML/PKDD 2013 Workshop
Tensor Methods for Machine Learning
| When | Friday, 27th of September |
| Where | Room R6-66, Congress Centre U Hájků, Prague |
Updated Schedule!
| 10:55 - 11:35 | Morten Mørup |
Tensor Decompositions for Machine Learning and the Modelling of Neuroimaging Data |
| 11:35 - 12:15 | Lieven de Lathauwer | Advances in (Numerical) Linear Algebra |
| 12:15 - 13:45 | Lunch Break | |
| 13:45 - 14:25 | Taylan Cemgil |
Probabilistic Latent Tensor Factorization with Applications to Audio Processing and Source Separation |
| 14:25 - 14:45 | Denis Krompass | Non-Negative Tensor Factorization with RESCAL |
| 14:45 - 15:00 | Spotlight Talks | |
| 15:00 - 15:45 | Poster Session and Coffee Break | |
| 15:45 - 16:25 | Steffen Rendle | Factorization Machines |
| 16:25 - 17:05 | Pauli Miettinen | Boolean Tensor and Matrix Factorization |
| 17:05 - 17:15 | Discussion | |
Workshop Description
Tensors, as generalizations of vectors and matrices, have become increasingly popular in different areas of machine learning and data mining, where they are employed to approach a diverse number of difficult learning and analysis tasks. Prominent examples include learning on multi-relational data and large-scale knowledge bases, recommendation systems, computer vision, mining boolean data, neuroimaging or the analysis of time-varying networks. The success of tensors methods is strongly related to their ability to efficiently model, analyse and predict data with multiple modalities. To address specific challenges and problems, a variety of methods has been developed in different fields of application. This workshop should serve as a basis for an interdisciplinary exchange of methods, ideas and techniques, with the goal to develop a deeper understanding of tensor methods in machine learning, further advance existing approaches and enable new approaches to important problems. A particular focus of this workshop is to uncover underlying principles in tensor methods, their applications and associated problems. The workshop is intended for researchers in the machine learning, data minining and tensor communities to discuss novel methods and applications as well as theoretical advances.
The workshop consists of contributed talks, poster sessions and a number of invited talks which will cover important work and recents developments in tensor methods. Furthermore, the workshop will include open discussion sessions to encourage the exchange of ideas and the development of a common understanding of problems and methods among the participants of the workshop.
Call for Papers
We invite the submission of short and regular papers to the workshop. Submitted papers should be at most 5 (extended abstract) or 10 (regular) pages long and formatted according to the Springer LNAI guidelines.
All submitted manuscripts will be subject to peer reviews by members of the program committee. Selected papers will be presented as full-length or spotlight talks during the morning and afternoon sessions of the workshop. All authors of selected papers are also invited to participate in poster sessions. Topics of interest include, but are not limited to
- Theoretical Analysis Statistical analysis and learning theory related to tensor methods, factorizations or multi-way analysis.
- Algorithms and Methods Novel techniques and methods for tensor factorization or tensor completion, such as new factorization models, loss functions or regularization methods, probabilistic/Bayesian approaches etc. We also encourage submissions approaching large scale or distributed problems.
- Applications and Empirical Studies Novel applications of tensor methods in machine learning and statistics. We also encourage submissions that provide new insight into tensor methods through empirical studies.
- Related Learning Methods Related techniques and methods in machine learning such as matrix factorizations that can advance the understanding and versatility of tensor methods.
Paper Submission
To submit your manuscript via EasyChair, please follow the link
Important Dates
For paper submission, please consider the following deadlines
- Paper Submission (Extended!) Friday, July 5th, 2013
- Acceptance Notification Friday, July 19th, 2013
- Camera-Ready Paper Submission Friday, August 2nd, 2013
Organizers
- Maximilian Nickel, LMU Munich
- Volker Tresp, Siemens AG
Program Committee
- Alwin Stegeman, University of Groningen
- Evrim Acar Ataman, University of Copenhagen
- Franz Király, TU Berlin
- Jaakko Hollmén, Aalto University
- Pauli Miettinen, Max-Planck Institut für Informatik
- Rainer Gemulla, Max-Planck Institut für Informatik
- Ryota Tomioka, University of Tokyo
- Shipeng Yu, Siemens Medical Solutions USA
- Taylan Cemgil, Bogazici University Istanbul
Invited Talks
-
Morten Mørup
Tensor Decompositions for Machine Learning and the Modelling of Neuroimaging Data
Tensor decompositions have several advantages over (two-way) matrix factorization methods for unsupervised learning/exploratory data analysis including uniqueness of solution and the ability to explicitly exploit the multi-way structure that is lost when collapsing some of the modes of the tensor in order to analyze the data by matrix factorization approaches. This talk will in particular focus on tensor decompositions for the modeling of neuroimaging data where important challenges include extracting consistent, reproducible patterns of activation across trials, subjects, and/or conditions. Emphasis will be given both to the extension of tensor decomposition methods for the modeling of latency and shape changes in EEG and fMRI as well as the modeling of multi-subject brain connectivity using non-parametric relational modeling approaches.
References
- K. W. Andersen, M. Mørup, H. Siebner, K. H. Madsen, L. K. Hansen, Identifying Modular Relations In Complex Brain Networksi, Machine Learning for Signal Processing (MLSP), 2012 IEEE International Workshop on, pp. 1-6, 2012.
- M. Mørup, Applications of tensor (multiway array) factorizations and decompositions in data mining, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 1(1), pp. 24-40, 2011
- M. Mørup, L. K. Hansen, K. H. Madsen, Frequency Constrained ShiftCP Modeling of Neuroimaging Data, invited paper, Asilomar-SSC, 2011
- M. Mørup, L. K. Hansen, K. H. Madsen, Modeling Latency and Shape Changes in Trial Based Neuroimaging Data, invited paper, Asilomar-SSC, 2011
- M. Mørup, K. H. Madsen, A. M. Dogonowski, H. Siebner, L. K. Hansen, Infinite Relational Modeling of Functional Connectivity in Resting State fMRI, Neural Information Processing Systems, 2010
- M. Mørup, L. K. Hansen, Automatic Relevance Determination for Multiway Models, Journal of Chemometrics, Special Issue: In Honor of Professor Richard A. Harshman, vol. 23(7-8), pp. 352 - 363, 2009
- M. Mørup, L. K. Hansen, S. M. Arnfred, L. Lim, K. H. Madsen, Shift Invariant Multilinear Decomposition of Neuroimaging Data, NeuroImage, vol. 42(4), pp. 1439-50, 2008
-
Lieven de Lathauwer
Advances in (Numerical) Linear Algebra
Recently important progress has been made in the understanding of the conditions under which tensor decompositions are unique. The uniqueness properties of decompositions such as the Canonical Polyadic Decomposition are at the heart of tensor based signal processing, data analysis and machine learning. We briefly sketch the state of the art.
Also in numerical multilinear algebra important progress has recently been made. It has been recognized that tensor product structure allows a very efficient storage and handling of the Jacobian and (approximate) Hessian of the cost function. On the other hand, multilinearity allows global optimization in (scaled) line and plane search. Although there are many possibilities for decomposition symmetry and factor structure, these may be conveniently handled. We demonstrate the algorithms using Tensorlab, a MATLAB toolbox for tensors and tensor computations that we have recently released.
References
- Tensorlab http://www.esat.kuleuven.be/sista/tensorlab/
-
I. Domanov, L. De Lathauwer,
“On the Uniqueness of the Canonical Polyadic
Decomposition — Part I: Basic Results and
Uniqueness of One Factor Matrix”
,
SIAM J. Matrix Anal. Appl., Vol. 34, No. 3, 2013, pp. 855–875. http://epubs.siam.org/doi/abs/10.1137/120877234 -
I. Domanov, L. De Lathauwer,
“On the Uniqueness of the Canonical Polyadic
Decomposition — Part II: Overall Uniqueness”
,
SIAM J. Matrix Anal. Appl., Vol. 34, No. 3, 2013, pp. 876–903. http://epubs.siam.org/doi/abs/10.1137/120877258 -
L. Sorber, M. Van Barel, L. De Lathauwer,
“Optimization-Based Algorithms for Tensor
Decompositions: Canonical Polyadic Decomposition,
Decomposition in rank-(Lr,Lr, 1) Terms and a New
Generalization”
,
Tech. Report 12-37, ESAT-SISTA, KU Leuven (Leuven, Belgium), 2012, SIAM. J. Opt., to appear. ftp://ftp.esat.kuleuven.ac.be/pub/SISTA/ida/reports/12-37.pdf
-
Steffen Rendle
Factorization Machines
Tensor factorization approaches have shown high predictive accuracy in several important machine learning problems. However, tensor factorization models usually lack flexibility and are restricted to categorical variables.
In this talk, I present factorization machines which are based on standard feature-engineering / design matrices. I will discuss the relationship of factorization machines to standard linear and polynomial models as well as to well-known factorization models. Several learning methods for factorization machines are presented, among them coordinate descent and MCMC inference with Gibbs sampling.
References
-
Steffen Rendle,
Factorization Machines with libFM
,
in ACM Transactions on Intelligent Systems and Technology (TIST 2012), 3(3), ACM, May 2012. http://dl.acm.org/authorize?6798743 -
Steffen Rendle,
Scaling Factorization Machines to Relational Data
,
in Proceedings of the 39th international conference on Very Large Data Bases (VLDB 2013), 2013, Trento, Italy. http://www.vldb.org/pvldb/vol6/p337-rendle.pdf -
Steffen Rendle, Christoph Freudenthaler, Lars Schmidt-Thieme,
Factorizing Personalized Markov Chains for Next-Basket
Recommendation
,
in Proceedings of the 19th International World Wide Web Conference (WWW 2010), ACM. http://dl.acm.org/authorize?237503
-
Steffen Rendle,
Factorization Machines with libFM
,
-
Pauli Miettinen
Boolean Tensor and Matrix Factorization
Boolean matrix decomposition represents a given binary matrix as a Boolean product of two (possibly smaller) binary matrices. Similarly, Boolean tensor decompositions decompose binary tensors into binary factors. The crux of these methods is the use of Boolean algebra, replacing addition with logical OR, giving the decompositions more combinatorial flavour.
Boolean matrix and tensor decompositions have been studied and used in many fields, including extremal combinatorics, communication complexity, and psychometrics, to name a few. In recent years, they have seen an increased amount of interest in data mining, providing a powerful tool that can be used to generalize many existing data mining problems. The Boolean algebra can help on sparsity and interpretability, and allows modelling different type of behaviour than normal algebra, but its use usually comes with increased computational complexity.
In this talk we go thru the basics of Boolean matrix and tensor decompositions, explain the main similarities and dissimilarities between Boolean and normal decompositions, and talk about applications of Boolean tensor decompositions to data mining and information extraction. We will cover the main algorithmic ideas and point out some open problems.
References
-
Pauli Miettinen,
Boolean Tensor Factorizations.
Proc. 11th IEEE International Conference on Data Mining (ICDM2011), 2011, 447–456. 10.1109/ICDM.2011.28 http://www.mpi-inf.mpg.de/~pmiettin/papers/BooleanTensorFactorizationsICDM.pdf -
Dóra Erdős Pauli Miettinen, Discovering Facts with Boolean Tensor Tucker Decomposition,
Proc. 2013 ACM International Conference on Infortmation and Knowledge Management (CIKM '13), 2013. http://www.mpi-inf.mpg.de/~pmiettin/papers/erdos13discovering.pdf -
Pauli Miettinen,
Sparse Boolean Matrix Factorizations.
Proc. 10th IEEE International Conference on Data Mining (ICDM2010), 2010, 935–940. 10.1109/ICDM.2010.93. http://dx.doi.org/10.1109/ICDM.2010.93
-
Pauli Miettinen,
Boolean Tensor Factorizations.
-
Ali Taylan Cemgil
Probabilistic Latent Tensor Factorisation, with applications to Audio Processing and Source separation
Algorithms for decompositions of matrices and tensors are of central importance in machine learning, signal processing and information retrieval. In the recent years tensor methods, that compute decompositions of multiway arrays have gained significant popularity (Kolda and Bader, 2009; Cichocki et. al. 2008). Notable extensions include coupled factorizations where multiple observed tensors are factorized collectively; such methods are in particular useful for information fusion.
We will discuss a subset of such tensor models from a statistical modelling perspective, building upon probabilistic generative models and generalised linear models (McCulloch and Nelder). Probabilistic interpretations of factorisation models facilitate the construction of application specific models. Here, the factorisation is implicit in a well-defined statistical model and factorisations can be computed via maximum likelihood.
We express a tensor factorisation model using a factor graph and the factor tensors are optimised iteratively. In each iteration, the update equation can be implemented by a message passing algorithm, reminiscent to variable elimination in a discrete graphical model. This setting provides a structured and efficient approach that enables very easy development of application specific custom models, as well as algorithms for coupled factorizations. Full Bayesian inference and model selection are also feasible via variational approximations or Markov Chain Monte Carlo (MCMC) methods. Well known models of multiway analysis such as Nonnegative Matrix Factorisation (NMF), Parafac, Tucker, and audio processing (Convolutive NMF, NMF2D, SF-SSNTF) appear as special cases and new models can easily be developed. We will illustrate the approach with applications in audio and music processing and informed source separation.
References
- U. Simsekli, Y. K. Yilmaz and A. T. Cemgil, “Learning the beta-Divergence in Tweedie Compound Poisson Matrix Factorization Models”, ICML, 2013.
- U. Simsekli and A. T. Cemgil, “Score Guided Musical Source Separation Using Generalized Coupled Tensor Factorization” in 20th European Signal Processing Conference (EUSIPCO), pp. 2639 - 2643, 2012.
- U. Simsekli, A. T. Cemgil and Yilmaz, Y. K., “Score Guided Audio Restoration via Generalised Coupled Tensor Factorisation” in International Conference on Acoustics Speech and Signal Processing ICASSP, pp. 5369 - 5372, 2012.
- K. Y. Yilmaz, A. T. Cemgil and U. Simsekli, “Generalized Coupled Tensor Factorization” in NIPS, 2011.
- Y. K. Yilmaz and A. T. Cemgil, “Algorithms for Probabilistic Latent Tensor Factorization” Signal Processing, vol. 92, no. 8, pp. 1853 - 1863, 2011.
- A. T. Cemgil, U. Simsekli and Y. C. Subakan, “Probabilistic Latent Tensor Factorization Framework for Audio Modeling” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics WASPAA '11, pp. 137-140, 2011.
Accepted Papers
-
Denis Krompaß, Maximilian Nickel, Xueyan Jiang, and Volker Tresp
Non-Negative Tensor Factorization with RESCAL
Non-negative data is generated by a broad selection of applications today, e.g in gene expression analysis or imaging. Many factorization techniques have been extended to account for this natural constraint and have become very popular due to their decomposition into interpretable latent factors. Generally relational data like protein interaction networks or social network data can also be seen as naturally non-negative. In this work, we extend the RESCAL tensor factorization, which has shown state-of-the-art results for multi-relational learning, to account for non-negativity by employing multiplicative update rules. We study the performance via these approaches on various benchmark datasets and show that a non-negativity constraint can be introduced by losing only little in terms of predictive quality in most of the cases but simultaneously increasing the sparsity of the factors significantly compared to the original RESCAL algorithm.
-
Roman Rosipal, Leonard J Trejo and Eran Zaidel
Atomic Decomposition of EEG for Mapping Cortical Activation
To improve the measurement and differentiation of normal and abnormal brain function we are developing new methods to decompose multichannel (electroencephalogram) EEG into elemental components or “atoms.” We estimate EEG atoms using multiway analysis, specifically parallel factor analysis or PARAFAC for modeling. Activation sequences of EEG atoms can identify functional brain networks dynamically, with much finer time resolution than fMRI. For example, EEG atoms activate in specific combinations during the sequential operations of brain networks, such as Default Mode, Somatomotor, Dorsal Attention and others. Guided by the score values of the identified atoms we inferred the volumetric brain sources of the selected networks using the sLORETA pseudoinverse algorithm. To confirm network identities, we compared 2-D and 3-D functional network maps derived from EEG atoms to known functional neuroanatomy of the networks. We find that multichannel EEGs in most individuals can be accounted for by a set of five to six standard atoms, which parallel classical EEG bands, and have unique power spectra, scalp and cortical topographies. We discuss how we may use the activation sequences of these atoms to describe the dynamic interplay of functional brain networks.
-
Wenjuan Gong, Michael Sapienza and Fabio Cuzzolin
Fisher Tensor Decomposition for Unconstrained Gait Recognition
This paper proposes a simplified Tucker decomposition of a tensor model for gait recognition from dense local spatiotemporal (S/T) features extracted from gait video sequences. Unlike silhouettes, local S/T features have displayed state-of-art performances on challenging action recognition testbeds, and have the potential to push gait ID towards real-world deployment. We adopt a Fisher representation of S/T features, rearranged as tensors. These tensors still contain redundant information, and are projected onto a lower dimensional space with tensor decomposition. The dimensions of the reduced tensor space can be automatically selected by keeping a proportion of the energy of the original tensor. Gait features can then be extracted from the reduced “core” tensor, and ranked according to how relevant each feature is for classification. We validate our method on the benchmark USF/INIST gait data set, showing performances in line with the best reported results.
-
Xueyan Jiang, Volker Tresp and Denis Krompass
A Logistic Additive Model for Relation Prediction in Multi-relational data
This paper introduces a new stepwise approach for predicting one specific binary relationship in a multi-relational setting. The approach includes a phase of initializing the components of a logistic ad- ditive model by matrix factorization and a phase of further optimizing the components with an additive restriction and the Bernoulli modelling assumption. By using low-rank approximations on a set of matrices de- rived from various interactions of the multi-relational data, the approach achieves data efficiency and exploits sparse matrix algebra. Experiments on three multi-relational datasets are conducted to validate the logistic additive approach.
-
Praneeth Vepakomma and Ahmed Elgammal
Embedding Super-Symmetric Tensors of Higher-Order Similarities of High-Dimensional Data
In this paper we propose an algorithm for non-linear embedding of affinity tensors obtained by measuring higher-order similarities between high-dimensional points. We achieve this by preserving the original triadic similarities using another triadic similarity function obtained by sum of squares of diadic similarities in a low-dimension. We show that this formulation reduces to solving for the nonlinear embedding of a graph which has a specific kind of a graph Laplacian. We provide an iterative algorithm for minimizing the loss, and also propose a simple linear-constraint that prevents non-zero solutions for embedding problems unlike the existing variants of quadratic orthonormality constraints used in the literature, that require eigen decompositions to solve for the embedding.
This workshop has been kindly supported by
