Professor Schommer studied Computer Science at the
University of Saarbrücken and at the German Research Centre for
Artificial Intelligence. In 2000, he received his PhD degree from the
Goethe-University of Frankfurt/Main.
From 1997 until 2003, Professor Schommer worked for IBM Research and Development as an IT Architect in several Business Intelligence projects worldwide. Since 2003, he is an Associate Professor at the University of Luxembourg and an international expert for Data Science.
Q1. Is domain knowledge necessary for a data scientist?
Absolutely yes! And this is, for example, for situations where additional (potentially unknown) data needs to be added, where seasonal effects emerge, and where decisions have to be taken regarding the preparation of the data in general. Also, knowledge about cultural differences is strictly important, for example, concerning applied color schemes in visualisations.
Q2. What should every data scientist know about machine learning?
I, personally, see Machine Learning as an attractor of a data life
cycle and the heart of each data science process. All this data, whether
it is in a raw or in a prepared state, will finally end there some
time. From that point of view, I believe that each data scientist should
not only be a specialist in the own field but should have a level of
armamentarium that enables to put oneself in a Machine Learner’s place.
Moreover, I believe as well that each data scientists, who works with the results of Machine Learning processes, should brace oneself for a comprehension.
Q3. What are the most effective machine learning algorithms?
Simplicity, comprehensibleness, and powerfulness are – in my eyes – the most conclusive arguments for an effectiveness. For that reason, the apriori algorithm (Association Discovery), C4.5 (Decision Tree; Classification), and k-means (Clustering) are to me the most effective algorithms. This is also confirmed by a voting among experts, which was published during an ICDM 2006 panel session (organised by Xindong Wu and Vipin Kumar): here, C4.5, k-means and apriori were ranked under the Top4 (together with SVM, which appeared on the third position).
Q4. What is your experience with data blending? (*)
It is a very tedious, but responsible, process. Particularly, because this kind of data fusioning takes place before the analytics takes place, but after a construction of a (potentially) stable data architecture is made. Working with data blending, therefore, means to me to risk a re-building and a changing of a running data system on the one side, but to ensure a data quality with respect to a further processing on the other. To work on such a bridge does not seem to be an easy task.
Q5. Predictive Modeling: How can you perform accurate feature engineering/extraction?
A certain number of factors should be implemented: first, a
predictive modeling should not be performed alone but in a team, which
itself should be composed of experts of different fields (domain
experts, statisticians, data engineers, machine learning experts).
Second: it should be clear that predictive patterns do not necessarily
justify a causality. It is wise to critically check gained results and
to involve domain experts (see question 1). Third, a developped
predictive model is not necessarily
the best one. Instead, alternatives should be developed and tested under different conditions.
Q6. Can data ingestion be automated?
I do not think that an automatization will be complete, ranging from collecting until taking decisions, which data can/should be stored and which data can/should be removed. But I believe that – particularly in the age of big data/texts – that a symbiosis of a human (data) care, a high-performance computing, and the right use of AI-related inventions (e.g., robots, self-healing) may become highly effective.
Q7. How do you ensure data quality?
To keep a data quality is mostly an adaptive process, for example,
because provisions of national law may change or because the analytical
aims and purposes of the data owner may vary. Therefore, the ensuring of
a data quality should be performed regularly, it should be consistent
with the law (data privacy aspects and others), and should be commonly
a team of experts of different education levels (e.g., data engineers, lawyers, computer scientists, mathematicians).
Q8. When data mining is a suitable technique to use and when not?
The application of data mining in a serious way requires a sufficient
amount of data. To remind, the idea of exploring data exists since
hundreds, even thousands, of years! It is dangerous and wrong to reason
and to interpret a causality if a solid data quantity is missing. A
second aspect is that Data Mining does not follow a straightforward
standard procedure but still follows a data-driven principle: from that
perspective, each data mining-project is new in itself and requires an
Also, it should also reminded that “if there is nothing in the data, then nothing can be found”. Even this last point should be respected. Apart from that, Data Mining is one of several directions in the Data Life Cycle: analytical results should always be verified with other methods.
Q9. How do you evaluate if the insight you obtain from data analytics is “correct” or “good” or “relevant” to the problem domain?
In my understanding, an insight is already a valuable/evaluated
information, which has been received after a detailed interpretation and
which can be used for any kind of follow-up activities, for example to
relocate the merchandise or to deeper dig in clusters showing a
However, it is less oportune to rely only on statistical values: an association rule, which shows a conditional probability of, e.g., 90% or more, may be an “insight”, but if the right-hand side of the rule refers to a plastic bag only (which is to be paid (3 cents), at least in Luxembourg), the discovered pattern might be uninteresting.
Q10. What were your most successful data projects? And why?
The most succesful projects, where I have been involved in, are
certainly those, where an effect could be seen immediately after having
taken some kind of reaction. In this regard, I remember a business
project regarding the detection of fraud in telco data as well as a
diverse number of Market Basket Analysis-projects, where the customers’
behavior and profiling
patterns have been used to improve a customer satisfaction.
Q11. What are the typical mistakes done when analyzing data for a large scale data project? Can they be avoided in practice?
I believe that missing expertises, an inappropriate communication
among the team members, and the favorising of quick-and-dirty solutions
are serious problems. Personally, I am not a friend of sampling. The
reason is that interesting data patterns may disappear and that a subset
of the data does not necessarily reflect the overall data structure.
Also, statistical values should not have the final word (see question
10) and should not be the only reason for an insight. The analysis of
is also a multi-disciplinary and multi-cultural concern and should performed correspondingly.
Q12. What are the ethical issues that a data scientist must always consider?
Because of the masses of data sensors – that emerge day by day and
that affect a data loading and a further use of data likewise -, each
data scientist bears a kind of responsibility in terms of, e.g., data
correctness, data privacy, and
data availability. As a central ethical issue, this (new) burden of work should be accepted; each data scientist should be aware of that. Data Science should also be sensed as a chance to do something good and meaningful.