It started as a small hole in the office floor.
No one really paid any attention to it.
The next day there were a few chairs set up around it, so no one would stumble over it and hurt themselves.
A small notice read: “Apologies for the inconvenience”.
As the hole grew bigger and deeper, the noise from the hole grew louder and louder.
It’s being dealt with, we were told.
On the way out of the office one evening, curiosity got the best of me.
I walked over to the hole and peeked in. It was already quite deep.
Through the dirt and the dust I could see someone working inside.
Hello! I called out. What are you doing down there?
The worker put down his pickaxe and looked up.
Me?, he replied. I’m process mining.
You’re what? I asked.
Process mining, he explained, I’m discovering and improving processes by extracting knowledge from the event logs.
Process mining enables you get a process model from the data. This way, the real process, and actual business rules, can be discovered.
Can’t you start digging somewhere else and tunnel under the office, maybe from the IT department, I asked.
Oh no, he replied. You start digging where the business guys sit, and just keep on digging down until you hit something.
When do you expect to finish? I questioned.
Oh, it will probably take months, he replied. These things do take time.
The next day, I raised the issue of the miner with the department manager.
Who authorized the mining? I asked.
It was probably approved by management, he replied scratching his head, but I’ll check.
After the weekend, we needed to relocate the finance department to another part of the office. The office looked like it had been hit by a bomb, dust flying everywhere.
The hole was gigantic, the noise deafening. It seems the pickaxe had been replaced with a hydraulic drill.
As weeks went by we slowly got used to the noise and dust from the mining shaft.
There were ideas about turning it into a swimming pool at the end of the project.
Some employees were running bets on when the miner would hit earth’s core.
From time to time, the miner would stop digging and there would be a brief period of silence whilst he analysed the data.
The CTO and CIO were always arguing about who was responsible for this enormous mining project in the middle of the office.
The CTO claimed that discovering, monitoring and improving processes was the responsibility of the CIO therefore all complaints about the mining should go to him.
The CIO claimed that BPM projects were run by the CTO, therefore his responsibility.
In the end they agreed that audit trails and transaction logs fall under the responsibility of the IT Director, so he was to blame.
Communication with the miner has been lost. All attempts to stop him have failed.
We have moved into the next door office as our previous office space is now a big hole. There is less noise now, although the lava eruptions from the hole are annoying.
On the positive side, it looks like we will be opening a new office in China…
Adam Deane 
Hi Adam,
Great post although i believe that process mining is a more Anglo-saxon topic then an European one as there is a difference in culture see my post on the ACM group in Linkedin http://www.linkedin.com/groups?home=&gid=2452802&trk=anet_ug_hm
Greetings Freddie
By: freddievanrijswijk on 12/12/2011
at 10:19 am
Hi Freddie,
The idea for my story this week was from the new process mining manifesto publication that has been released recently.
Most the names of the contributors there were European..
Globalisation blurs the fences between countries and cultures.
I don’t think the gap in understanding the ACM concepts can be put down to consequentialism. There are more reasons than just that…
Embracing ACM doesn’t mean the customer should not use partly defined processes, or structure some of their business rules.
If a customer wants to embrace the ACM approach, but currently doesn’t have any of their processes defined, and they would like to structure them better, or create business rules to reduce loops etc, without loosing the ACM benefits then process discovery would be beneficial (expensive.. might be able to be done another way… but still beneficial)
Process mining and process discovery would actually bring more value to ACM than BPM.
Cheers,
Adam
By: Adam Deane on 12/12/2011
at 3:02 pm
Hi Adam, SUPERCOOL! As I said, ‘process mining’ is in my view an oxymoron because you can mine anything but not strictly defined process flows … the best question: ‘When are you finished mining?’ Regards, Max
PS: I know they mean ‘mining FOR processes’ but I see it of little value as it misses the rationale, which you could only get from the data context. In fact you can only get it from the performer knowledge, which is why our User-trained Agent ‘mines user actions’ but not for processes, but for data patterns. When these data patterns link up … presto, you have a process. But as in the real world, you couldn’t show it upfront in a flowchart.
By: Max J. Pucher on 12/12/2011
at 10:19 am
Hi Max,
I think they mean process discovery…
The advantage of process mining, as claimed by the vendors is basing the objective analysis of real data, not the subjective views of different stakeholders (each 2 stakeholders have 3 views on how the process currently runs and 4 views on how it should run..)
End users also have a subjective view on how the process runs and how it should run..
The value of process mining/discovery is building the process model based on objective data, not the subjective views of humans.
Cheers,
Adam
By: Adam Deane on 12/12/2011
at 3:10 pm
Hi Adam, there is no such thing as objectivity. First the mining was defined to use data collected by someone who made a subjective judgement (hrrrm, guess?). Second, the data then have to be subjectively interpreted as to what they actually mean. Third, when people interact in a certain manner using their autonomy I can virtually guarantee you that this is not how they want to work if they are told by a BPM solution to do so!!!!
By: Max J. Pucher on 15/12/2011
at 9:35 am
Hi Adam:
I do not agree with your view this time, thus let me be the narrator of this wonderful tale:
Once upon a time a very big an intelligent had some problems in a very important business process. It were identified by a ultra intelligent real time dashboards.
This company had very agile and process awareness people. They were recognized by the market a very smart company leading business transformation.
Process improvement belongs to their roots and was part of its culture.
Immediately, the improvement team start digging to find the cause of the problem, but it was taking to much time and effort and they figured it out it was better to hire consultants. First came the lean experts, they here saying the problem as related of poor normalization, but the problem persisted, then the six sigma jumped because they prove the problem was about variation, after 6 months another team of experts chimed in, they were the reengineering gurus and flipped the process upside down. Still the process was performing poorly and the management team decided to go for intelligent automation with best of breed business rules, ontology, semantics.
After one year of constant failure the rating agencies put the company in negative outlook. Shares plummeted.
On the other side of town a miner was reading the news and thought this was a good opportunity to introduce Process Mining.
We knocked at the company door.
“Who is this?” (Said a person that worked in the company)
“I’m a Miner. I provide Process Mining services that can improve business processes based on facts, not in assumptions.”
“You are a blue collar Miner? We are not interested on what you are trying to sell. That is not strategic. That’ some kind of operational technology. We are a very big and intelligent company and we only buy strategic things! We are on negative outlook, but is rating agencies fault. We are a very strong company!”
The Miner did not felt offended and asked: “But can you give me an opportunity”?
Go on, you are going to waste your time, said the other person.
Then the miner started working looked to the data were no person could (or wanted to) see the causes of bad performance, got the facts and presented it immediately where the problems were, why the company was under performer. The managers couldn’t believe the speed and accuracy. They were used to weeks of work just to figure it out something and most of times they were only finding 10% of the problems.
The process was improved quickly in the right direction, customers were happy and shares price grow.
By: Alberto Manuel on 12/12/2011
at 2:24 pm
Hi Alberto,
Loved the Blue collar miner story… but…
As I understand, process mining is about creating the process model from the data.
If I followed your story correctly, it was about improving the process by analysing the real data. But isn’t that BAM and regular reporting?
Looking at the data, and finding the problems, bottlenecks and improvements is process analysis through the reporting module.
I might be mistaken, but the objective of process mining is creating the process model from the data.
Cheers,
Adam
By: Adam Deane on 12/12/2011
at 2:39 pm
Hi Adam:
Process Mining can be used to discover a process model automatically with no human intervention. But this is only one of of the areas you can use process mining. And still you do not need to discover a full process model to make an analysis. You can for example concentrate on one particular activity and figure it out why there are so many waste (it’s the execution time? is the data they access that is difficult to get? is a specific role or person unable to perform?)
Second process mining can help with conformity analysis. Is a business rule being followed or not? Are we compliant with SOX or our socks are red? Is there any fraud risk inside of our company and among our partners on the business transactions?
Third brings a performance lens. Dashboards are great to show performance KPI but does not show why the company is missing the targets.
Last for enhancement, meaning you can cross relate data in order to understand how people interact, like a social network or other any kind of analysis dimension (time, cost …)
Least, Process Mining can identify patterns around knowledge work including many routine activities that don’t involve judgment or more important identifying expertise around the workplace: substance, sequence, timing, and outcomes achieved. Does not matter the argument that next time people execute differently. This is so wrong and shows clearly that people never experienced Process Mining, you know that you can understand reasoning behavior, simply for the fact that it leaves a trace on IT systems?
Regards
Alberto
By: Alberto Manuel on 12/12/2011
at 4:35 pm
Hi Alberto,
You clearly illustrate that Consequentialists believe there is a direct relationship between deed and the outcome and having analysed the facts (the past) its just a matter of fine-tuning the “how” as a business would be not part of a complex world but is a machine in a controlled lab environment. The result of the analysis will be that more data has to be collected hence more beaurocracy because the more we know the more we realise we dont know enough in our search for the One Best Way.
I am not saying that analysing data is a waste of time, but as non-consequentialist (european) I believe that there is more to take into consideration and that there should be more focus on the what then the detailed how and that trust, Intention and the goals you provide is more important hence the difference between top-down command & control versus bottum-up guidance and support, european believe there are more roads that lead to Rome 😉
By: freddievanrijswijk on 12/12/2011
at 3:09 pm
Hi Freedie:
The reason I stop using classic approaches is they take to much time (more than one week to be precise) to deliver results.
Regards
Alberto
By: Alberto Manuel on 12/12/2011
at 4:12 pm
George Varvaressos added this comment on LinkedIn:
• The attached blog makes reference to the recently produced Process Mining Manifesto. I would recommend this document to anyone interested in process mining.The appropriate links are listed at the bottom of this comment.
The blog has some comments that should be clarified;
One is that “process mining is a more Anglo-saxon topic then an European one’. The Process Mining Task Force has produced the manifesto in ten languages, all contributed by the authors; English, Spanish, German, French, Chinese, Italian, Dutch, Korean, Portuguese, and Greek. If you look at the Process Mining group the interest is global and the background of the members is diverse.
Another comment was ‘Process mining and process discovery would actually bring more value to ACM than BPM’. Process mining is an important part of BPM. It is really a bottom up approach that sits well with existing Process Frameworks like SCOR and APQC which are top down. It’s a different perspective.
BPTrends rates Process Mining Low in Hype and High in Value. http://www.bptrends.com/publicationfiles/advisor20110726.pdf
Process Mining certainly isn’t BAM. BAM will not show you bottlenecks, business rules, conformance and organisational issues. It certainly won’t show you what the process actually looks like!
Also process mining is not just process discovery. To quote from the first page of the manifesto, ‘Process mining is a relatively young research discipline that sits between
computational intelligence and data mining on the one hand, and process modeling and analysis on the other hand. The idea of process mining is to discover, monitor and improve real processes (i.e., not assumed processes) by extracting knowledge from event logs readily available in today’s (information) systems. Process mining includes (automated) process discovery (i.e., extracting process models from an event log), conformance checking (i.e.,monitoring deviations by comparing model and log), social network/organizational mining, automated construction of simulation models,model extension, model repair, case prediction, and history-based recommendations.’
There is a lot of discussion recently about Advanced Analytics and Big Data. It has become a huge industry. They seem to ignore process mining. The interesting point is that all business data is created by business processes.
My point is that if you want to understand (big) business data then you also need to understand and analyse the processes that created it.
References:
IEEE Task Force on Process Mining: http://www.win.tue.nl/ieeetfpm/
* Manifesto: http://www.win.tue.nl/ieeetfpm/doku.php?id=shared:process_mining_manifesto
* Direct link to PDF of Manifesto: http://www.win.tue.nl/ieeetfpm/lib/exe/fetch.php?media=shared:process_mining_manifesto-small.pdf (788 KB)
* Three-fold flyer introducing the Manifesto: http://www.win.tue.nl/ieeetfpm/lib/exe/fetch.php?media=shared:process_mining_manifesto_flyera4.pdf (242 KB)
* High-resolution PDF: http://www.win.tue.nl/ieeetfpm/downloads/Process%20Mining%20Manifesto.pdf (9,639 KB)
By: Adam Deane on 13/12/2011
at 10:03 pm
I must admit that I do not really understand the original post. The metaphor could as well apply to traditional data mining approaches. In our experience (having applied it in over 100 organizations), process mining is not intrusive and can be done is a relatively short time (compared to modeling). Data extraction can often be done in half a day and the computation time of most process mining algorithms is very short compared to more traditional approaches such as data mining and simulation. Most time is spent on discussing the insights.
I propose to use the metaphor of making a map. Process mining will provide an organization with maps showing how processes are really executed. How detailed the map should be and what aspects it should cover depends on the purpose.
Some people may argue that there is no need to produce a map if you already know the real processes. This is true, but (a) how many organizations know their real processes and (b) the coupling of event logs to discovered process models allows for the projection of additional information based on facts (e.g., bottlenecks).
Other people may argue that there is no point in making maps as processes changes all the time (we call this concept drift). In my view, such changes make process mining even more important as this will allow organizations to closely monitor these changes. This is must better that producing piles of Visio diagrams that are not maintained.
Some people may argue that maps are uninteresting as they would not like to “Pave the cowpath”, i.e., it is not relevant where the roads are now, but one should focus on where the new roads should be. I agree that the focus of process mining should be on process improvement. However, a reengineer that does not know the cowpaths in the organization will probably create a big mess.
I can assure you; very few cowpaths bring you to China via the center of the earth!
By: Wil van der Aalst on 14/12/2011
at 6:31 pm
Hi Wil,
Thanks for your response.
The blog was my way of adding a little satire to the discussion around process mining.
Process mining has great value when used in the right situation and the right context.
But if process mining was as quick and simple as you describe it – everyone would be doing it.
Drinking coffee – 2 minutes, Data extraction – 2 hours, Understanding the insights – 2 months…
Your example of a map is a good example to contradict.
If you open Google Maps to plan the best route between two points in London:
* If the resolution is too high, all you see is a ton of roads.
* If the resolution is too low, all you see is a house.
* Not taking the rush hour traffic into consideration will cause you to be late.
* Taking the road through congestion charge zone will be expensive
Knowing how to find the best route in a map – is a skill, (and not a simple skill)
Process Mining without conclusive recommendations is also a waste.
If you have been searching Google maps for that route that I asked for, and after an hour you come back to me and said:
“There are 6 routes you can use” – useless, give me one usable recommendation.
“59% of the people turn left on Regent street” – useless, should I follow them or go the other way
“There are roadworks on Oxford street” – useless, don’t tell the problems, give me a solution.
Process mining has great value when used in the right situation and the right context.
Cheers,
Adam
By: Adam Deane on 14/12/2011
at 10:58 pm
Adam, why don’t you post this latest response on the BPTrends Discussion that you started. I’m sure the group would be interested in discussing this.
By: George on 14/12/2011
at 11:31 pm
[…] same week I started making public the case study (if you want a copy send an e-mail) a lot of noise part 1 and part 2 appeared around process mining. Good I thought. A lot of discussion, fear, anger and […]
By: A Process Mining Project « End to End BPM on 18/12/2011
at 11:56 pm
It’s nice to see such a vivid discussion here!
I’m Antti Manninen from QPR Software. As Adam noted in his other blog posting, QPR has developed a tool called QPR ProcessAnalyzer. I’d like to share with you some of our ideas and insights we’ve encountered developing the software and consulting customers.
I’d like to have our say about the methodology name. Firstly, we go by the analyst definition, ‘Automated Business Process Discovery’, as used by Gartner and Bloor Research. Secondly, we don’t want to hijack an entire academic research subject. A business application area can never cover the entire scope of an academic research field, especially when it comes to ontologies. It would be a pretty bold statement for any vendor or service provider to say they have it all covered! Rather than hijacking academic terminology, we think it’s better to have a business word. We see that ABPD is to process mining as BI is to statistics.
Finally, there is also a very pragmatic reason for the naming convention: for many BPM people, ‘process discovery’ means drawing existing processes based on information provided by the organization. (Usually the information comes from interviews, workshops, and direct observation). There’s a nice article by Sandeep Jadhav on BPTrends about the differences between manual and automatic discovery methods: http://www.bptrends.com/publicationfiles/THREE%2002-01-2011-ART-Business-Process-Discovery-Jadhav.pdf.
“Business Process Discovery can be done using two techniques.
o Manual – Process Analysts using interviewing techniques to derive the processes.
o Automated – Process Analysts use ABPD tools that can extract the process from
databases or logs available with IT.”
But so much for definitions. There is one very vital difference between ABPD and manual methods. It is this: Unlike traditional business process mapping, the end result of ABPD is not a static drawing. Instead, you get a dynamic model that allows for adjusting the level of detail. Also, certain cases or entire parts of a process can be excluded or focused on. You can bring in as much background data as you want to enrich the analysis. Data drilldown is supported – or should I say, encouraged! You’re able to ‘move around’ in the universe provided by the data and take several viewpoints in order to prove or disprove your hunches, guesses, thoughts, and prior knowledge.
Making the analysis becomes an iterative process that starts with an initial analysis and continues with a number of subsequent analyses. Sometimes just a visualization of the overall process is enough to start with. In many cases, problems or areas of improvement are blatantly obvious. When shown the factual process, we’ve experienced that customers usually ask questions like ‘Why are there loops in the process?’, ‘Why are process steps executed in the wrong order?’, and ‘Why are the transitions times so long?’. These are valid questions to answer with further analysis.
Our experience from customer projects show that it’s just as easy to get lost in the actual process with a complex SAP Order-to-Cash processes that it is to do so with the sales process in Microsoft Dynamics CRM. The simplest of processes can turn out to be a zoo of variations.
Then a few words about objectivity. In many ways, traditional process discovery reminds me of the scene from Hamlet where Hamlet and Polonius are gazing at an odd-shaped cloud in the sky:
HAMLET
Do you see yonder cloud that’s almost in shape of a camel?
POLONIUS
By th’ mass, and ’tis like a camel indeed.
HAMLET
Methinks it is like a weasel.
POLONIUS
It is backed like a weasel.
HAMLET
Or like a whale.
POLONIUS
Very like a whale.
So, which animal is it if it’s just up to your own opinions to decide?
Without getting philosophical here, it’s debatable whether anything can ever be 100% objective. But, there is a big difference between subjective opinions and fact-based subjective opinions. ABPD provides a fact-based view of how processes have been executed in reality and leaves any interpretation up to humans. Manual methods provide an opinion-based view of processes that leave any interpretation up to humans. If we’ve been able to automate the first part, we’ve certainly come a long way.
By: Antti Manninen on 21/12/2011
at 6:45 am
Antti, Fantastic response (I must do a post on BPM and Shakespeare…)
I agree. There was a lively debate on the topic. Process Mining usually doesn’t come up in BPM community discussions, so it was an interesting topic to debate, and it brought the subject to the knowledge of those that haven’t used it before.
Cheers,
Adam
By: Adam Deane on 22/12/2011
at 11:26 pm