Given the generic definition of 'business process', the overall workflow/BPM market ought to be huge, at least a few 10's of billions of dollars huge. But, it remains a very fragmented market with no one controlling a significant piece of the pie. One Aberdeen Group report in 2003 stated that IBM dominates the BPM market with about 16 percent market share, but I'd venture to guess it actually has less share than that, depending on how we define BPM.
There are no pure-play workflow/BPM vendors that are big. IBMs, BEAs and Oralces don't count as pure-plays since they offer comprehensive solutions to coporate IT departments, not stand-alone workflow products. Most workflow vendors address their own specific vertical markets with tons of add-ons. Most pure-play workflow startups end up changing their business models to offer industry-specific solutions, or going out of business.
Given the current state, will the BPM market be consolidated? What will lead to such consolidation? Will there be a couple of BPM players controllong 95% of the market? I tend to think it is not going to happen in the foreseeable future. The reasons are many, but the primary reason lies in the fact that 'BPM' is not all about technology. No vendor today can sell a BPM product to a customer and walk away expecting the customer to implement a meaningful solution to his/her business problem. Given a BPM system, most customer do not know how to convert a business problem to a technical BPM solution. Vendors end up needing to know too much about the domain in order to help their customers implement meaningful solutions, often requiring them to add domain-specific extensions. No vendor can afford to know enough about all the domains or even half of them.
It is conceivable that, at some point in the future, there may be one or two major vendors offering the 'core' BPM technology, with many middle tier vendors offering domain specific extensions/solutions. The emerging standards such as BPEL may help such consolidation, when the market truly accepts them as standards. Some version of BPEL in the future, may be BPEL 4.0, may become such a standard, but there is a long way between BPEL 1.1 and BPEL 4.0. Or may be it is an impossible way.
It will be interesting to watch how the BPM market evolves with all the major players (Microsoft, IBM, Oracle, BEA and others) vying for a big share of the market.