Evans Data Corporation just released a report titled “Application Servers 2008 Ranking”. You can get your hands on the free report here.
Evans asked over 700 developers to rate leading application servers on over 21 attributes including performance, security, DB connectivity, scalability and support. Respondents had to have experience with the app server in order to rate it. Eight products were considered: Apache Geronimo, ColdFusion, JBoss, Netweaver, Oracle WebLogic, Sun GlassFish, WebSphere Application Server and Windows Server.
Apache Geronimo came in at #2, the highest ranked open source application server by developers. This warms my heart since I used to be the product manager for WebSphere Application Server Community Edition (WAS CE). Many of you know that WAS CE is IBM’s distribution of Apache Geronimo. It’s been an uphill battle for the Geronimo community, but this is an achievement they should all savor.
Since we’ve discussed #2 in the rankings, let’s turn to numero uno. The #1 application server as ranked by developers is…drum roll please…WebSphere Application Server (WAS).
It’s one thing for WebSphere to be ranked highly by IT decision makers. This is an audience that IBM resonates with. But this survey is based on respondents who have actually used the products, namely developers. Yes, early versions of WAS (i.e. v3.5) were not the slickest or easiest to use. But we’ve focused on changing that perception over the past 5 years. The Evans study is proof that we’ve made significant progress. With IBM WAS version 7 released last week, I’m extremely confident that developers, and their peers on the operations team, will be even more impressed with WebSphere going forward.
I do not want this to be a ra-ra-IBM post. Many of you know that I don’t cover IBM news very much here. Additionally, readers will note that I usually write positive things about “my competitors”. I’ve always believed that strong competitors are a customer’s nirvana.
I do however, want to take a minute and repeat what I believe will be the most successful strategy for software vendors over the next decade. Providing customer choice that includes free/open source options.
A keen eye on developer and administer needs has been at the heart of WebSphere’s success. However, by itself, this would not have been enough. I strongly believe that our embrace of free/open source is the reason that WebSphere grew while other closed-source vendors had challenges against JBoss.
Offering Geronimo/WAS CE and the proven WAS family gave customers choice. Outgrowing the market is evidence that choice resonates with customers. The Evans study is icing on the cake that speaks to the quality of products that our customers have been choosing.
[UPDATE 2009-01-05: PLEASE see Eduardo's comment about the Evans not weighting the results by # of respondents. I think that is a mistake on the part of Evans from a pure market research standpoint].
10.03.08 at 8:30 am
[...] Comments [...]
10.05.08 at 8:24 pm
Hi,
I believe you may have missed the best application server on the market. Alpha 5 Platinum. Shame you didn’t include this in your comparison.
Mat
10.05.08 at 9:32 pm
Hi Mat, I’m simply quoting Evans Data Corp’s study. Not sure what Alpha 5 Platinum is, but you should contact Evans to maybe get that product in the next rev of the survey?
10.16.08 at 11:58 pm
Hi Savio.
I think the subtitle for the report is a more accurate description: a “user satisfaction survey”. Without more information about the respondents for each category, actual usage, etc, it is hard to infer much more than that. Taken to the extreme, a single, very happy Geronimo user could lead to a high satisfaction index.
– eduard/o
10.17.08 at 11:14 am
Savvio, the results are curious and don’t agree with other reports from top-tier analysts such as Forrester, Gartner, Burton, etc.
Any thoughts on why the results are so different.
Also can you confirm that the survey was completely independent – ie. wasn’t wholly or partially sponsored or influenced by any of the vendors in the survey.
More commentary on my blog[1] and on TSS [2].
[1] http://blog.softwhere.org/archives/409
[2] http://www.theserverside.com/news/thread.tss?thread_id=51008
- Rich
10.20.08 at 8:53 am
Hi Rich,
I checked with our Analyst Relations team about the report.
I can’t speak for any other vendor, but IBM did not sponsor the Evans study in any way.
Evans conducted a briefing with IBM on 2008-07-30. The discussion focused on the capabilities of WAS V7. You can see that the report makes reference to the improvements in WAS V7. Evans informed IBM that similar discussions would be held with other vendors. I cannot confirm if these discussions were used to frame the results of the user survey, or for another Evans report, or some other purpose.
I really can’t comment on difference in results vs. a study from another research vendor. It really comes down to the survey methodology. The only thing I will mention is no vendor, Garnter, IDC, Forrester, etc. has a better view of developers than Evans. The other vendors generally interview IT management. Evans has historically been developer focused. I’m sure you’ve seen the 10s of “Evans Developer Studies” that Evans publishes each year.
As an aside, we can both agree that BEA WebLogic was one of the revenue leaders, and hence, had decent usage penetration right? Well, in every piece of primary market research we conducted, we’ve had a really difficult time finding WLS users. I should note that we used a third party to conduct the research and we did not provide an IBM client list or anything like that. In fact, the over 5 different research vendors we used, each who used their own set of calling lists, always came back to us with an under sampling of WLS users. To this day, I cannot make sense of this vs. the revenue data from BEA.
10.20.08 at 9:00 am
Hi Eduardo,
>Taken to the extreme, a single, very happy Geronimo user could lead to a high satisfaction index.
I have to believe that Evans would weight the results by the # of respondents. That just makes good research sense. However, having said this, I haven’t seen any more details on the Evans methodology. If I do, I’ll post it here.
Cheers,
Savio
10.20.08 at 5:03 pm
Rich, one other point (that I missed the first time I read your comments).
I’m sure you’ve seen the various analyst reports that show IBM WebSphere as a leader…whether it’s the Gartner Magic Quadrant or the Forrester Wave or the Gartner and IDC market share publications for app servers.
In all of these highly tracked reports (by vendors and customers), IBM WebSphere does quite well ;-)
So, I’d say that the Evans report is consistent with “leadership” or market share reports from Gartner, IDC and Forrester.
10.26.08 at 7:34 pm
> I haven’t seen any more details on
> the Evans methodology. If I do, I’ll
> post it here.
Thanks, I’ll try to do the same. I’m still trying to find the “Enterprise” report.
> I have to believe that Evans would
> weight the results by the # of
> respondents.
Not necessarily. For example, as far as I can tell, the car satisfaction indexes from, say JD Power, do not weigh based on sales, and I agree with them. My main objection to the several articles that have commented on this EDC report is that they don’t make it clear it is just a satisfaction index.
I also wished that EDC provided more data on, say, the number of users rather than giving me the weight factor to two decimal points…
– eduard/o
12.15.08 at 7:55 pm
As promised, I’ve finally tracked more details of the report talking directly to EDC.
Savio writes…
> I have to believe that Evans would weight
> the results by the # of respondents. That
> just makes good research sense.
EDC confirmed that the # of respondents does not impact the result. Since I was also told that the # of respondents per AppServer is not available in the “Enterprise” report, I decided not to purchase it.
I know you are a Geronimo advocate and I’m sure it is a fine AppServer, but I never encounter it in the marketplace. I’m the first to point out the strong limitations of any sampling but see the Devoxx whiteboard on AppServers [1].
[1]http://www.devoxx.com/plugins/advanced/gallery-slideshow.action?imageNumber=7&pageId=655379&decorator=popup
– eduard/o
01.05.09 at 12:10 pm
Eduardo, thanks for the follow up. I agree that it’s a mistake for EDC not to weight the results by # of respondents. I have updated the original post to that fact.
Savio