Wednesday, August 25, 2010

Rankings

Subsequent to my post on Shanghai Rankings, a commenter enquired, "Is the low ranking of Indian institutes due to lack of Nobel Prize winners, Science/Nature papers etc? Would the ranking be significantly higher if one considers only citations or papers?" Some of my colleagues asked whether it is possible to rank Indian institutions by considering only citations and/or papers. Well, there is the essential science indicators (ESI) ranking by the ISI web of knowledge. Please note that the rankings are based purely on the publications and the citations obtained. This is for a 10 year window and this is updated every quarter. Here is the ranking of IISc in different fields:

Chemistry - 136
Physics -311
Mat. Sci. -73
Biology and biochem: -387
Eng. -175
Geosci - 404
Comp. Sci - 128
All fields: 457

ESI also ranks scientists and the top 1% of scientists find place in this list. How many scientists from IISc figure in this list? Well, it is as follows:

Chemistry - 8
Biology - 1
Physics - 1
Geosciences - 1
Engineering - 3
Material Science - 2


A typical response that these do not mean anything and citations and publications do not count for much. Only in India is a scientist who does not publish (and therefore does not get cited) called scholarly. In any case, the data is presented and please make your own interpretations.

43 comments:

Kaustubh said...

May I please have a link to the ESI rankings page?

Anonymous said...

if you consider citations per paper as the yardstick for ranking then ranking in engineering detoriates further

Giri@iisc said...

Kaustubh,

I do not know whether you institution subscribes to ESI. Go to isiknowledge.com, click under additional resources, click under institutions and search for "indian inst sci"

Thanks

Giridhar

Anonymous said...

Prof. Giri,
Could you please also post the ESI ranking of IITs?
Thanks.
SN

Anonymous said...

I am not sure if they take into account in which institution the research prize's winners did most of their work, or if they count their current affiliation.

That would explain why UCSB is on the 15th place in Engineering. They have many senior researchers which moved there after/close to retirement, including Nobel Prize winners.

Overall this ranking seems ok. I was actually glad to see very good places acknowledged in this ranking.

We had some similar discussion (a year ago?) in this blog about rankings. Everybody seems to agree that there is a need for some kind of measuring of the research quality to be used for grants, fellowships, promotions, etc. However, the problem still remains; what is best way/formula to evaluate it?

M.

Anonymous said...

"A typical response is that..."

Another typical response is that Feynman published only 37 papers.

Giri@iisc said...

But Feynman had 20,000 citations. If an Indian scientist has 20,000 citations (irrespective of the number of papers he/she has published), he/she needs to be applauded. Low number of publications does not necessarily mean high quality or vice-versa.

Thanks

Giridhar

Anonymous said...

The ICM had a eminent panel today to discuss metrics; the problems with ISI/ESI were highlighted (actually they were roundly trashed). There were all sort of problems from irrational 2 year window to unscrupulous editors, etc.

One of them spoke eloquently of inverse-McCarthyism; instead of guilt by association, it become praise by association (to a journal).

There is a need for some measure; maybe some better version of the currently flawed metrics. But these metrics should still be taken with a pinch of salt. Over reliance on any of these indices will spell disasters.

I did not know that Feynman had 20,000 citations and I don't think he was great because he had 20,000 citations. And I see no reason to applaud any Indian with 20K citations unless this person did something to change the direction and perspective of the people working in that field.

A good researcher has a large citation does not mean large citation implies good researcher.

Anonymous said...

yes, do not applaud any indians who has 20,000 citations after all there is only one such indian.

So, you do not mind that the majority of professors in IITs have less than 20 papers in their lifetime. they tell exactly what you say

Anonymous said...

Point is a person with over 20 K citations is likely to be a very good researcher.
I agree that there may not be a perfect correlation between citations and the stature/impact of work but it is unlikely that a researcher will garner high citations if his/her work is not good. Now one can always argue that good work may not mean breakthrough work which is true.
In a similar note persons who have published less and has very low citations obviously can not be termed as productive/good researcher.
My conclusion is citations are a good yardstick and reasonable measure of the quality of a researcher. It may have its own flaws and it may be biased and in some cases error prone but on an average over a psna of 30 years (life time of a researcher) it tends to give a good indication of one's achievements. I also agree that high citations cannot be equated to breakthrough works (like feynman's). However breakthroughs of such magnitude require geniuses and general researchers like us do not fall in that category. So we as research community try to advance knowledge in small steps which while not breakthrough cannot termed immaterial either. So people like us citations is a good indicator of our impact.

Giri@iisc said...

I have said this many times before. Irrespective of whether publications or citation measurement are useful for assessing individuals, it is a good tool to assess big groups such as universities and countries.

In India, we have a fashion of quoting outliers to trash the system.

Thanks

Giridhar

Anonymous said...

"Irrespective of whether publications or citation measurement are useful for assessing individuals, it is a good tool to assess big groups such as universities and countries."

If this is the stated position, then what is this top 1% of scientists doing in the post?

The danger is not with outlier (and/or their generalizations to all Indians), but somehow believing that one dimensional numbers can capture any significant information about the tail of any distribution. Citations are no good when looking at any university beyond 50 (ie., small); after that the metrics are no longer robust and wild change in rankings will occur on small pertubations in metrics.

Giri@iisc said...

"If this is the stated position, then what is this top 1% of scientists doing in the post?"

Because the shanghai rankings include them in the ranking of universities. The ranking of universities depend on how many scientists figure in the top 1% and top 1/2%.

"Citations are no good when looking at any university beyond 50 (ie., small); after that the metrics are no longer robust and wild change in rankings will occur on small pertubations in metrics."

IF a small "pertubation" [sic] is all that is required, then why cannot Indian institutions figure in the top 100? Maybe we can perturb the metric in just one year, find a place in the top 50 and go back to our old ways in the next year. Please look at the number of citations obtained by an university ranked 51 and IISc and you will know the difference. Post the difference here and let there be a healthy debate whether it is small.

Thanks

Giridhar

Giri@iisc said...

"A good researcher has a large citation does not mean large citation implies good researcher."

Fine. I have no problem with this statement. This, however, implies that good researchers need to have large number of citations.

Thanks

Giridhar

Ankur Kulkarni said...

Why are we arguing over metrics when the rest of the world has, albeit grudging, accepted them? What's wrong with accepting them as just another non-ideal part of life and excelling at them nevertheless?

Anonymous said...

As many others have already pointed out, just because there are some outliers who are doing good research but may not satisfy certain publication metrics does not mean that we should abandon all such metrics. Instead enough provision and flexibility should exist (I am sure that is very hard) to recognize good researchers who are these outliers.

On the other hand, within the Indian context where tenure, salary and promotions are not dependent on research quality/quantity/metrics, I am not sure where these discussions will lead to, unless one is thinking of changing the current system itself. Any thoughts by Prof Madras on this?

Anonymous said...

You are right in that promotions do not depend on research quality or output. But don't you think it is unfortunate?

Even in Institutes of national importance like IIT-Bombay, recently, a faculty was promoted from assistant professor to associate professor with ZERO publications while someone else in the same department (chemical engineering) was denied saying he had only 5 publications.

Even assuming all 5 publications were of low quality, isn't it better than zero publications? Apparently not, because the person who had not published any paper is similar to Frederick Banting (who discovered insulin, got a nobel prize) but published only 11 papers in his lifetime.

So, why this discussion about rankings and publications? Because people who actively publish should not be discouraged. If a person publishes 5 papers a year, he is automatically branded as doing poor quality research while a person publishing 1-2 papers per decade (which never gets cited) is called scholarly.

Anonymous said...

At the end of the day, at one of the following is necessary for a researcher to be considered worth his salt: either good research contributions or good metrics. There is no evidence of a good scientist had neither.

Anonymous said...

But what is good research contributions? How many times have we heard that the scholar has published only 1-2 papers in a decade but those papers should have been published in science or nature. But the researcher did not send it there because he wanted to support indian journals. It did not get cited because it is a Indian journal that no one reads and is not even available online. But the paper is a very important research contribution. The only problem is that no one has read it (other than the scholar himself, hopefully)

Anonymous said...

For any research contribution to be considered good, its goodness should firstly be verifiable. Goodness has to be assessed objectively, not subjectively by the author alone. And therefore it is the responsibility of the author to publish his work in a medium that gives it the widest possible chance for critique.

Secondly the goodness of the work has to be certified by the scientific community. The author should be able to produce evidence of goodness either in terms of statistics such citations etc, or in terms of opinions of reviewers and leading people of the field. To claim that a contribution is "good" without any is simply unscientific.

I find this entire debate that tries to assess the quality of a researcher without using any credible evidence of it, tending to the absurd.

Anonymous said...

Giri, the anon you replied to.

You seem to be making inferences that was not intended! For example, I did not say that somehow IISc deserves to be at position 51. I was merely pointing out to the shortcomings of the metric.

Does IISc have deadwood professors? Plenty. There are too many mediocre here. Do I have any sympathy for the "scholars" at IISc and IIT with 10 publications and a citation index of 100? None at all.

However, this has nothing to do with the fact that the metrics are naive. Here is a simple example: you have a h-index of 27 (from your site). This accounts for 27*27 = 729 citations. The total number of citations you have is 3068. So the h-index "explains" less than 25% of your citation. This also means that only 1/8 of your papers have a citation more than 27 (216 total papers). Now, it is likely that in your area, 1/8 is doing much much better than an average; which bring one to the culture of different areas/sub areas as well. None of these metrics are "normalized" in any way.

Btw, just for the record, I am your average scientist/engineer with 20 odd publications, an average citation of about 40, and max citation near 250. As you can see, we come from vastly different cultures and mindset.

I looked at my numbers only yesterday, and checked a few other people in the dept etc and was surprised how favourably I stood in comparison! Based on what I saw, I agree with you on this: IISc/IIT can do much much much better.

Based on this 24hr experience, my preliminary view of a "good scientist" would be:
1) max citation of about 500 or above, (breakthrough; this number is again area dependent)
2) average citation of about 20, (consistent)
3) a total citation of over 3000 (long innings)

Vijay Kumar (ECE/IISc) has a paper with 630 citations and that paper was a breakthrough. Manindra Agrawal has a similar max citation number.

Balaram has a paper at 500. CNR Rao has a few at 500 and one at 1000. GNR has 2 over 1000 and 2 near 500! But I am not qualified to judge their merit. It may be interesting to know of other examples-- say 20 -- along with the experts in the area saying why they are breakthroughs.

Are better metrics possible which can capture breakthrough/consistent/long-innings? I hope so. By the way, just for fun, I calculated the h-index on Tendulkar, Dravid, and Sehwag; 72, 65, 47 resp. But when we want that triple, we have to look at Sehwag!

PS: Nothing is to be taken personally. You said healthy debate and here is one.

Giri@iisc said...

1. Your statement "None of these metrics are "normalized" in any way." is wrong. ESI is field dependent and normalized. The classification is based and normalized on citations in that field. Thus, a top 1% scientist in mathematics will have much lower citations than a top 1% scientist in chemistry/biology. For example, Vijayakumar whom you mention is in the top half% of scientists but not Prof. Balaram though the latter has much more citations than the former. Similarly, NCBS and inst. math. sci. work in different fields and are appropriately compared.

2. Your statement on h-index and citations is also flawed. It has been well established in literature that Nc=A*h*h. Nc is the total number of citations, h is the h-index and A varies from 2.5-5 for most the cases. Do the number of citations and divide by h^2 for many professors you mention and you will find A varies from 2.5 to 5. Studies of over 10,000 scientists with h-index of 20 or more showed the above trend and it is also reasonably known in which field it is around 3 and in which fields it is around 5. There are several papers published on this aspect of Nc versus h in scientometric journals. There are also papers published on indices that includes a composite of citations per paper, number of papers, h-index and normalized to the subfield.

3. Are metrics available that can capture breakthrough or consistent and/or long-innings? Yes, there are. There are indices for individuals and institutions. For example, they are based on the number of top 1% cited papers in the field, top 5% etc. Unfortunately, as you see, they are based on citations and publications though they remove citations accrued to review papers and method papers.

4. You said "Citations are no good when looking at any university beyond 50 (ie., small);" Therefore, unless one believes that citations and publications are some measure of research productivity, it is very difficult to develop indices based on subjective opinion. I do believe that citations and publications are a good measure when it measured for large group of individuals and universities and certainly for countries. This is because large groups will have individuals who publish a lot, get cited a lot; publish a lot, get cited less; publish very few, get cited a lot and publish very few and not be cited.

5. Your comparison of Sehwag, Dravid and Tendulkar is not appropriate. All are considered excellent cricketers. You may want a treble, and thus pick Sehwag; So, what? Nobel prizes are not given to the person who has the highest h-indices. However, people who have good h-index are probably good researchers but have not achieved anything close to a Nobel prize.

The anon comment at 3:39 am
"I find this entire debate that tries to assess the quality of a researcher without using any credible evidence of it, tending to the absurd." is probably the best and should end this discussion.

BTW, I never said many professors were deadwood etc. These are your statements and not mine. Research productivity is not everything in an institution. You need good teachers, excellent administrators who run hostels, serve on many committees etc. Ranking of an institution may depend on citation etc. but running an institution requires all kinds of dedicated faculty. You do not need to call them scholars (as someone else alleged) but to call them deadwood is an insult.

Thanks

Giridhar

Jayaram Krishnaswamy (mysorian) said...

Learn cut and paste and you have a voluminous and impressive resume. Citations may help, especially if you are looking for tenure.

Do I appear skeptical? Not by a long shot. My eyes have seen things.....

Anonymous said...

You are not skeptical but you are insulting scientists like Prof. Madras.

Oh, by the way, there is no tenure system in IISc/IIT.

Anonymous said...

Last Anon - there is no tenure system in IITs, but IISc has something similar - you get only a 5 year contract in the beginning. They can terminate you after 5 years in IISc, while it is a permanent appointment in IITs, salary and all other benefits being the same.

Anonymous said...

In principle, yes. But not a single person has been terminated on this issue.

Heck, recently, a person in chemical engineering with ZERO publications in five years was granted "tenure".

So, yes, IISc can terminate one after five years but not for lack of publication or teaching. Maybe if you shoot the director !

Anonymous said...

Anon@8:05 am: "not a single person has been terminated.."

Might be very well true, but the possibility still exist. What i don't understand is how can iisc and IITs, both being under MHRD, have different norms for appointment - and that too only in this particular aspect? What if someone goes to court?

For the record - I'll be joining a top univ in Canada and have an informal offer from IITK. Being a south Indian, I'd love to join IISc and was seriously considering to apply at one point - but this 'tenure' thing scared the hell out of me, to put it honestly. The long wait times for promotion at IISc was another issue.

KR

Giri@iisc said...

Dear KR,

I will not respond to your comment on "but this 'tenure' thing scared the hell out of me, to put it honestly" I hope the top univ in Canada does not have tenure.

However, your comment
"What i don't understand is how can iisc and IITs, both being under MHRD, have different norms for appointment - and that too only in this particular aspect? " is intriguing.

So, you mean NITs, IITs and IISc should have same rules for everything because we get the same salaries? The appointment for NITs and IITs are only through interview. For example, there is no "promotion" to associate professor in NIT/IIT, one has to appear for an interview against a general advertisement. The mode of promotion in IISc is entirely different. Also, there are several differences between NIT and IIT on recruitment, "promotion" etc. After all, NIT is also run by MHRD.

There are several differences between IIT and IISc. IISc is not an Institute of national importance, also come under UGC. Some older rules of UGC are also in place.

Oh, finally, the appointment in IITs also are on contract basis these days...so your statement "but the possibility for termination)still exist" also exists in IITs

Best wishes,

Giridhar

Anonymous said...

Dear KR,

Why are you scared of Tenure. All the good universities in US follow this system. Infact the criteria for IISc tenure is not very demanding at all.

iitmsriram said...

Anon Aug 27 10:22 PM said "On the other hand, within the Indian context where tenure, salary and promotions are not dependent on research quality/quantity/metrics ...". Wow! There is no tenure system in India, so that part of the statement is irrelevant. Salary is fixed and with fixed annual increments (except in the case of promotions), so OK, leave that alone. But it is a rather unjustifiable claim that in the Indian context, promotion is not dependent on research quality or quantity or metrics. In the IIX's, promotions are very much dependent on these, I believe.

Anonymous said...

@iitsriram

I was the anon your replied to. Many comments in these columns claim there are examples of assistant professors with 0 publications being promoted; professors with lifetime 10-20 publications and so on. My comment was based on those, but ts good to know promotion depends on tenure.

For someone extremely motivated, I agree that being in US or India or any incentives just does not matter. He/She goes to do his/her best. For others, I am not sure how you can just "leave" other two points I raised, on salary and tenure. However flawed they may be, do they not play an important role in raising research standards? Added to that we lack a critical mass of good scientists, and you know the long list that many people have come up with :-).

Anonymous said...

"but ts good to know promotion depends on tenure."

What do you mean promotion depends on tenure? There is no tenure.

Also, the cases of professors with less than 30 publications in their lifetime is not uncommon. actually it is very common. You do not need to believe it..you can check the webpages of faculty in these "premier institutes" No need for belief.

See

http://unfairpromotionsatiits.blogspot.com/2009/11/bloats-on-iit-system.html

Anonymous said...

@above

sorry for the typo. it must have been "but ts good to know promotion depends on performance."

May be not anymore, after seeing your link :-).

Anonymous said...

@Prof. Giri:

"I hope the top univ in Canada does not have tenure"

Snide remarks aside, I am not worried abt the tenure in Canada as much as I do in IISC because
(1) IMHO, if I don't get tenure, the chances of getting another job anywhere in the world (including India) are much brighter - can that be said of IISc? - your esteemed opinion might be different

(2) I'll at least have some significant savings (imagine getting 35000 p.m in Bangalore, and if you are asked to leave at
the end of 5 years, what will you be left with?)

(3)I believe the univ in Canada has far more transparent and consistent policies for tenure and promotions - My experience
with IITB doen't give me any confidence in the Indian system in this aspect.

(4)from a personal perspective, I've got a fully equipped lab with assured funding for 3 years - so i can start some work right away - not so easily possible in India

I didn't mean IITs, NITs etc should have the same rules for everything - however, i believe policies for life and death
situations like throwing someone out of a job can't be left to individual discretion at different univs in the Indian context, esp. when the salaries and every other benefits are the same. I'd have had no complaints if IISc pays 10 times more and throws someone out after 5 years. You say IISc also comes under ugc - why doesn't then any univ under ugc have this
policy, which is definitely more appropriate to them than IISc?

Finally, the contract appointments at iits don't have any meaning, as someone suggested it is an automatic rollover to the permanent post.

regards
KR

Giri@iisc said...

"Finally, the contract appointments at iits don't have any meaning, as someone suggested it is an automatic rollover to the permanent post."

How is the contract appointment at IIT different from IISc? As someone else pointed out, it seems to be an automatic rollover in both cases. The only difference seems to be that the contract appointment in IITs is 3 years while it is five in IISc. There is no word called tenure in IISc context or this word is mentioned anywhere.

NCBS and a few other institutions, however, have tenure. Not IISc/IIT/NIT. All these institutions have contract appointments for varying periods but seem to automatic rollover.

Thanks

Giridhar

iitmsriram said...

Anon Sept 2 7:24 PM states that several postings "claim there are examples of assistant professors with 0 publications being promoted". I would very seriously doubt the accuracy of such claims. I checked out IITB Chemical Dept website as there is a claim that someone with zero papers was promoted associate professor - I could find no such candidate in the web site. Maybe, zero publications over some check period ... At IITM, a candidate with zero journal publications would not be considered for assistant professor. Of course, there is some exception possible, if the candidate worked in some defence lab, for example, and had done a lot of work but nothing in open literature. I have found selection committees to generally consider many factors, and not just a single metric like number of papers.

iitmsriram said...

Anon @Aug 28 12:01 claims that (in) "IIT-Bombay, recently, a faculty was promoted from assistant professor to associate professor with ZERO publications"; the IITB Chemical Engineering website does not bear this out. There are 30 faculty members listed, 16 professors, 6 asso profs and 8 asst profs. Every one of the asso profs has about 10 papers in the last 5 years and the asst profs each have about 5 papers in the last 5 years and there is no one with zero publications. Perhaps, the original poster meant zero papers in some particular calendar year or some such.

Anonymous said...

I am Anon @Aug 28 12:01. Let me clarify that the person listed in the website is still marked as assistant professor, though he has been promoted to associate professor last month. His profile: joined IITB in 2007, published two papers of the post doc work with post doc advisor as corresponding author in 2008 and nothing else. Got promoted in 2010. So, if you take 2005-2010, he has more than 5 five publications but not a single paper from IIT-Bombay as affiliation. I am his neighbour and know him well. This is just an example. I can give you a list of such professors from IIT-Delhi.

See

http://unfairpromotionsatiits.blogspot.com/2009/11/bloats-on-iit-system.html

Giri@iisc said...

Dear Anon,

I kindly request you to close the discussion regarding the promotions. This post is on rankings of institutes, not on individual promotions. As I have written before, several factors other than research can go into assessing performance and potential of a candidate. While these parameters of productivity are good at the institute level, it may not be applicable at an individual level. Further, an institute can not survive on faculty with research excellence alone. They also need faculty like me who will sit on so many committees and do institute work !

Thanks

Giridhar

Anonymous said...

The same person also managed to get a job as assistant professor for his spouse in some other department at IIT/Bombay.
For Info: His spouse has much poorer record compared to the person itself.

Anonymous said...

To Anonymous:
``His spouse has much poorer record compared to the person itself.''

A little addition: He also did entire masters and PhD work for his spouse ..so it is expected that the record of the spouse is poor...as he could not manage to do so many things simultaneously

Anonymous said...

Not a single Indian university in the list of 200 by THES..kind of demoralising for any student..

Hope the ranking of engg. and tech schools salvage some pride..

On a side note.. after being educated at one of the universities that has made it in the top 200 (way down though), I can say that the faculty interaction, knowledge (things that matter to a student) are much much better at the IIX's..

Money makes crucial differences I suppose..

-RG

chandra said...

This is incredible advice!Very insightful and straightforward.!@bose
Messages