Hide Notes 

Slide 1: Article-Level Metrics (at PLoS & beyond) • These are the slides and audio from a webinar organised by NISO on May 13th, 2009. The topic of the webinar was “New Applications of Usage Data”. • This segment (the second of three) was given by Peter Binfield, Managing Editor of PLoS ONE • More information (as well as details of the other presentations) can be found at the NISO site at: http://www.niso.org/news/events/2009/usage09/ www.plos.org
Slide 2: Committed to making the world’s scientific and medical literature a public resource Article-Level Metrics (at PLoS & beyond) Peter Binfield, Managing Editor of PLoS ONE, pbinfield@plos.org www.plos.org
Slide 3: Presentation Outline • • • • • Brief overview of PLoS What is the problem? Current assessment methods Possible assessment methods Article-Level Metrics at PLoS and beyond (also, see the NISO resource page for relevant links) www.plos.org
Slide 4: The Public Library of Science (PLoS) • Largest Not for Profit, Open Access Publisher • Based in San Francisco and Cambridge, UK • Publisher of 7 journals – PLoS Biology, PLoS Medicine, PLoS Pathogens, PLoS Computational Biology, PLoS NTDs, PLoS Genetics, PLoS ONE • Largest journal is PLoS ONE – Which is almost doubling in volume each year www.plos.org
Slide 5: What is the Problem? As a consumer of research output: • How do you measure the ‘worth’ or the ‘impact’ of research output – typically, but not exclusively, a journal article • and how do you filter what you read? – You can’t read everything, so how do you filter to find the ‘best’ material for your interests? • Aside: ‘impact’ is a loosely used term – We could use words like ‘utility’ ; ‘value’ ; ‘relevance’ ; ‘interest (to me)’ ; ‘Return on Investment’ etc. – For this presentation I will just use the word ‘impact’ www.plos.org
Slide 6: What is the Problem? Again, as a consumer of research output: • At what level of granularity do you want to measure ‘impact’ and why are you measuring it? – – – – Journals - to know which titles to subscribe to? Research groups / institutions - to know who to fund? Individual researchers - to know who to promote? Individual articles - to know what to read? • and how do you compare ‘impact’ across different journals / disciplines / sub-disciplines / publishers? www.plos.org
Slide 7: What is the Problem? • Given these inherent complexities, how is ‘impact’ currently measured? – Citation based measures (mostly) – Usage based measures (occasionally) – Anecdotal / personal evaluations (often) www.plos.org
Slide 8: Current Assessment Methods (quick overview) • Mostly journal-based measures – the Impact Factor – COUNTER usage reports • Some article-level or ‘person’-level measures – the h-index – Article level citation information – Article level usage data www.plos.org
Slide 9: Current Assessment Methods • Article level usage data – What does it mean to have ‘a download’? Can a download be equated to ‘impact’? If you downloaded it, did you read it? – Easy to game? – What is a high vs. low number in any given field, or over any given time frame? – An apparent fear by data owners (publishers) that publicizing this data could be dangerous – Very little provision of this data – only the ‘Journal of Vision’ and the ‘Frontiers’ series currently provide hard data. Some other journals provide summaries (e.g. ‘top downloaded articles’ from HighWire titles) but no actual numbers are provided www.plos.org
Slide 10: Journal of Vision Screenshot www.plos.org
Slide 11: Frontiers in Neuroscience Screenshot www.plos.org
Slide 12: NEJM Screenshot www.plos.org
Slide 13: Possible assessment methods • There are many possible ways to assess the ‘impact’ of any piece of research (or any researcher): – Citations ; Usage ; Media Coverage ; Blog Coverage ; Discussion Thread activity ; Expert Ratings ; Social Bookmarking activity ; whether or not public policy was affected; where the research was done ; who is citing it ; who is reading it etc etc • Current technology, and Web 2.0, is now making it possible to automate many of these metrics www.plos.org
Slide 14: Possible assessment methods • If these types of measures could be collated and evaluated correctly then they could be more powerful (and useful) than citations or usage metrics alone • This is what PLoS is attempting to do with our Article-Level Metrics program - to implement new approaches to the evaluation and filtering of journal articles. www.plos.org
Slide 15: Article-Level Metrics (at PLoS) • Important to note: Article-Level Metrics at PLoS are not just about citations and usage. The concept refers to a whole range of additional measures which might give a window on ‘impact’ • We are providing metrics at the article-level, for every article, in every one of our titles. • We are the first publisher to provide this range of data, but we hope that others will follow www.plos.org
Slide 16: Article-Level Metrics (at PLoS) Rollout Plan: • We are starting by acquiring and presenting the relevant data. • We have a preference for data that is not ‘owned’ by third parties, and we are seeking ‘open’ APIs onto the data (so that anyone else can verify or replicate these measures themselves). • ‘Phase 2’ will provide data analysis, filtering and navigation tools using this data. www.plos.org
Slide 17: Article-Level Metrics (at PLoS) In March 2009, we launched with: • Number of Citations – PubMedCentral and Scopus • Amount of Blog coverage – Postgenomic, Nature Blogs and Bloglines • Number of Social Bookmarks – CiteULike and Connotea • Commenting, notation and ‘star ratings’ Each measure is provided as a simple number, but with a link to a landing page at the third party www.plos.org
Slide 18: Cu rre nt Vie w( Ma rch ‘09 ) www.plos.org
Slide 19: Cu rre nt Vie w( Ma rch ‘09 ) www.plos.org
Slide 20: Citeulike Landing Page www.plos.org
Slide 21: Cu rre nt Vie w( Ma rch ‘09 ) www.plos.org
Slide 22: Postgenomic Landing Page www.plos.org
Slide 23: Article-Level Metrics (at PLoS) • Future Expansion of Current Provision – Add ‘behavioural’ or ‘reference management’ tools (e.g. Mendeley) – Expand the number of sources for each data type (e.g. more blog aggregators, more citation sources) – Add expert ratings (e.g. F1000 or similar) – Add news coverage (hopefully) – Add usage data www.plos.org
Slide 24: Article-Level Metrics (at PLoS) Adding Usage Data • In mid-2009, we will add usage data to every PLoS article, showing: – HTML Page Views – PDF Downloads – XML Downloads To be displayed numerically and graphically, including historical data, and in the context of other articles within the journal. www.plos.org
Slide 25: Article-Level Metrics (at PLoS) Future Developments • Provide an open data set • COUNTER certification of the usage data • Inclusion of usage data from third parties (e.g. PubMed Central) • De-duplication of the data sources • Provision of tools to act on all these metrics for: – Discovery, Filtering and Comparison purposes • Adherence to any standards that may evolve www.plos.org
Slide 26: Article-Level Metrics (beyond PLoS) • Article-Level Metrics could be very significant for the academic community on a number of levels. • We do not want this to be a PLoS-only initiative. By demonstrating it, we hope other publishers will follow • We hope that acquisition and presentation standards will evolve or be created • We hope that by creating an open data set of standardized (cross journal) results, academia will then develop new and powerful ways to combine and evaluate this data • There is likely to be a range of ways to evaluate impact, for a range of different fields and different purposes, but the important thing will be consistent, reliable and comparable data sets www.plos.org
Slide 27: Article-Level Metrics We could be at the start of a major new development in academic publishing. www.plos.org
Slide 28: Article-Level Metrics We could be at the start of a major new development in academic publishing. To misquote from Jerry Macquire: www.plos.org
Slide 29: Article-Level Metrics We could be at the start of a major new development in academic publishing. To misquote from Jerry Macquire: “Show Me The Metrics!” www.plos.org
Slide 30: Article-Level Metrics We could be at the start of a major new development in academic publishing. To misquote from Jerry Macquire: “Show Me The Metrics!” Peter Binfield Managing Editor, PLoS ONE pbinfield@plos.org www.plos.org