Alternative Metrics Initiative

NISO Alternative Assessment Metrics (Altmetrics) Initiative

In June 2013, the Alfred P. Sloan Foundation awarded NISO a grant to undertake a two-phase initiative to explore, identify, and advance standards and/or best practices related to a new suite of potential metrics in the community. This initiative was a direct outgrowth of a breakout discussion group during the altmetrics 12 meeting in Chicago, IL. This project, which was accomplished in two phases, is seen as an important step in the development and adoption of new assessment metrics, which include usage-based metrics, social media references, and network behavioral analysis. The NISO Altmetrics Initiative also explored potential assessment criteria for non-traditional research outputs, such as data sets, visualizations, software, and other applications. The first phase, which took place from 2013-2014, exposed areas for potential standardization and the community collectively prioritized those potential projects. The second phase, which took place from 2014-2016, advanced work in several areas and developed those into recommended practices prioritized by the community and approved by the membership.

NOW PUBLISHED! NISO RP-25-2016, Outputs of the NISO Alternative Assessment Project

Information Resources

Original proposal (public version)

Information Standards Quarterly special issue on Altmetrics (Summer 2013)

Altmetrics Bibliography by Charles W. Bailey, Jr.

First In-person Meeting

NISO hosted the first of three free and open in-person meetings in support of this work on Wednesday, October 9, 2013 in San Francisco, CA. The objective of this gathering, as well as planned future meetings (see below), is to elicit and prioritize themes for greater discussion over the coming months. 

The Meeting Agenda page contains all archived recordings, documents, and other output from the day-long gathering.

Second In-person Meeting

NISO hosted the second of three free and open in-person meetings in support of this work on Wednesday, December 11, 2013 in Washington, DC.. The objective of this gathering, as well as planned future meetings (see below), is to elicit and prioritize themes for greater discussion over the coming months. 

The Meeting Agenda page contains all archived recordings, documents, and other output from the day-long gathering.

Third In-person Meeting

NISO hosted the third free and open in-person meeting in support of this work on Thursday, January 23, 2014 in Philadelphia, PA. The objective of this gathering was to elicit and prioritize themes for greater discussion over the coming months. 

The Meeting Agenda page contains all archived recordings, documents, and other output from the day-long gathering. 

Output from the Project

NISO anticipates several outputs from this project. The first major output will be a white paper describing the outcomes, recommendations, and action items identified during Phase One. This paper will be developed from the in-person meetings, circulated and vetted publicly by the research community and experts, and will identify the Recommended Practices and/or standards that need to be developed by NISO (possibly in conjunction with others) in order to speed the adoption of altmetrics. The white paper will be used as the basis for a draft New Work Item Proposal within NISO for the Phase Two work.

With the draft and community input in hand, NISO will be positioned to leverage the community used in developing the paper to create the working groups needed to develop those practices and standards. Presuming the approval of the NISO membership and leadership committees, the final output of this initiative will be a published standard(s) or recommended practice(s) regarding altmetrics. There will also be training and resources produced about how to implement and adopt the recommended practice(s)/standard(s).

A Steering Committee has been established.  All documents (including MP3 recordings of conference calls) and email related to this project will be made publicly available via this page. 

Phase One White Paper, Comments, and Prioritization Survey

NISO has released a White Paper summarizing the Phase 1 work, summarizing the findings from the meetings and interviews, along with the identification of potential action items for further work in Phase II of the project. More than 250 ideas were generated by participants in the meetings and interviews, and NISO condensed these to 25 action items in nine categories: definitions, research outputs, discovery, research evaluation, data quality and gaming, grouping and aggregation, context, stakeholders’ perspectives, and adoption.

The White Paper was released for public comment from June 9 through July 18, 2014. A summary of the received comments is avaialble here. Further prioritization input of the proposed action items was solicited via a survey from August 11-29, 2014, the results of which are summarized here.

Phase One Community Input Survey

Following the close of the comment period for the NISO draft white paper summarizing Phase I of its Alternative Assessment Metrics (Altmetrics) Project, NISO made available a survey inviting the community to provide input as to what action items in the realm of alternative assessment are most important for NISO to work on in its upcoming Phase II. 

The survey ran from August 11-August 29, 2014.  The text of the survey and summary results are available.

Estimated Timeline

A high-level, projected timeline for project activities that will be accomplished is as follows:

  • July 2013 – Negotiate contract with identified consultants to moderate Phase One work and to write mid-term whitepaper. Make arrangements concerning in-person meetings and issue invitations to contributors.

  • August 2013 – Set up wiki, online workspace, and e-mail lists. Announce fist meeting and availability of informational resources about the project.

  • November/December 2013 – Hold first face-to-face meeting, possibly to coincide with CNI. Build initial list of issues and concerns. Schedule and announce two follow-up webinars.

  • January 2014 – Hold the second in-person meeting in conjunction with ALA-Midwinter meeting in Philadelphia. Consultant begins drafting paper based on in-person sessions.

  • February/March 2014 – Public webinars to engage larger community and to discuss meetings and draft paper.

  • April/May 2014 – Consultant finalizes draft of white paper and first draft is released. Presentation of results and discussion at Altmetrics14 (TBD), CNI, and elsewhere.

  • June 2014 – Presentations at ALA Annual Conference in Las Vegas (June 27-July 1) and session held at Joint Conference on Digital Libray (JCDL), or similar venues, where the initial draft of the paper is presented, along with an outline of areas where recommended practices and standards are needed. Gather public feedback.

  • July 2014 – Consultant finalizes the whitepaper and draft new work item proposal for standards/best practices based on the study. Paper released. Proposal vetted by NISO leadership and members. WG members identified.

  • August/September 2014 – Proposal approved and working group formed.¹

  • September 2014 – August 2015: Working group develops recommended practices/standards. Public draft for comment or draft standard for trial use released. Discussion/input at webinars to report project outputs.

  • Fall/Winter 2015 – NISO publishes standards/recommended practices and related training materials.²

  • November 2015 – NISO submits report to the Sloan Foundation.

¹ The second phase standards/recommended practices project is subject to approval of the the Business Information Topic Committee as well as the NISO Voting Membership. Approval is not guaranteed.

² Final publication is subject to the approval of the the Business Information Topic Committee as well as the NISO Voting Membership. Approval is not guaranteed.

Phase 2 Projects

NISO Voting Members reviewed and approved a proposal to develop several standards or recommended practices during Phase 2 of the Altmetrics Initiative. Areas/topics to be addressed are:

  • Development of specific definitions for alternative assessment metrics – This working group will come up with specific definitions for the terms commonly used in alternative assessment metrics, enabling different stakeholders to talk about the same thing. This work will also lay the groundwork for the other working groups.

  • Definitions for appropriate metrics and calculation methodologies for specific output types – Research outputs that are currently underrepresented in research evaluation will be the focus of this working group. This includes research data, software, and performances, but also research outputs commonly found in the social sciences. The working group will come up with recommendations for appropriate metrics for these research outputs and will develop guidelines for their use.

  • Development of strategies to improve data quality through source data providers – Data quality is essential before any alternative assessment metrics can be used for research evaluation. This working group will look at issues of data quality and will recommend strategies to overcome these issues, or to clarify the limitations of particular assessment metrics.

  • Promotion and facilitation of use of persistent identifiers in scholarly communications – Persistent identifiers are needed to clearly identify research outputs for which collection of metrics is desired, but also to describe their relationships to other research outputs, to contributors, institutions, and funders. This working group will work closely with other initiatives in the space of identifiers.

  • Descriptions of how the main use cases apply to and are valuable to the different stakeholder groups – Alternative assessment metrics can be used for a variety of use cases from research evaluation to discovery. This working group will try to identify the main use cases, the stakeholder groups to which they are most relevant, and will also develop a statement about the role of alternative assessment metrics in research evaluation.

Community members interested in participating in this project should email Nettie Lagace before February 15, 2015.