SciTS 2011 Conference: Sessions

Evaluation of Team Science: A Multilevel Systems Perspective (Panel)

Tuesday, April 12 • 10:00 AM – 12:00 PM

The panel will consider major challenges in the evaluation of team science, examining the local, interpersonal and network level and the broader macro level of big science and translational research. Although science is concerned with examining ideas about the world, fundamentally it involves an interpersonal process of exchange among scientists. Local evaluations of team science can benefit by taking this interpersonal process into account when attempting to understand scientific collaboration and its relationship to innovation and discovery. In addition to the interpersonal dynamics that influence the outcomes of team science, network dynamics can also be utilized to inform the evaluation of team science. Specifically, panelists will discuss the role of social and knowledge networks in shaping the assembly of scientific teams and evaluate how these network influences imprint their subsequent performance. While many evaluations tend to emphasize accountability concerns and national sponsors, panelists will also discuss how evaluations can and should address local needs and stakeholders, in particular the scientist managers who are asked to lead complex team science initiatives. Findings and practices from multi-level evaluations that emphasize "improvement-focused evaluation” will be discussed. Finally, one of the major challenges for the evaluation of team science occurs because collaboration often requires a greater initial investment of time than individual investigator research. Evaluations that assess how long it takes to move research from basic ideas to practice and impact (such as process model analyses of translational research) run the danger of making it appear that team science slows down the translational process, unless we expand the scale to examine potential gains over the longer run that accrue because of team collaboration. The panelists will consider the challenges and benefits of incorporating a “systems perspective” in the evaluation of team science that addresses interpersonal and network perspectives, multiple stakeholder views, and process assessment at multiple levels.

Noshir Contractor, Ph.D., Northwestern University, Professor, Industrial Engineering and Management Sciences, Communication Studies, and Management and Organizations
Denis Gray, Ph.D., NC State University, Professor, Department of Psychology, Psychology in the Public Interest
Jacob Kraemer Tebes, Ph.D., Yale University, Associate Professor of Psychology in Psychiatry, Child Study Center, and Epidemiology and Public Health
William Trochim, Ph.D., Cornell University, Director, Office for Research on Evaluation; Director of Evaluation, Weill Cornell Clinical and Translational Science Center; Director of Evaluation for Extension and Outreach

Video

Watch: Evaluation of Team Science: A Multilevel Systems Perspective

Panelists

Noshir Contractor

Noshir Contractor, Ph.D.is the Jane S. & William J. White Professor of Behavioral Sciences in the McCormick School of Engineering & Applied Science, the School of Communication and the Kellogg School of Management at Northwestern University, USA. He is the Director of the Science of Networks in Communities (SONIC) Research Group at Northwestern University. He is investigating factors that lead to the formation, maintenance, and dissolution of dynamically linked social and knowledge networks in a wide variety of contexts including communities of practice in business, translational science and engineering communities, public health networks and virtual worlds. His research program has been funded continuously for over a decade by major grants from the U.S. National Science Foundation with additional current funding from the U.S. National Institutes of Health (NIH), Air Force Office of Research Support, Army Research Institute, Army Research Laboratory and the MacArthur Foundation.

Professor Contractor has published or presented over 250 research papers dealing with communicating and organizing. His book titled Theories of Communication Networks (co-authored with Professor Peter Monge and published by Oxford University Press) received the 2003 Book of the Year award from the Organizational Communication Division of the National Communication Association. He is the lead developer of C-IKNOW (Cyberinfrastructure for Inquiring Knowledge Networks On the Web), a socio-technical environment to understand and enable networks among communities, as well as Blanche, a software environment to simulate the dynamics of social networks."

Denis Gray

Denis Gray, Ph.D.is Alumni Distinguished Graduate Professor, Psychology in the Public Interest Program, Department of Psychology, North Carolina State University. Dr. Gray’s research focuses on science and technology policy issues, particularly the outcomes and implications of team-based cooperative research centers. For the past twenty years he has led a unique, multi-faceted ‘improvementfocused’ evaluation of the National Science Foundation’s (NSF) longest-running cooperative research center program, the Industry-University Cooperative Research Centers (IUCRC). His books include: Innovation U.: New university roles in a knowledge economy (co-author); Managing the industry/university cooperative research center (Battelle Press) and Technological innovation: Strategies for a new partnership (Elsevier) (senior editor). He will co-edit (with Craig Boardman) a forthcoming book for Springer Press on cooperative research centers and open innovation. Current research projects focus on evaluating S&T human capital formation and economic impact of cooperative research. A native of New York City (and former taxi cab driver), Dr. Gray received his B.S. in Psychology from Manhattan College. He received his Ph.D. and M.S. degrees in Ecological/ Community Psychology from Michigan State University.

Jacob Kraemer Tebes

Jacob Kraemer Tebes, Ph.D. is Associate Professor of Psychology in Psychiatry, Child Study Center, and Epidemiology & Public Health at the Yale University School of Medicine, where he also serves as Co-Director of the Division of Prevention and Community Research and Deputy Director of The Consultation Center. Dr. Tebes earned a B.S. in Psychology from Georgetown University and a Ph.D. in Clinical/Community Psychology from the State University of New York at Buffalo. He is a fellow of the American Psychological Association and of the Society for Community Research and Action. Dr. Tebes is also Editor of the American Journal of Community Psychology, and is on the editorial boards of the Journal of Community Psychology and Child Abuse and Neglect. His research has focused on the promotion of resilience in at risk populations, the prevention of adolescent substance use, the evaluation of system-level and community interventions, prevention and evaluation research methodology, and meta-science. Dr. Tebes currently serves as Principal Investigator of the evaluation for the Interdisciplinary Research Consortium on Stress, Self-Control, and Addiction that is funded through the NIH Roadmap Initiative, and is also an advisor to the national evaluation of interdisciplinary research consortia funded through that initiative. At Yale, Dr. Tebes teaches advanced seminars in prevention research methods and clinical methods, and co-directs NIDA training and education programs in substance abuse prevention research and interdisciplinary team science.

William Trochim

William Trochim, Ph.D., received his doctorate degree from the Department of Psychology at Northwestern University in the area of Methodology and Evaluation Research. His research is in the area of applied social research methodology, with an emphasis on program planning and evaluation methods. He is known for the development of a number of methodologies in the behavioral, social and medical sciences. He is a prolific writer, including several widely used introductory research methods texts, and articles that have appeared in American Journal of Evaluation, New Directions for Program Evaluation, Evaluation and Program Planning, Evaluation Review, the American Journal of Public Health, Journal of Clinical Epidemiology, Consulting and Clinical Psychology, Controlled Clinical Trials, Performance Improvement, and Medical Decision Making, among others. He is actively engaged in research with the National Science Foundation (NSF) incorporating systems approaches in the evaluation of Science, Technology, Engineering and Mathematics (STEM) education programs. He is working with Dr. Gilbert Botvin, Weill Cornell Medical Center, on an NIH project in studying the use of Life Skills Training and its role in the dissemination, adoption, implementation, and sustainability (DAIS) of this evidence-based program. He is the Director of Evaluation for the Weill Cornell Clinical and Translational Science Center -- a collaboration of Weill Cornell Medical College, Memorial Sloan Kettering Cancer Center, The Hospital for Special Surgery, Hunter College and Cornell Cooperative Extension -- and participates actively in the NIH Clinical and Translational Science Awards national evaluation. He has also served multiple terms on the Board of Directors for the American Evaluation Association, including serving as the 2008 AEA President.