Andy Meneely


Journal Conference Workshop

Publications

Word cloud visualization of the paper
Socio-Technical Developer Networks: Should We Trust Our Measurements?
Andrew Meneely, Laurie Williams
International Conference on Software Engineering (ICSE), 2011, to appear
Software development teams must be properly structured to provide effective collaboration to produce quality software. Over the last several years, social network analysis (SNA) has emerged as a popular method for studying the collaboration and organization of people working in large software development teams. Researchers have been modeling networks of developers based on socio-technical connections found in software development artifacts. Using these developer networks, researchers have proposed several SNA metrics that can predict software quality factors and describe the team structure. But do SNA metrics measure what they purport to measure? The objective of this research is to investigate if SNA metrics represent socio-technical relationships by examining if developer networks can be corroborated with developer perceptions. To measure developer perceptions, we developed an online survey that is personalized to each developer of a development team based on that developer's SNA metrics. Developers answered questions about other members of the team, such as identifying their collaborators and the project experts. A total of 124 developers responded to our survey from three popular open source projects: the Linux kernel, the PHP programming language, and the Wireshark network protocol analyzer. Our results indicate that connections in the developer network are statistically associated with the collaborators whom the developers named. Our results substantiate that SNA metrics represent socio-technical relationships in open source development projects, while also clarifying how the developer network can be interpreted by researchers and practitioners.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities
Yonghee Shin, Andrew Meneely, Laurie Williams, and Jason Osborne
IEEE Transactions on Software Engineering (TSE), 2010, to appear
Security inspection and testing requires experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely-used open source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all the three types of metrics together predicted over 80% of the known vulnerable files with less than 25% false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71% and 28%, respectively, for both projects.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Strengthening the Empirical Analysis of the Relationship between Linus' Law and Software Security
Andrew Meneely and Laurie Williams
Empirical Software Engineering & Measurement (ESEM) 2010
Open source software is often considered to be secure because large developer communities can be leveraged to find and fix security vulnerabilities. Eric Raymond states Linus' Law as "many eyes make all bugs shallow", reasoning that a diverse set of perspectives improves the quality of a software product. However, at what point does the multitude of developers become "too many cooks in the kitchen", causing the system's security to suffer as a result? In a previous study, we quantified Linus' Law and "too many cooks in the kitchen" with developer activity metrics and found a statistical association between these metrics and security vulnerabilities in the Linux kernel. In the replication study reported in this paper, we performed our analysis on two additional projects: the PHP programming language and the Wireshark network protocol analyzer. We also updated our Linux kernel case study with 18 additional months of newly-discovered vulnerabilities. In all three case studies, files changed by six developers or more were at least four times more likely to have a vulnerability than files changed by fewer than six developers. Furthermore, we found that our predictive models improved on average when combining data from multiple projects, indicating that models can be transferred from one project to another.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Improving Developer Activity Metrics with Issue Tracking Annotations
Andrew Meneely, Mackenzie Corcoran, and Laurie Williams
Workshop on Emerging Trends in Software Metrics (WETSoM) 2010
Understanding and measuring how groups of developers collaborate on software projects can provide valuable insight into software quality and the software development process. Current practices of measuring developer collaboration (e.g. with social network analysis) usually employ metrics based on version control change log data to determine who is working on which part of the system. Version control change logs, however, do not tell the whole story. Information about the collaborative problem-solving process is also documented in the issue tracking systems that record solutions to failures, feature requests, or other development tasks. To enrich the data gained from version control change logs, we propose two annotations to be used in issue tracking systems: solution originator and solution approver. We examined the online discussions of 602 issues from the OpenMRS healthcare web application, annotating which developers were the originators of the solution to the issue, or were the approvers of the solution. We used these annotations to augment the version control change logs and found 47 more contributors to the OpenMRS project than the original 40 found in the version control change logs. Applying social network analysis to the data, we found that central developers in a developer network have a high likelihood of being approvers. These results indicate that using our two issue tracking annotations identify project collaborators that version control change logs miss. However, in the absence of our annotations, developer network centrality can be used as an estimate of the project's solution approvers. This improvement in developer activity metrics provides a valuable connection between what we can measure in the project development artifacts and the team's problem-solving process.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Protection Poker: The New Software Security "Game"
Laurie Williams, Andrew Meneely, and Grant Shipley
IEEE Privacy & Security, 2010
Tracking organizations such as the US CERT show a continuing rise in security vulnerabilities in software, increasing awareness of insecure coding practices. Not all discovered vulnerabilities are equal - some have the potential to cause much more damage to organizations and individuals than others. In the inevitable absence of infinite resources, software development teams need to prioritize security fortification efforts to prevent the most damaging attacks. We propose the Protection Poker "game" as a collaborative means of guiding this prioritization. A case study of a Red Hat IT software maintenance team demonstrated the potential of Protection Poker for improving software security practices and team software security knowledge.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Secure Open Source Collaboration: An Empirical Study of Linus' Law
Andrew Meneely and Laurie Williams
Computer and Communications Security (CCS) 2009
Open source software is often considered to be secure. One factor in this confidence in the security of open source software lies in leveraging large developer communities to find vulnerabilities in the code. Eric Raymond declares Linus' Law "Given enough eyeballs, all bugs are shallow." Does Linus' Law hold up ad infinitum? Or, can the multitude of developers become "too many cooks in the kitchen", causing the system's security to suffer as a result? In this study, we examine the security of an open source project in the context of developer collaboration. By analyzing version control logs, we quantified notions of Linus" Law as well as the "too many cooks in the kitchen" viewpoint into developer activity metrics. We performed an empirical case study by examining correlations between the known security vulnerabilities in the open source Red Hat Enterprise Linux 4 kernel and developer activity metrics. Files developed by otherwise- independent developer groups were more likely to have a vulnerability, supporting Linus' Law. However, files with changes from nine or more developers were 16 times more likely to have a vulnerability than files changed by fewer than nine developers, indicating that many developers changing code may have a detrimental effect on the system's security.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Protection Poker: Structuring Software Security Risk Assessment and Knowledge Transfer
Laurie Williams, Michael Gegick, and Andrew Meneely
Int'l Symp. Engineering Secure Software and Systems (ESSoS) 2009
Discovery of security vulnerabilities is on the rise. As a result, software development teams must place a higher priority on preventing the injection of vulnerabilities in software as it is developed. Because the focus on software secu- rity has increased only recently, software development teams often do not have expertise in techniques for identifying security risk, understanding the impact of a vulnerability, or knowing the best mitigation strategy. We propose the Protection Poker activity as a collaborative and informal form of misuse case development and threat modeling that plays off the diversity of knowledge and perspective of the participants. An excellent outcome of Protection Poker is that security knowl- edge passed around the team. Students in an advanced undergraduate software engineering course at North Carolina State University participated in a Protection Poker session conducted as a laboratory exercise. Students actively shared misuse cases, threat models, and their limited software security expertise as they dis- cussed vulnerabilities in their course project. We observed students relating vul- nerabilities to the business impacts of the system. Protection Poker lead to a more effective software security learning experience than in prior semesters. A pilot of the use of Protection Poker with an industrial partner began in October 2008. The first security discussion structured via Protection Poker caused two requirements to be revised for added security fortification; led to the immediate identification of one vulnerability in the system; initiated a meeting on the prioritization of security defects; and instigated a call for an education session on preventing cross site scripting vulnerabilities.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
On Preparing Students for Distributed Software Development with a Synchronous, Collaborative Development Platform
Andrew Meneely and Laurie Williams
SIGCSE 2009
Working remotely is becoming the norm for both professionals and students alike. Software development has become a global industry due to outsourcing, teleworking, flex time, and companies' desire to use the best and/or most economical talent regardless of where that talent is located. Professionals are not alone because students usually work from home despite having sufficient resources on campus. In this paper we share our experiences from using Jazz, a synchronous, collaborative development platform, with our inevitably distributed software engineering students. Eleven students optionally used the tool while working on a five-week team project. Students primarily used the version control, chat, and work items features in Jazz. We collected their reactions in retrospective essays and found that all Jazz students supported using Jazz in future semesters of the course. We also examined grade differences and found that the students who used Jazz were more successful than those who did not use Jazz.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Jazz Sangam: A Real-Time Tool for Distributed Pair Programming on a Team Development Platform
John Vijay Sena Devide, Andrew Meneely, Chih-Wei Ho, Laurie Williams, Michael Devetsikiotis
iReCoSE workshop at FSE 2008
Pair programming has proven to be a useful technique for developing high quality code while sharing knowledge throughout a team. Rapid global dispersion of software development teams, however, makes co-located pair programming a challenge, motivating the need for development tools tailored specifically for distributed pair programming. Previously, the Sangam Eclipse plug-in was developed to support distributed pair programming. More recently, the Jazz collaborative software development platform was built to support team communication and the sharing of life-cycle resources and to integrate a variety of disparate tools used by team members. We have ported Sangam to the Jazz platform to enable teams to pair program within their integrated team environment. In this paper, we describe Jazz Sangam, highlight the choices that lead to Sangam's current design, and discuss how Jazz Sangam can improve the distributed pair programming experience.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Predicting Failures with Developer Networks and Social Network Analysis
Andrew Meneely, Laurie Williams, Will Snipes, Jason Osborne
FSE 2008
Software fails and fixing it is expensive. Research in failure prediction has been highly successful at modeling software failures. Few models, however, consider the key cause of failures in software: people. Understanding the structure of developer collaboration could explain a lot about the reliability of the final product. We examine this collaboration structure with the developer network derived from code churn information that can predict failures at the file level. We conducted a case study involving a mature Nortel networking product of over three million lines of code. Failure prediction models were developed using test and post-release failure data from two releases, then validated against a subsequent release. One model's prioritization revealed 58% of the failures in 20% of the files compared with the optimal prioritization that would have found 61% in 20% of the files, indicating that a significant correlation exists between file-based developer network metrics and failures.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
ROSE: A Repository of Education-Friendly Open-Source Projects
Andrew Meneely, Laurie Williams, Edward Gehringer
ITiCSE 2008
Open-source project artifacts can be used to inject realism into software engineering courses or lessons on open-source software development. However, the use of open-source projects presents challenges for both educators and for students. Educators must search for projects that meet the constraints of their classes, and often must negotiate the scope and terms of the project with project managers. For students, many available open-source projects have a steep learning curve that inhibits them from making significant contributions to the project and benefiting from a "realistic" experience. To alleviate these problems and to encourage cross-institution collaboration, we have created the Repository for Open Software Education (ROSE) and have contributed three open-source projects intended for an undergraduate computer science or software engineering course. The projects in ROSE are education-friendly in terms of a manageable size and scope, and are intended to be evolved over many semesters. All projects have a set of artifacts covering all aspects of the development process, from requirements, design, code, and test. We invite other educators to contribute to ROSE and to use projects found on ROSE in their own courses.

A word cloud of the paper's text. Generated by TagCrowd.com
Word cloud visualization of the paper
Fifteen compilers in fifteen days
Jeremy D. Frens, Andrew Meneely
SIGCSE 2006
Traditional approaches to semester-long projects in compiler courses force students to implement the early stages of a compiler in depth; since many students fall behind, they have little opportunity to implement the back end. Consequently, students have a deep knowledge of early material and no knowledge of latter material. We propose an approach based on incremental development and test-driven development; this approach solves the emphasis problem, provides experience with useful tools, and allows for such a course to be taught in a three or four weeks.

A word cloud of the paper's text. Generated by TagCrowd.com

Copyright information: Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.