Home
  

  About Us
  Research Projects
  Directory
  Contact Info
  VIP Room
  Calendar
  Library
  Search

SERC Technical Reports

New Technical Reports

Title: Metrics-Directed Analysis of VHDL Models
Author(s): Steven Sprunger, Raytheon, Dolores Zage, & Wayne Zage
SERC#: SERC-TR-291
Publication Date: 09/2007

Description: This paper explores the use of design metrics for the Very High Speed Integrated Circuit Hardware Description Language (VHDL). Design Metrics are one way an engineer can determine how well a system has been designed.  The purpose for using metrics is to provide guidance for both managerial and technical decisions during the life cycle of a project. Metrics help to gauge the general health (or quality) of a project while providing indicators to potential trouble areas within the project before they propagate into larger ones. While using standard metrics in the design decision process is commonplace in the software realm, using a set of metrics to influence Very High Speed Integrated Circuit Hardware Description Language (VHDL) designs decisions (which generally target a hardware component) are not as well understood.  The set of metrics selected for the evaluation of VHDL designs is based upon the research performed by the Zages and their research team at Ball State University.

Description:
TraceGraph 4 is a tool to assist a software engineer in locating and understanding the code for a specific software feature. It has been developed as part of a project of the Software Engineering Research Center (SERC) with input and support from SERC affiliate Motorola. This report describes a case study of TraceGraph 4 carried out at Motorola in May of 2007. The main objective of the study was to see if the tool was ready for possible application to large Motorola software systems. The two subjects in the study used TraceGraph 4 to locate two seeded feature-related bugs in a 123 KLOC Motorola system. Each bug was located correctly in about 20 to 25 minutes of work. The subjects made several suggestions for non-critical improvements and, overall, the tool was judged to be sufficiently robust for trials on larger Motorola projects.

Description:
Statecharts, now an integral part of the Unified Modeling Language (UML), serve as a requirement and/or a design specification. The effectiveness of statecharts as a tool to express the desired behavior of security protocols and a source of tests was investigated. Specifically, the TLS protocol was modeled as a statechart and tests generated from the flattened version of the model. The GnuTLS implementation of the TLS protocol (about 40 KLOC in size) was then tested against the generated tests and their adequacy assessed using MC/DC coverage. The MC/DC coverage of different portions of the implementation varied from 51% to 81%. A ?hat if?analysis revealed that while some defects in the uncovered portion of the code will not lead to any security vulnerability due to in-built error detection, a few others might lead to improper authentication, integrity failure, session hijacking, denial of service, and loss of confidentiality. The analysis suggests that statecharts alone might not be an adequate tool as a source of tests for implementations of security protocols and that tests so generated must be augmented through other formal means such as random testing, stress testing, and code coverage analysis.

Description:
The objective of the work reported here is to develop a detailed cell-level computational model of the human auditory pathway is under development. The model, once fully developed and validated against experimental data, will assist in the study of neural plasticity observed in the central auditory pathway as a consequence of auditory training in children with leaning and attention deficit disorders. Researchers have quantified the effect of auditory training by the brainstem evoked auditory potential, which the proposed complete computational model is expected to reproduce. Specifically, a complete and validated computational model will be used as a tool to assist in understanding the effect of (a) non-intrusive treatments in children with learning disabilities, and (b) the fault tolerance of the pathway to time varying defects in its cellular substance and structure. This report summarizes the progress made towards the stated objective

Description:
Malware is a worldwide epidemic. Studies suggest that the impact of malware is getting worse. Malware detectors are the primary tools in defense against malware. The quality of such a detector is determined by the techniques it uses. It is therefore imperative that we study malware detection techniques and understand their strengths and limitations. This survey examines 45 malware detection techniques and offers an opportunity to compare them against one another aiding in the decision making process involved with developing a secure application/system. The survey also provides a comprehensive bibliography as an aid to researchers in malware detection.

Description:
Representation of Role Based Access Control (RBAC) policies as finite state models and three conformance testing procedures for generating tests from these models are proposed. A test suite generated using one of the three procedures has excellent fault detection ability but is astronomically large. Two approaches to reduce the size of the generated test suite were investigated. One is based on a set of six heuristics and the other directly generates a test suite from the finite state model using random selection of paths in the policy model. A fault model specific to the implementations of RBAC systems was used to evaluate the fault detection effectiveness of the generated test suites; the model incorporates both mutation-based and malicious faults. Empirical studies revealed that adequacy assessment of test suites using faults that correspond to first-order mutations may lead to a false sense of confidence in the correctness of policy implementation. The second approach to test suite generation is most effective in the detection of both first-order mutation and malicious faults and generates a significantly smaller test suite than the one generated directly from the finite state models.

Description:
The goal of the Metrics Directed Verification of UML Designs project is to assess the utility and effectiveness of the design metrics, De and Di, in identifying change-prone software components, supporting Raytheon’s CMMI and Six Sigma initiatives. The study data consisted of a 400,000 line Java project which was reverse engineered to obtain the current design representation in UML. For each of the 2,828 resulting classes, a change count and 22 metrics were collected, including the primitives that comprise De and Di. For this study, we actually have two sets of process change data, changes from all relevant Cos (DS1) and change data where changes due to enhancements are not included (DS2). We then determined if UML classes with high numbers of change orders were highlighted as stress points by De. The result is that De correctly classified UML classes 89.1% of the time for DS1 and 87.4% of the time for DS2, suggesting that De can accurately identify the most problematic UML classes, as given by the number of changes. The internal design metric Di correctly classified UML classes 88.2% of the time for DS2 suggesting that Di can identify internal complexity. If both De and Di are used to highlight classes, then fewer classes will be highlighted with very few false positives (1.2%).

Description:
A method is proposed for generating tests for implementations of Role Based Access Control (RBAC) policies. First step in the method is construction of a finite state model that expresses the desired behavior of an RBAC implementation. Six heuristics are proposed to scale down the model for large systems consisting of thousands of users, roles, and permissions. Next, the model is input to a test generator that employs the existing automata theoreticW- or Wp- methods for test generation. Depending on the heuristic used, a combination of stress and random testing is recommended to enable detection of faults that might be missed by tests generated from the scaled down model. The fault detection effectiveness of the proposed method is evaluated against a fault model that corresponds well with the one used for analyzing the effectiveness of the W- and Wp- methods.

Description:
Events generated by the execution of a distributed system are related by causality and concurrency. While providing a means of reasoning about the relative occurrence of events, this partial order fails to represent the timeliness of occurrence. In this paper, we develop a novel means of assigning weights to events where the weights are reduced as the temporal proximity to an anchor event increases. This weight quanties the strength of the causal or concurrent relationship with respect to an anchor event. Those events that causally succeed the anchor are the focus of this paper with concurrency and causally preceding being part of future work plans. Three methods of computing event weights for causally succeeding events are dened. Each contains a tunable parameter to determine the rate of weight decrease. The methods are piece-wise linear, exponential, and relevant vector dierence decay. A case study has been performed that applied quantitative causality to the well-known software engineering problem of feature location. A summary of the case study results is provided to illustrate the utility of quantitative causality for succeeding events.

Description:
The E-method for the automatic generation of tests for graphical user interfaces (GUIs) is based on Event Sequence Graph (ESG) model of the expected GUI behavior. TheW-method and its variants like theWp method, are based on Finite State Machines (FSM) and can also be used for the same purpose. We show that tests generated using the E- and the W-methods have the same fault detection effectiveness when the FSM distinguishability index is known and used for test generation; in the absence of this knowledge, the ESG method could lead to fewer test cases thereby leaving undetected faults in the GUI under test. Based on the fault detection and modeling characteristics of the two methods, we make recommendations to help a tester decide which method to use in a given scenario. A prototype tool named BEASTT incorporates both the E- and the W-methods.

Description:
The purpose of this paper is to describe an approach to usability testing based on a heuristic expert review model. This model will be the basis for a tool that will enable companies to identify usability levels, potential usability problems and predict full usability test results. This report discusses the research issues behind choosing this method. The process of defining the attributes that are quantified in this model and tool as well as the early testing to validate the tool will be described. This report also discusses the methods by which full usability test results will be predicted from this tool. A technical report, “Miniator: An XML-based Program for Applying Rubrics,” describing the underlying software development of the tool is published separately. At present the Expert Reviewer Tool© (ER tool©) is in prototype release. We are in the early stages of determining the validity of each item in the review (Appendix A), developing the optimal manner to give feedback to developers and the most efficient way to report bugs found during inspection.

Description:
This report contains a survey of the state of the art in software engineering for secure software. Secure software is defined and techniques used in each phase of the software lifecycle to engineer the development of secure software are described. Also identified are open questions and areas where further research is needed.

Description:
Software development has become a globally sourced commodity. Software development processes such as requirements elicitation, development of components and maintenance tasks are often distributed across sites and countries. This research paper focuses on the distribution of the testing process, which is not a straightforward task. The research presented here has examined the management of virtual teams within a distributed testing environment in two multi-national companies in Ireland. This has elicited many issues over and above the ‘socio-cultural’ issues so often discussed. Some of the difficulties encountered include language differences, use of communication tools, different process maturity levels, tools and standards, technical ability and knowledge management. Apart from the management of the testing process, we are also considering the technical issues involved in the global distribution of testing. A related study is being undertaken with teams of students from Masters classes in our two Universities to investigate a global testing infrastructure to identify critical factors that reduce the time to product stability while improving the competitive position of developing organizations. Teams of graduate computer science students in Ireland and in the United States will conduct operational testing on an industrial software system following several scenarios. We will compare various approaches of processing trouble reports and software updates to determine their impact on defects uncovered, the severity levels and types of defects, and the complexity of the modules in which the defects occurred. The results will be analyzed using theoretical models and compared to industry standards for defects at delivery, latent errors and time to product stability

Description:
Access control is the key security service used for information and system security. The access control mechanisms can be used to enforce various security policies, but the desired access control objectives can only be achieved if the underlying software implementation is correct. It therefore becomes essential to not only verify that the implementation conforms to the given policy but also to confirm the absence of any violations in it. We propose a model-based strategy for testing implementations of access control systems that employ the RBAC policy specification. Our approach is based on the construction of a structural and behavioral model of the corresponding RBAC specification. The model is then used to generate static and dynamic test suites for the corresponding implementation. The code coverage and mutation score were used as metrics to determine the efficacy of the proposed approach in a case study. The results of the case study show that the tests generated using the proposed approach not only provide good control flow coverage of the implementation but are also effective in detecting faults induced via mutation operators.

Description:
We present in this paper a new methodology for prioritizing threats rated with ordinal scale values while preserving the meaning of ordinal values and respecting the rules that govern ordinal scales. Our approach is quite novel because we present a formal algebraic system called the k/m algebra to derive the equivalence classes into which threats will be placed and define an operation called k/m dominance which orders the equivalence classes. The operations of our algebra always respect the rules that govern ordinal scales and preserve the meaning of ordinal values. We also describe and present the results from a preliminary case study where we applied our k/m algebra to prioritize threats ranked using data from an existing threat modeling system.

Description:
Software Engineers supporting a large software system often need to locate the code that performs a specific user feature. One method to solve this problem is software reconnaissance, which compares execution traces taken when the feature was active with background execution traces when it was not. Software components executed in the first set but not in the second tend to be involved in the feature of interest. The software reconnaissance method has been tried in a number of contexts and academic software tools, such as the Recon3 toolset, are freely available. However companies might be more willing to apply this method if they could use commercial, industrial-strength tools, of known reliability. This report describes a study performed with Motorola, Inc. to see if Metrowerks CodeTEST and Klocwork inSight could be used for feature location. Both tools are currently in use in Motorola and are know to be robust and effective. CodeTEST is a dynamic analysis tool and can produce traces of execution, while inSight is a static analysis tool which allows browsing and architectural analysis of a large system. The two tools were combined with TraceGraph, a trace comparison tool from the Recon3 toolset, in a case study of four features in a large open-source software system. The study showed that the tool combinations were effective for feature location, though about 180 hours of effort was needed for tool adaptations to get them to work together. Tool integration was still less than optimal, with manual steps being required to get data from one tool to the next. The typical time to locate, understand and document each feature was only about 4 hours. In most cases the software engineer only had to study a few hundred lines out of the more than 200,000 lines making up the system. We conclude that CodeTEST and inSight can be used effectively for feature location. We plan enhancements to the TraceGraph component to improve the ease of use of the combination.

Description:
The need to manually specify temporal properties of software systems is a major barrier to wider adoption of software model checking, because the specification of software temporal properties is a difficult, time-consuming, and error-prone process. To solve this problem, we propose to automatically extract temporal specifications from code. Our approach uses a model checker to check a set of API usage rule candidates against known good programs using that API, and identifies valid rules based on model checking results. These valid rules can be used to verify new programs through the same model checking process. We tested our approach by extracting API usage rules from C programs using BLAST. We successfully extracted OpenSSL API usage rules from three OpenSSL applications in product release and used them to verify an OpenSSL application in beta release.

Description:
The goal of the Eclipse Platform for System Modeling, Design, Testing, and Deployment Project is to explore the integrated Eclipse platform to support full life-cycle software development within Motorola. This research project investigates the capabilities and limitations of the Eclipse platform as a software development environment for modeling, code generation, debugging, and validation. It is expected that this open and integrated platform could improve productivity and drive down engineering cost by bringing together the best-in-class tools to work seamlessly.

Description:
This paper presents a model for estimating the effect of design decisions on software reliability based on design metrics developed in the Software Engineering Research Center (SERC). The paper introduces the concepts of design significance and stress points, and a method to identify and measure these in software. After a brief overview of selected software reliability models, the problem of validating life-critical software is presented. The paper then investigates the proposition that a relationship exists between the design metric D(G) and the defects that are found in the field. A study performed on a subset of a large defense software system provides empirical evidence to support the proposition. The last section of the paper describes a high reliability engineering process that has been developed based on the concepts in this paper. The process is implemented on an active defense software development program.

Description:
There are as many economic benefits as there are problems in developing software in globally distributed locations. One of the most pressing problems is the absence of a globally distributed software development process. One of the specific key areas within that process is software testing. The focus of our research and the supporting environment outlined in this paper is the identification of the essential information and infrastructure required to support effective testing in a globally distributed test environment.

Description:
This paper describes an approach to the feature location problem for distributed systems, that is, to the problem of locating which code components are important in providing a particular feature for an end user. A feature is located by observing system execution and noting time intervals in which it active. Traces of execution in intervals with and without the feature are compared. Earlier experience has shown that this analysis is difficult because distributed systems often exhibit stochastic behavior and because time intervals are hard to identify with precision. To get around these difficulties, the paper proposes a definition of time interval based on the causality analysis introduced by Lamport and others. A strict causal interval may be defined, but it must often be extended to capture latent events and to represent the inherent imprecison in time measurement. This extension is modeled using a weighting function which may be customized to the specific circumstances of each study. The end result of the analysis is a component relevance index, which can be used to measure the likely relevance of a software component to a particular feature. Software Engineers may focus their analysis efforts on the top components as ranked according to this index. Two case studies are presented, a small study of a game program to illustrate the feasibility of the method, and a study of the message logs of a large military software system. Both studies indicate that the suggested approach could be an effective guide for a Software Engineer who is maintaining or enhancing a distributed system.

Description:
Improvements in the software development process depend on our ability to collect and analyze data drawn from various phases of the development life cycle. Our design metrics research team was presented with a large-scale SDL production model plus the accompanying problem reports that began in the requirements phase of development. The goal of this research was to identify and measure the occurrences of faults and the efficiency of their removal by development phase in order to target software development process improvement strategies. Through our analysis of the system data, the study confirms that catching faults in the phase of origin is an important goal. The faults that migrated to future phases are on average ten times more costly to repair. The study also confirms that upstream faults are the most critical faults and more importantly it identifies detailed design as the major contributor of faults, including critical faults. When the entire correction process is accounted for, this project follows the Pareto principle, or the 80/20 rule. However, when observing only downstream activities this ratio is much more extreme, approximating a 95/5 distribution.