Product Development Systems & Solutions Inc.-click to homepage
News from PDSS Inc.
"Leading the Future in Product Development" 
August 2015- Vol 8, Issue 8
In This Issue
Data Integrity: Sample Size (2 of 3)
Hypothesis Formation & Testing: A Tutorial
We continue with part 2 of 3 of the summer series on data integrity in technology and product development. This article is about the importance of using appropriate sample sizes and the trade-offs that must be examined to ensure the integrity of your data. The final article in the series will be about adjustable prototypes. See you in September!
-Carol
Data Integrity: Sample Size (2 of 3)
This series covers three interrelated topics as they affect the integrity of functional response data during development of technology and product performance capability. They are:
  1. Measurement Systems
  2. Sample Size
  3. Adjustable Prototypes
 
Last month addressed measurement systems. In this article, we address sample size: collecting the appropriate amount of data to learn about the parametric relationships that control the base-line and robust (stressed) functional performance of our systems, subsystems and subassemblies. Sample size is important because:
  1. It establishes a context for stating the risk associated with an assertion (hypothesis) we are testing.
  2. It commits material and human resource costs to the project for running Characterization Studies, Designed Experiments or t-Tests.
  3. It commits time to the project schedule for conducting the afore-mentioned tasks.
We begin with modeling and simulation (M&S) to tell us which parameters are important and well-behaved (we call them ECO: Easy, Common & Old) and which parameters are candidates for critical status because they are not well-understood (we call them NUD: New, Unique & Difficult). Physical prototypes must be used at some point after initial math modeling and Monte Carlo simulations have obtained reasonable estimates of how the design parameters work together to control the NUD Y as a function of X relationships. Iterative cycles of M&S, physical experimentation and verification of converging agreement between the analytical and empirical models must be done.
Sample sizes are easy to select during M&S when Monte Carlo simulations are conducted on math models. Many thousands of cycles can be run cost-effectively based upon random iterations of the probabilistically exercised "Y as a function of X" relationships that underwrite the integrity of our expressions. These are entirely based upon the laws of physics/chemistry, etc. When we move on to design, build, instrument and use adjustable, modular prototypes, the topic of sample "runs" is more challenging. It is here where risk, cost and time are negotiated between the technical team and the project manager.
Larger sample sizes drive our resolving power to see engineering-significant differences from experimental results. Sample sizes that are too small risk not detecting a meaningful change when you need to! You could miss the truth about a set point change that might be significant for your design. This is particularly true when the differences are subtle. Understanding the metrics of adjustability, hyper-sensitivity and X-variable contributions to Y-variable behavior, in light of independence and interactivity, may be compromised.
In summary, by settling for too-small sample sizes, you can end up misleading yourself, your technical team, the project manager and your upper management team responsible for your business. The lack of technical understanding of potential critical parameters can significantly affect reliability, safety and customer satisfaction. In a word - you miss the chance to prevent a real problem that will hide until later.
Cost, Project Cycle Time & Sample Sizing
The issue of sample sizes and their effect on prototype costs, experimental time durations, and resource costs can be contentious. Studying design capability performance and identifying critical parameters without using appropriate sample sizes is a setup for design scrap and re-work cycles. If you invest in proper definition of sample sizes in the context of understanding the truth about your null and alternative hypotheses based upon 95% confidence intervals (see tutorial, below, for an explanation of these concepts), you are pro-actively preventing down-stream development re-work. If you restrict sample sizes and force developmental learning short-falls, your schedule will likely slip later because the lack of learning at the right stage of the development process. You will force the technical team to revisit the design's surprise difficulties that have accumulated and revealed themselves late in the development cycle. The only time it is more costly and damaging to your business is when the under-developed product gets released to unsuspecting customers.
Doing things right the first time in technology and product development heavily depends on the proper determination of sample sizes based upon reasonable confidence intervals (typical between 90-95-99%). Ask yourself; can I stand behind my statements in a design review or am I pretending to know something when, in fact, I simply have rushed past realistic sample sizes that properly forecast future functional behavior?
HypothesisHypothesis Formation & Testing: A Tutorial
Establishing risk during developmental learning cycles begins with Hypothesis Formation and Testing. A hypothesis is an assertion that something is true or not. For example, the effect of a variable is significant (or not) and thus the state of expected performance is the same (or not) as controlling variables are changed. A hypothesis is expressed as follows:
  • The Null Hypothesis is the assertion of "NO DIFFERENCE". It is represented using the symbol Ho.
  • The Alternative Hypothesis is the assertion of "A DIFFERENCE EXISTS". It is represented using the symbol Ha.
The risk typically assigned to hypotheses is that we want to be right about them 95% of the time. Conversely, we will accept the risk of being wrong about our assertion 5% of the time. This is called Alpha Risk.
A Confidence Interval is a range of values which represents the future population behavior we are forecasting with our samples of experimental data. The magnitude (width) of the confidence interval of our design's functional performance is directly affected by the sample size we take.
There are two types of error that can occur during developmental learning, known as Type 1 and Type 2 Error. Type 1 Error is associated with Alpha risk and happens when we incorrectly reject the null hypothesis, although it is actually true. An example of Type 1 Error is when we run an experiment and use the results to say that changing a set point on an X variable significantly changed our Y variable. This is a "false positive" and is called Manufacturer's (Experimenter's) Risk. We want to make this mistake a rare occurrence. So we run our experiments in such a way that our confidence interval of being right about an assertion being correct 95% of the time is narrow. Type 2 Error is the opposite of Type 1 error, where we run the risk of saying a difference does not exist when it really does, which is incorrectly accepting the null hypothesis! The risk for making this mistake is typically set at 20% and is called Beta Risk. That means we want to be right for this assertion 80% of the time in the context of avoiding a Type 2 Error. This is called Consumer's (or Management/Design Reviewer's) Risk.
Other factors used by software statistical packages such as MiniTab or JMP to compute recommended sample sizes include the standard deviation history or an estimate for the standard deviation, the engineering-significant difference we want to see as a meaningful difference in Y from our experimental exploration of set point changes in X(s), the Alpha risk (typically set a 0.05), and the Power of the Test expressed as 1-Beta. These are necessary inputs to enable the software to provide a reasonable recommendation for a sample size you need.
 
 
Is there a topic you'd like us to write about? Have a question? We appreciate your feedback and suggestions! Simply "reply-to" this email. Thank you!
  
Sincerely,
Carol Biesemeyer
Business Manager and Newsletter Editor
Product Development Systems & Solutions Inc.
About PDSS Inc.
Product Development Systems & Solutions (PDSS) Inc.  is a professional services firm dedicated to assisting companies that design and manufacture complex products.  We help our clients accelerate their organic growth and achieve sustainable competitive advantage through functional excellence in product development and product line management.
  
Copyright 2015, PDSS Inc.
Join Our Mailing List!
 
See PDSS Inc.'s Archived E-Newsletters