Some Preliminaries 2007 Prentice Hall 16-1 Basics of Analysis The process of data analysis Observation Data Encode Informatio n Analysis Example 1: Gift Catalog Marketer Mails 4 times a year to its customers Company has I million customers on its file
2007 Prentice Hall 16-2 Example 1 Cataloger would like to know if new customers buy more than old customers? Classify New Customers as anyone who brought within the last twelve months. Analyst takes a sample of 100,000 customers and notices the following. 2007 Prentice Hall 16-3 Example 1 5000 orders received in the last month
3000 (60%) were from new customers 2000 (40%) were from old customers So it looks like the new customers are doing better 2007 Prentice Hall 16-4 Example 1 Is there any Catch here!!!!! Data at this gross level, has no discrimination between customers within either group. 2007 Prentice Hall A customer who bought within the last 11 days is treated exactly similar to a customer who bought within the last 11 months. 16-5
Example 1 Can we use some other variable to distinguish between old and new Customers? Answer: Actual Dollars spent ! What can we do with this variable? Find its Mean and Variation. We might find that the average purchase amount for old customers is two or three times larger than the average among new customers 2007 Prentice Hall 16-6 Numerical Summaries of data The two basic concepts are the center and the Spread of the data
n Center of data xi x i 1 - Mean, which is given by n - Median - Mode 2007 Prentice Hall 16-7 Numerical Summaries of data Forms of Variation n ( x x) Sum of differences about the mean: i i 1 n
Variance: 2 ( x x ) i i 1 n 1 Standard Deviation: Square Root of Variance 2007 Prentice Hall 16-8 Confidence Intervals In catalog eg, analyst wants to know average purchase amount of customers
He draws two samples of 75 customers each and finds the means to be $68 and $122 Since difference is large, he draws another 38 samples of 75 each The mean of means of the 40 samples turns out to be $ 94.85 How confident should he be of this mean of means? 2007 Prentice Hall 16-9 Confidence Intervals Analyst calculates the standard deviation of sample means, called Standard Error (SE). It is 12.91 Basic Premise for confidence Intervals 95 percent of the time the true mean purchase amount lies between plus or minus 1.96 standard errors from the mean of the sample means. C.I. = Mean (+or-) (1.96) * Standard Error
2007 Prentice Hall 16-10 Confidence Intervals However, if CI is calculated with only one sample then Standard Error of sample mean = Standard deviation of sample n Basic Premise for confidence Intervals with one sample 95 percent of the time the true mean lies between plus or minus 1.96 standard errors from the sample means. 2007 Prentice Hall 16-11 Example 2: Confidence Intervals for response rates
You are the marketing analyst for Online Apparel Company You want to run a promotion for all customers on your database In the past you have run many such promotions Historically you needed a 4.5% response for the promotions to break-even You want to test the viability of the current fullscale promotion by running a small test promotion 2007 Prentice Hall 16-12 Example 2: Confidence Intervals for response rates Test 1,000 names selected at random from a new list. To break-even the list must be expected to have a response rate of 4.5 percent Confidence Interval= Expected Response (+/-) 1.96*SE
= p(+/-) 1.96*SE In our case C.I. = 3.22 % to 5.78%. Thus any response between 3.22 and 5.78 % supports hypothesis that true response rate is 4.5% 2007 Prentice Hall 16-13 Example 2: Confidence Intervals for response rates The list is mailed and actually pulls in 3.5% Thus, the true response rate maybe 4.5% What if the actual rate pulled in were 5% ? Regression towards mean: Phenomenon of test result being different from true result Give more thought to lists whose cutoff rates lie within confidence interval 2007 Prentice Hall 16-14
Frequency Distribution and Cross-Tabulation 2007 Prentice Hall 2007 Prentice Hall 15 16-15 Chapter Outline 1) Frequency Distribution 2) Statistics Associated with Frequency Distribution i. Measures of Location ii. Measures of Variability iii. Measures of Shape 3) Cross-Tabulations i. Two Variable Case ii. Three Variable Case
iii. General Comments on Cross-Tabulations 4) Statistics for Cross-Tabulation: Chi-Square 2007 Prentice Hall 16-16 Internet Usage Data Table 15.1 Respondent Number Sex 1 2 3 4 1.00 2.00 2.00 2.00 1.00 2.00 2.00 2.00 2.00 1.00 2.00 2.00 1.00
2.00 2.00 2.00 16-17 Frequency Distribution In a frequency distribution, one variable is considered at a time. A frequency distribution for a variable produces a table of frequency counts, percentages, and cumulative percentages for all the values associated with that variable. 2007 Prentice Hall 16-18 Frequency Distribution of Familiarity with the Internet Table 15.2 Value label Not so familiar Very familiar Missing Value
6 5 4 3 2 1 0 2007 Prentice Hall 2 3 4 Familiarity 5 6 7 16-20 Statistics for Frequency Distribution: Measures of Location The mean, or average value, is the most commonly used measure of central tendency. The X mean, ,is given byn X = X i /n
i=1 Where, Xi = Observed values of the variable X n = Number of observations (sample size) The mode is the value that occurs most frequently. The mode is a good measure of location when the variable is inherently categorical or has otherwise been grouped into categories. 2007 Prentice Hall 16-21 Statistics for Frequency Distribution: Measures of Location The median of a sample is the middle value when the data are arranged in ascending or descending order. If the number of data points is even, the median is the midpoint between the two middle values. The median is the 50th percentile. 2007 Prentice Hall
16-22 Statistics for Frequency Distribution: Measures of Variability The range measures the spread of the data. The variance is the mean squared deviation from the mean. The variance can never be negative. The standard deviation is the square root of the variance. The coefficient of variation is the ratio of the standard deviation to the mean expressed as a percentage, and is aCV unitless = sx /X measure of relative variability. 2007 Prentice Hall 16-23 Statistics for Frequency Distribution: Measures of Shape
Skewness. The tendency of the deviations from the mean to be larger in one direction than in the other. Tendency for one tail of the distribution to be heavier than the other. Kurtosis is a measure of the relative peakedness or flatness of the frequency distribution curve. The kurtosis of a normal distribution is zero. -kurtosis>0, then dist is more peaked than normal dist. -kurtosis<0, then dist is flatter than a normal distribution. 2007 Prentice Hall 16-24 Skewness of a Distribution Fig. 15.2 Symmetric Distribution Skewed Distribution Mean Media n Mode (a) 2007 Prentice Hall Mean Median
Mode (b) 16-25 Cross-Tabulation While a frequency distribution describes one variable at a time, a cross-tabulation describes two or more variables simultaneously. Cross-tabulation results in tables that reflect the joint distribution of two or more variables with a limited number of categories or distinct values, e.g., Table 15.3. 2007 Prentice Hall 16-26 Gender and Internet Usage Table 15.3 Gender Internet Usage Light (1) Heavy (2) Column Total 2007 Prentice Hall Male
Female Row Total 5 10 15 10 5 15 15 15 16-27 Two Variables CrossTabulation Since two variables have been cross-classified, percentages could be computed either columnwise, based on column totals (Table 15.4), or rowwise, based on row totals (Table 15.5).
The general rule is to compute the percentages in the direction of the independent variable, across the dependent variable. The correct way of calculating percentages is as shown in Table 15.4. 2007 Prentice Hall 16-28 Internet Usage by Gender Table 15.4 Gender I nternet Usage 2007 Prentice Hall Male Female Light 33.3% 66.7% Heavy 66.7% 33.3% Column total
100% 100% 16-29 Gender by Internet Usage Table 15.5 Internet Usage Gender Light Heavy Total Male 33.3% 66.7% 100.0% Female 66.7% 33.3% 100.0%
2007 Prentice Hall 16-30 Introduction of a Third Variable in Cross-Tabulation Fig. 15.7 Original Two Variables Some Association between the Two Variables No Association between the Two Variables Introduce a Third Variable Introduce a Third Variable Refined No Association Association between the Two between the Two Variables Variables 2007 Prentice Hall
No Change in the Initial Pattern Some Association between the Two Variables 16-31 3 Variables Cross-Tab: Refine an Initial Relationship As can be seen from Table 15.6, 52% (31%) of unmarried (married) respondents fell in the highpurchase category Do unmarried respondents purchase more fashion clothing? A third variable, the buyer's sex, was introduced As shown in Table 15.7, - 60% (25%) of unmarried (married) females fell in the high-purchase category - 40% (35%) of unmarried (married) males fell in the high-purchase category. Unmarried respondents are more likely to fall in the high purchase category than married ones, and this effect is
much more pronounced for females than for males. 2007 Prentice Hall 16-32 Purchase of Fashion Clothing by Marital Status Table 15.6 Purchase of Fashion Clothing Current Marital Status Married Unmarried High 31% 52% Low 69% 48% 100% 100%
700 300 Column Number of respondents 2007 Prentice Hall 16-33 Purchase of Fashion Clothing by Marital Status and Gender Table 15.7 Purchase of Fashion Clothing Sex Male Female Married Not Married Married Not Married
High 35% 40% 25% 60% Low 65% 60% 75% 40% Column totals Number of cases 100% 100% 100% 100% 400
120 300 180 2007 Prentice Hall 16-34 3 Variables Cross-Tab: Initial Relationship was Spurious Table 15.8 shows that 32% (21%) of those with (without) college degrees own an expensive automobile Income may also be a factor In Table 15.9, when the data for the high income and low income groups are examined separately, the association between education and ownership of expensive automobiles disappears,
Initial relationship observed between these two variables was spurious. 2007 Prentice Hall 16-35 Ownership of Expensive Automobiles by Education Level Table 15.8 Own Expensive Automobile Education College Degree No College Degree Yes 32% 21% No 68% 79% 100% 100%
250 750 Column totals Number of cases 2007 Prentice Hall 16-36 Automobiles by Education Level and Income Levels Table 15.9 Income Own Expensive Automobile Low Income High Income College Degree No College Degree College Degree
No College Degree Yes 20% 20% 40% 40% No 80% 80% 60% 60% Column totals 100% 100% 100% 100% 100
700 150 50 Number of respondents 2007 Prentice Hall 16-37 3 Variables Cross-Tab: Reveal Suppressed Association Table 15.10 shows no association between desire to travel abroad and age. In Table 15.11, sex was introduced as the third variable. Controlling for effect of sex, the suppressed association between desire to travel abroad and age is revealed for the separate categories of males and females. Since the association between desire to travel abroad and age runs in the opposite direction
for males and females, the relationship between these two variables is masked when the data are aggregated across sex as in Table 15.10. 2007 Prentice Hall 16-38 Desire to Travel Abroad by Age Table 15.10 Desire to Travel Abroad Age Less than 45 45 or More Yes 50% 50% No 50% 50% Column totals
100% 100% 500 500 Number of respondents 2007 Prentice Hall 16-39 Desire to Travel Abroad by Age and Gender Table 15.11 2007 Prentice Hall 16-40 Tabulations No Change in Initial Relationship Consider the cross-tabulation of family size and the tendency to eat out frequently in fast-food restaurants as shown in Table 15.12. No association is observed.
When income was introduced as a third variable in the analysis, Table 15.13 was obtained. Again, no association was observed. 2007 Prentice Hall 16-41 Fast-Food Restaurants by Family Size Table 15.12 2007 Prentice Hall 16-42 Restaurants by Family Size and Income Table 15.13 2007 Prentice Hall 16-43 Statistics Associated with Cross-Tab: Chi-Square H0: there is no association between the two variables
Use chi-square statistic H0 will be rejected when the calculated value of the test statistic is greater than the critical value of the chi-square distribution 2007 Prentice Hall 16-44 Statistics Associated with Cross-Tab: Chi-Square 2 compares the of the observed cell frequencies (fo) to the frequencies to be expected when there is no association between variables (fe) The expected frequency for each cell can be calculated by using a simple formula: nr nc fe n nr=total number in the row
nc=total number in the column n=total sample size 2007 Prentice Hall 16-45 Statistics for Cross-Tab: ChiSquare From Table 3 in the Statistical Appendix, the probability of exceeding a chi-square value of 3.841 is 0.05. The calculated chi-square is 3.333. Since this is less than the critical value of 3.841, the null hypothesis can not be rejected Thus, the association is not statistically significant at the 0.05 level. 2007 Prentice Hall 16-46
The Other Prodigal Son. Luke 15:1-2. Tax collectors and other notorious sinners often came to listen to Jesus teach. This made the Pharisees and teachers of religious law complain that he was associating with such sinful people—even eating with them!
PDS Initiative. AT&L tasked DPAP (June 5, 2007) to develop a procurement data strategy for DoD. The Procurement Data Standard (PDS) is a system-independent data standard that is intended to be adopted and implemented DoD-wide for creation, translation, processing, and...
ee: /opt/apps/samples/training. This is how to view and manipulate Slurm. See the Web site for additional information on queues/partitions including what resource limits are set. A limited set of PBS/Torque commands such as qstat and qsub are available. 10/3/17. Template-WSU...
Members who joined the system on or after July 1, 2007, become vested in the system with eight (8) years of creditable service. Employees who joined the system on or before June 30, 2011 are eligible to retire at any...
Nonsalicylates: Contraindications and Precautions . Contraindicated in patients with hypersensitivity. Used cautiously in patients: With severe or recurrent pain or high or continued fever. Acetaminophen used cautiously during pregnancy and lactation
Practice-based Learning & Higher Education : Let the learners 'lead the way' - re-introducing independent learning & development. Nick Rowe - Institute of Health & Community Studies, Bournemouth University - U.K.