More MIT Sloan Home Search Sitemap
Organization Studies Group
ABOUT US
METHODS & RESEARCH
PUBLICATIONS
DEMONSTRATIONS
DOWNLOAD CODE
SPONSORSHIP
TEAM
HOME

 

For a summary the Virtual Customer methods developed prior to 2002, including applications and critical comparisons, please see The Virtual Customer (PDF). For a review of how Virtual Customer methods are used in the product-development process, please see Product Development: Managing a Dispersed Process (PDF), which is a chapter in the Handbook of Marketing. For a review of research on innovation see Research on Innovation: A Review and Agenda for Marketing Science. For a review of conjoint analysis see Conjoint Analysis, Related Modeling, and Applications. However, many of the methods below are new and have not yet made it into these review articles.

FastPace Family of Conjoint Analyses

Adaptive Choice-based Conjoint Analysis (with Polyhedral methods)
In choice-based conjoint analysis, customers are presented with a set of products, described by features, and asked to simply choose the product they prefer. When this task is repeated a sufficient number of times, we can estimate the value that each customer places on the features. Existing methods use a fixed set of choice sets or undertake an initial pretest to improve the fixed set of choice sets. By using polyhedral methods we have developed a method by which the choice sets for each respondent are adapted efficiently based on that respondent's choices. This method has been tested with simulation and has been applied and tested empirically to the design of new executive education programs. For more details see Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis (PDF).

Analytic Center Estimation
Web-based methods often deal with complex products but try to ask very few questions. Otherwise, the respondent loses interest and does not complete the questionnaire. If the questions are chosen efficiently, analytic-center (AC) estimation provides an effective method to summarize the information in terms of parameters (partworths) that describe customer preferences. AC estimation provides an alternative to Hierarchical Bayes (HB) estimation in some situations. For more details see Fast Polyhedral Adaptive Conjoint Estimation (PDF) for metric-pairs data and Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis for choice-based data.

FastPace (Fast Polyhedral Adoptive Conjoint Estimation) Question Design
Web-based respondents are impatient. FastPace is a question-selection method that selects metric paired-comparison preference questions efficiently for maximal information content. It can be used with a variety of estimation methods. Both Monte Carlo simulations and empirical tests suggest that it offers improvements relative to existing fixed and adaptive methods. See Fast Polyhedral Adaptive Conjoint Estimation (PDF) for theory, application, and simulations. For demos see Virtual Customer Research Methods.

Generalized Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis
While polyhedral methods for choice-based conjoint analysis provide a means to adapt choice-based questions at the individual-respondent level and provide an alternative means to estimate partworths, these methods are deterministic and are susceptible to the propagation of response errors. They also assume, implicitly, a uniform prior on the partworths. In this paper we provide a probabilistic interpretation of polyhedral methods and propose improvements that incorporate response error and/or informative priors into both individual-level question design and estimation. These improvements can also be used to incorporate population information into individual-level estimates in an analogy to shrinkage estimates. Simulations suggest that response-error modeling and informative priors improve polyhedral methods in the domains were they were previously weak. The empirical context, over 2600 leading-edge wine consumers in the US, Australia, and New Zealand, suggests that the new methods have good predictive ability relative to existing methods. See Generalized Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis. For a demo see Generalized Polyhedral Methods for Wine Closures.

Support Vector Machines for Robust Conjoint Estimation and Question Selection
When preferences are highly non-linear (many interactions) or when data are particularly noisy, many conjoint methods are challenged. By using concepts from machine learning, the authors are able to develop a robust method that handles non-linearities and noise, yet still provides practical estimates. This method is particularly useful for describing user's choices as revealed by clicks on the Internet and for estimating conjoint models that include many interactions among features. For estimation, see Generalized Robust Conjoint Estimation (PDF). For question design see An Optimization Framework for Adaptive Questionnaire Design. For new approaches to heterogeneity see Optimization Conjoint Models for Consumer Heterogeneity.

Cards Family of Preference Measurement and Conjoint Analysis

GARDS (Non-compensatory Measurement: Must-Have Features) Practical methods to test whether respondents use non-compensatory processes and, if so, to infer the details of those processes from either consideration-then-choice or full-rank tasks. "Greedoid languages" provide a structure and theory to transform this problem and decrease estimation time by a factor of approximately 109 for practical 16-aspect problems. Monte Carlo experiments suggest that it is feasible to infer, albeit with noise, the process that respondents use to evaluate product profiles. Greedoid methods also provide a non-compensatory conjoint-like method to forecast consumer response and to find the minimum feature levels required to enter a product category. This method is particularly useful in categories where consumers are presented with large numbers of potential choices. Not only do Greedoid estimates appear to predict better (in most cases) than purely compensatory estimates, but the consideration-then-choice task is perceived by respondents as more enjoyable, more accurate, and more interesting. It also saves time and increases completion rates — both of which translate directly into cost savings. See Greedoid-Based Non-compensatory Two-Stage Consideration-then-Choice Inference. For an alternative approach based on a metric representation see Representation of Lexicographic Preference Models and their Variants.

CARDS (Conjoint Adaptive Ranking Database System)
Full-profile conjoint analysis remains an important data collection format with a proven track record of over thirty years. CARDS is the first adaptive method in which respondents are only asked to choose among those sets of profiles that are necessary to identify a full rank ordering. The method is based on data tries which recognize that only a small subset of all possible rank orders are consistent with utility maximization. For a demo see Cards Demo.

New Insights on Conjoint Measurement

Managerial Efficiency (M-Efficiency)
In many managerial situations some decisions are more critical than others and the researcher may wish to focus precision on those combinations of features of greater managerial interest. For example, in a conjoint analysis to support product design, some features may be critical and irreversible while others might be easy to change or vary after product launch. However, standard measures of precision (efficiency) do not provide differential focus. We propose alternative "managerial efficiency" criteria (M-errors) and explore their properties. We provide examples where managerial efficiency can be improved with a slight reduction in standard measures (A-errors or D-errors). We demonstrate that orthogonal designs are not always orthogonal in the estimates of managerial interest (M-orthogonal) nor vice versa. Further, unlike for standard criteria, experimental designs that minimize M-errors are not M-orthogonal nor vice versa. To address this issue, we propose modified managerial criteria to balance M-orthogonality and M-efficiency. Examples illustrate all concepts. Indeed, in one example, the M-criteria suggest that the price levels be set closer to feature costs than at extreme levels — the latter a common "best practice." Existing algorithms are readily adopted to the managerial criteria, however, we propose, in addition, a new algorithm that provides useful starting solutions which are closer to M-efficient and M-orthogonal designs. In a final section we illustrate how the concepts extend to choice data. See Note on Managerial Efficiency.

The Impact of Utility Balance on Adaptively Chosen Questions
Adaptive metric utility balance is at the heart of one of the most widely used and studied methods for conjoint analysis. We use formal models, simulations, and empirical data to illustrate how adaptive metric utility balance leads to smaller partworths being upwardly biased relative to larger partworths. In real problems such relative biases could lead to erroneous managerial decisions. Metric utility-balanced questions are also more likely to be inefficient and lead to response errors that are at least as large as non-adaptive orthogonal questions. This bias is due to endogeneity caused by a "winner's curse," and shrinkage estimates do not mitigate these biases. Combined with adaptive metric utility balance, shrinkage estimates of heterogeneous partworths lead to additional biases. Although biases are of the order of response errors, these biases can be avoided. We examine viable alternatives that researchers can use without biases or inefficiencies to retain the desired properties of (1) individual-level adaptation and (2) challenging questions. See The Impact of Utility Balance and Endogeneity in Conjoint Analysis.

Idea Generation

Idea Generation, Creativity, and Incentives
Idea generation (ideation) is critical to the design and marketing of new products, to marketing strategy, and to the creation of effective advertising copy. We describe a practical, Web-based, asynchronous "ideation game," which allows the implementation and test of various incentive schemes. This method has been used successfully to identify ideas for product development in three applications to date. Experimental testing suggests that the ideation game leads to more ideas and ideas that are better. See Idea Generation, Creativity, and Incentives.

Information Pump
The information pump uses either a Web-based interface or a paper-and-pencil interface to enable customers to talk to one another in a "parlor game" that provides incentives for truth-telling and thinking hard. This method literally pumps information from the customers to uncover the words and phrases that they use to describe products, advertising, or other concepts. It is ideal for identifying "unarticulated needs." For a description of the method and a theoretical paper, see the Information Pump Readings Packet (PDF).

Web-Based Advisors

Trusted Advisors and Listening In
By "listening in" to ongoing dialogues between customers and Web-based virtual advisors (e.g., Kelley Blue Book's Auto Choice Advisor) we identify new product opportunities based on new combinations of customer needs. These data are available at little incremental cost and provide the scale necessary for complex products (e.g., 148 trucks and 129 customer needs in our application). We describe and evaluate the methodologies with formal analysis, Monte Carlo simulation (calibrated on real data), and a "proof-of-concept" application in the pickup-truck category (over 1,000 Web-based respondents). The application identified opportunities for new truck platforms worth approximately $2.4-3.2 billion and $1-2 billion, respectively. See "Listening In" to Find Unmet Customer Needs and Solutions.

Truth Telling

Information Scoring
Information scoring is a fully-subjective scoring rule for non-verifiable judgments. For example, respondents might be asked to indicate the purchase likelihood of a new product, but we cannot verify the accuracy of those judgments. By asking respondents to provide a judgment (e.g., a probability) and to predict the distribution of others' judgments, information scoring defines a zero-sum game in which the true judgments and true predictions are the best strategies for respondents. Early tests demonstrate that the scoring rule is feasible and understood by respondents and that rewards are indeed maximized for "truth." Empirical testing is proceeding. See A Bayesian Truth Serum for Subjective Data.

Incentive Compatible Games
This research is attempting to develop an incentive compatible game in which respondents buy and sell information as they attempt to discern one another's preferences. The goal is to align incentives so that respondents will seek to supply information to one another so that their partners can accurately predict their answers. This research has just begun, so there is no working paper yet available.

Other Web-Based Methods

Stock-market-like Trading Games
By treating either products or product-features as securities, we have been able to set up a trading market in which respondents buy and sell these virtual concepts much like they would in a stock market. Early indications suggest strong internal consistency and convergent validity with other methods. For more information see Security Trading of Concepts (PDF).

Configurators for Market Research
The interactivity of the Web is enabling users to design their own virtual products thus enabling the product development team to understand complex feature interactions. In "user design" respondents drop and drag features to or from products (The Virtual Customer) (PDF). In the "design palette" respondents use menus to design their own products (Listening In). In both methods the virtual products change automatically. With "innovation toolkits" customers can invent their own products (Shifting Innovation to Users via Toolkits) (PDF). We have completed additional research identifying how configurator methods are best linked to managerial decisions. This research paper will be ready soon.

Genetic Algorithms
In a genetic algorithm the features of a product are treated like "genes." Consumers indicate their preferences for potential product concepts. Based on these preferences new concepts are developed automatically by "mating" the current concepts. Progeny concepts are based on the genes of their parents with some mutations. We are currently working with a commercial firm to validate the method. See Affinnova.com.

Virtual Concept Testing
By using the multimedia capabilities of the Web we have been able to gather information effectively on virtual product concepts. For more information see The Predictive Power of Internet-Based Product Concept Testing Using Visual Depiction and Animation (PDF). For a demo see Bike Pumps.

Web-based Conjoint Analysis
Conjoint analysis is, perhaps, the most widely used method to measure consumer preferences among alternative product features. We have developed a viable means to move this to the Web. For demos see Crossover vehicle and Laptop Bags and Ski resort (full profile methods) and Cameras (paired comparison methods).



Jim Orlin
MIT Sloan Professor Jim Orlin has been applying new ideas in operations research to consumer measurement.
Read the working paper >>