Two response variables that look identical and aren’t
, the thing I was trying to predict was a survey answer: a senior citizen’s stated intention to engage with a brand’s social media. Today, in ad measurement, the thing I predict is engagement itself, logged as a click or a purchase.
Same word, near enough. But one of those variables lives only in a person’s head and has to be reconstructed from indirect evidence, and the other is sitting in a database waiting to be queried. Almost every modeling decision downstream follows from that one difference. I have now built models on both sides, often with the same regression equations underneath, and what keeps surprising me is how much inverts the moment you cross from one to the other: where variables come from, what makes a variable “good,” what you are even trying to learn, and when you are allowed to stop.
This piece walks through both, with code for the parts that are easier shown than described. If you have only ever modeled behavioral data, the first half is the world you skipped. If you came from social science, the second half is the one you are walking into.
The model I built then
One of my dissertation projects modeled why older adults do or don’t engage with adaptive clothing brands online. The backbone was the Technology Acceptance Model (Davis, 1989), so the predictors were constructs like perceived usefulness, perceived ease of use, brand awareness, brand interaction, brand trust, and privacy concerns, all pointing at intention to engage.
The hard part is hiding in that list. None of those predictors is a column you can select. Perceived usefulness is a state of mind. So is privacy concern. You cannot log it, and you cannot get it cleanly from one survey question either, because any single question carries its own baggage: how it is worded, whether people answer honestly, how each respondent reads it. Ask one question and you measure all of that noise along with the thing you wanted.
The standard fix is to ask several questions that circle the same construct, then let the shared variance across them stand in for the construct and treat the rest as measurement error. That buys you a new problem, though. If you just average the items and drop the average into a regression, the measurement error rides along and quietly biases your coefficients. Structural equation modeling exists largely to deal with this. It fits two linked models at once: a measurement model that asks how well each cluster of survey items reflects its construct, and a structural model that estimates how the constructs relate, with the measurement error modeled rather than ignored.
Concretely, the specification for a piece of my model looked like this (using `semopy`, a Python SEM library, with synthetic variable names):
from semopy import Model
# `=~` defines a latent construct from its survey indicators (measurement model)
# `~` defines a structural path between constructs
model_spec = """
# measurement model: each construct built from its items
privacy_concern =~ privacy1 + privacy2 + privacy3
usefulness =~ useful1 + useful2 + useful3
intention =~ intent1 + intent2 + intent3
# structural model: what predicts intention
intention ~ privacy_concern + usefulness
"""
model = Model(model_spec)
model.fit(survey_df)
estimates = model.inspect() # loadings, path coefficients, p-values, standard errorsThat `=~` operator is the whole difference between the two worlds in one symbol. Before I can ask whether privacy concern predicts intention, I first have to build privacy concern out of `privacy1`, `privacy2`, `privacy3` and prove the construction holds. Inspecting `estimates` is where you check that: each item’s loading on its construct, plus reliability and validity metrics (Cronbach’s alpha, composite reliability, average variance extracted) that tell you whether the items really hang together.
Because every construct costs you statistical power, you cannot be generous with them. My usable sample was small (older adults who both knew adaptive clothing and were active on social media is not a large group), and each construct needs several indicators and enough respondents to support them. Adding a predictor is expensive. So theory, not data, decides what goes in. That is roughly the opposite reflex to “throw in the features and let regularization sort them out.”
The variable that misbehaved
Privacy concern is the one I remember best, because it refused to cooperate twice.
I expected it to matter a lot. For this group, engaging with an adaptive clothing brand can mean revealing something about a disability, so on paper privacy is exactly the kind of thing that should hold people back. When I looked at the indicators, one of the three behaved differently from the others: it loaded negatively, pulling against the construct instead of with it. That item was measuring a different flavor of privacy than the other two, so I dropped it and re-estimated with the two that cohered. In ML vocabulary, I had a feature that was a bad proxy for the thing I was trying to capture, and I cut it.
Then the bigger surprise. In the final model, privacy concern did not significantly predict intention. Neither did brand trust. The predictors that carried the result were perceived usefulness, ease of use, brand awareness, and brand interaction. The variable I was most sure about contributed the least. I reported the null and moved on, which is the part that turned out to be good training, for reasons I will come back to.
The model I build now
These days I work in measurement science for advertising, across a stack that includes incrementality and lift measurement at the geo and user level, quasi-experimental designs like difference-in-differences, propensity and uplift models, and synthetic-control-style counterfactuals. The shape of the model follows the question. “Who is likely to convert” is usually a binary classifier fed by behavioral and transactional features. “Did this campaign actually cause sales” is a different animal, built on treatment and control groups and an estimate of lift.
The first thing that flips: the variables already exist. A click is a column. A purchase is a column. There is no measurement model to fit first, no validity check standing between me and the regression, because nobody is reconstructing a hidden state from noisy items. The working assumption is just that recent behavior is a decent signal of near-future behavior, which is one of the better-supported regularities in behavioral research (Ouellette & Wood, 1998).
A propensity model in this world is close to ordinary supervised learning:
import xgboost as xgb
# features are observed behaviors: recency, frequency, monetary value,
# category views, prior purchases, etc. (synthetic example)
features = ["recency_days", "freq_90d", "monetary_90d",
"category_views_30d", "prior_category_purchase"]
dtrain = xgb.DMatrix(X_train[features], label=y_train) # y = converted (0/1)
params = {"objective": "binary:logistic", "eval_metric": "auc"}
booster = xgb.train(params, dtrain, num_boost_round=300)Notice what is missing compared to the SEM block: no construct definitions, no loadings to validate, no `=~`. The features are taken at face value.
But the thing I spent a PhD learning does not actually go away here; it just goes unspoken. Every one of those features is still a proxy for something I cannot see. `category_views_30d` is a stand-in for interest. `prior_category_purchase` is a stand-in for intent. The difference is that in industry nobody asks me to prove the proxy is any good. The assumption rides for free until something breaks, like a metric getting gamed or a model happily optimizing a proxy that has come loose from the outcome it was supposed to represent. When I see that happen, it looks exactly like my negatively loading privacy item, a measurement caught capturing the wrong thing, except now it surfaces in production instead of in a validity table.
The second thing that flips is the binding constraint. In the dissertation, data was scarce and every variable cost me power. In industry, rows are essentially free, and the things that used to be hard, like getting a clean approximately-normal sampling distribution, mostly take care of themselves at scale. What gets hard instead is that the data is messy in ways survey data never was, and that everything has a price attached. Ad measurement runs on incremental return on ad spend, so “this segment converts well” is not a finding I can stop at. I have to show that acting on it pays for itself, usually across more than one campaign. The question is not only whether an effect is real but whether it is real and worth the spend.
So variable selection inverts too. In the survey world, predictors enter because theory argues for them, and sample size caps how many you can afford. In the behavioral world, features enter because they help prediction, until they start interfering with each other.
One phenomenon, two opposite verdicts
Here is the part that I find genuinely funny now. The exact statistical property that signals a healthy survey model is the one I have to police against in a behavioral model: correlated inputs.
In SEM, the items inside a construct are supposed to correlate. That is what tells me `privacy1`, `privacy2`, `privacy3` are really measuring one underlying thing; high internal correlation shows up as good reliability. (Different constructs still need to stay distinct from each other, but within a construct, togetherness is the goal.) In a propensity model, that same redundancy among features is a headache: it destabilizes linear coefficients, and in tree models it smears feature importance across whichever correlated feature got picked first.
The reconciliation is that the two worlds want different things from the model. The survey model is built to be interpreted, so correlated inputs that muddy interpretation are bad. Multicollinearity mostly damages explanation, not raw prediction; a classifier can predict fine with redundant features, which is why I usually only chase it down when someone needs to read the model, not just run it. That distinction, prediction versus explanation, turns out to be the whole story.
What carried over unchanged
I went in expecting industry to throw out the academic rulebook. A lot of it transferred intact.
You still commit to the question before you see the answer. I fixed my hypotheses and model structure before collecting a single survey response and could not move them afterward. Good measurement work is the same: you specify the design and the metric up front and you live with what comes back. The thing enforcing the discipline changed from a committee to a budget, but the discipline is identical.
You still write down what could be wrong. Papers call it a limitations section; readouts call it caveats. Same act.
And methods still need a justification you can defend. I assumed industry would let me try anything that worked but lift estimation leans on its own body of theory, just a statistical one rather than a psychological one: randomization, the logic of a credible counterfactual, assumptions about contamination between groups, the behavior of estimators at scale. I did not stop relying on theory when I left academia. I swapped which theory I lean on.
There are smaller echoes everywhere. Pruning my bad privacy item and dropping a leaky feature are the same move under two names, item purification and feature selection. And the plain old tools travel: a colleague’s dissertation in cancer biology came down to t-tests with Bonferroni corrections, which any analyst running many comparisons at work would recognize on sight.
What I lost crossing over
Here is the cost, and it is a real one. My survey models could answer why. My behavioral models mostly answer who and what and go quiet on why.
In the SEM, the explanation was the output. If privacy concern had landed as a significant negative path, that would have told me something a brand could act on: people are holding back because of how exposed they feel. The constructs are the mechanism. Behavioral models rarely hand you that. They tell you a user is unlikely to convert; they do not tell you it is because the checkout felt unsafe.
When the “why” matters in my current work, I usually have to go dig it out by hand. The clearest case I have seen was a geo lift test that came back negative, as if the ads had pushed people to buy less of a snack-category product. That is not a believable causal story, so it is better read as a warning light: some assumption behind the estimate has failed, and the job is to find which one, even though the standard pre-launch checks had all looked clean.
We pulled the category trend across regions over one- and two-year windows. The product’s sales ramp toward a seasonal peak, but the timing of that ramp is not the same everywhere. Our test region peaked later than the regions used for comparison. So, during the campaign window, the test region was still climbing while the comparison regions were already cresting, and a straight comparison made the ads look like they were suppressing sales. After the campaign, the test region kept reading below the comparison group, which is the fingerprint of a category-timing gap, not an ad effect. The assumption that quietly broke was the one underneath every lift estimate: that the groups would have moved together if the ad had never run. It held before the campaign and failed during it.
The fix going forward was not a fancier model; it was a habit. Pre-period alignment tells you the groups matched in the past, not that they will match through the window. When seasonality itself varies by region, you check the timing against prior seasons before you trust the readout. No feature importance plot was ever going to surface that. A person had to go reconstruct the story, which is precisely the kind of “why” my survey work used to deliver by default.
A short field guide between the two worlds
If you are moving from one of these worlds to the other, here is the cheat sheet I wish someone had handed me.
If your outcome is an attitude, an intention, or a perception, you are in measurement-model territory.
The tell is that you cannot point to a column and say “that is the thing.” Intention, trust, perceived usefulness, satisfaction, and brand affinity all live in this category. In that case, plan to build each predictor before you use it: write several indicators that circle the same idea, collect them together, and check that they actually cohere before you interpret a single path coefficient. The checks are quick to state. Do the items load on the construct they were written for? Does the construct hold together internally (Cronbach’s alpha, composite reliability)? Does it explain more of its own indicators’ variance than it shares with other constructs (average variance extracted, discriminant validity)?
The failure mode here is quiet. If you average three items into a single score and drop it into a regression, nothing errors out. You get a coefficient, a p-value, a story. But the measurement error you just averaged over does not vanish; it biases the coefficient, usually toward zero, and you may report a null for a relationship that exists. My own privacy construct is the cautionary tale: one of its three items was pulling in a different direction, and had I averaged blindly I would never have known the construct I “measured” was two things wearing one name.
If your outcome is a logged behavior, the variables come for free, but the validity question does not go away.
Nobody in a production ML pipeline asks whether category_views_30d really captures interest. The column exists, the model trains, the AUC is fine. That silence is the risk. Every behavioral feature is still a proxy for something unobservable, and the proxy relationship can decay without any monitoring alert firing.
The practical habit I carried over is embarrassingly low-tech: for each important feature, write one sentence stating what unobservable thing it stands in for. prior_category_purchase is a stand-in for purchase intent. session_depth is a stand-in for engagement. Keep that list somewhere durable. When a model starts behaving strangely, or when a metric gets gamed, or when a team starts optimizing a number that has drifted from the outcome anyone cared about, that list is what lets you name the problem. Almost every “our north-star metric stopped meaning anything” story is a construct-validity failure that nobody was watching for, because in this world nobody is assigned to watch.
Pick your enemy based on what the model is for.
Correlated inputs are not universally good or bad. Within a construct in a survey model, correlated indicators are the point; that is the evidence they measure the same latent thing. Among features in a predictive model, that same redundancy destabilizes linear coefficients and smears importance across tree splits.
The rule I use: if a human is going to read the model and act on individual coefficients or importances, hunt multicollinearity down (variance inflation factors, correlation clustering, dropping or combining redundant features). If the model only needs to rank or score, correlated features are usually harmless and pruning them buys you little. The mistake in both directions is treating a diagnostic as universally mandatory. Ask what decision the model output feeds, then decide whether interpretation matters at all.
A nonsensical estimate is a diagnostic, not a result.
When a lift test says an ordinary, inoffensive ad made people buy less, the finding is not “the ad backfired.” The finding is “an assumption behind my estimate has failed, and I now know where to look.” This is the single most useful reflex I brought from academic work, where an implausible coefficient sends you back to the specification rather than into the discussion section.
Where to look, in rough order: Is the counterfactual credible, meaning would the groups really have moved together without the treatment? Is there contamination between test and control? Did something in the category or the calendar move underneath both groups? Is the outcome window catching the groups at different points in a cycle? In my case, pre-period fit passed every check and the assumption still broke, because the regions hit their seasonal peak at different times. Which leads to the last one.
Pre-period alignment validates the past, not the future.
Matching test and control on the weeks before launch is necessary and nowhere near sufficient. It tells you the groups moved together historically; it says nothing about whether they will keep moving together through your measurement window. Any force that hits the groups on different schedules, most commonly seasonality, promotions, regional buying rhythms, or supply shocks, can pull them apart precisely when you are measuring.
The cheap safeguard: before trusting a readout, look at the same calendar window in prior years, per group. If the groups diverge in that window historically, they will probably diverge in yours, and your lift estimate is measuring the divergence, not the ad. This costs an afternoon and has saved me from shipping a confidently wrong number.
And the rule that holds in both worlds.
Commit to the question before you see the answer, and write down what could be wrong. A dissertation committee enforces this by making you defend a proposal. A business enforces it by spending money on your recommendation. The enforcement mechanism is different, the discipline is identical, and the moment you loosen it, in either world, the model will happily tell you what you were hoping to hear.
The thing that actually transferred
People sometimes ask whether the social-science training was wasted now that I work in ML. It wasn’t, but not for the reason I’d have guessed. The lasting skill isn’t any particular method; it’s a way of thinking about a single relationship between two variables. How do you stand in for something you can’t observe, and how do you notice when your stand-in is lying to you? What do you commit to before the data arrives? Which tool actually fits the question in front of you?
The null result trained that better than any success would have. I was sure privacy concern mattered; the model said it didn’t; I had to believe the model over myself. Years later, looking at a lift estimate that insisted an ad made people buy less, the same reflex kicked in. Don’t argue with it, and don’t massage the model until it tells you what you wanted. Find out what the data is actually doing, and which assumption gave way.
That reflex is the part that moved with me. The methods on either side are just the local dialect.
Next, I want to dig into the second big difference between these two worlds, which is messier than the variables: what each one is willing to call causal, and why a path arrow in a survey model and a lift estimate from an experiment are not at all the same claim.
References
- Breiman, L. (2001). Statistical Modeling: The Two Cultures. Statistical Science, 16(3), 199–231.
- Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319–340.
- Ouellette, J. A., & Wood, W. (1998). Habit and Intention in Everyday Life: The Multiple Processes by Which Past Behavior Predicts Future Behavior. Psychological Bulletin, 124(1), 54–74.
- Shmueli, G. (2010). To Explain or to Predict? Statistical Science, 25(3), 289–310.
- Rani, S. (2024). Adaptive Clothing Brands’ Content Marketing Strategies and Their Influence on Seniors’ Behavioral Intentions [Doctoral dissertation, North Carolina State University]. NC State University Libraries. https://www.lib.ncsu.edu/resolver/1840.20/42020
- Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M., A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM); add the exact edition from your bibliography.