Customer Segmentation

This project segments 2,240 customers from a Portuguese retail loyalty program into distinct behavioral personas using K-Means clustering, then builds three separate recommendation systems on top of those segments: an association-rule engine for inventory planning, a hybrid recommender for personalized product suggestions, and a rule-based deal-matching system for targeted promotions. The entire pipeline, from raw data to clustering to recommendations, is implemented in PySpark, using its MLlib library for feature engineering, clustering, and frequent-pattern mining. The project began as a final project for IST 418: Big Data Analytics at Syracuse University and was later refactored from a single notebook into a modular, three-notebook pipeline for improved readability and reuse.

Retailers running broad, undifferentiated marketing campaigns waste spending on offers that don't match customer preferences and miss opportunities to deepen engagement with high-value customers. Without a systematic way to group customers by demographic and behavioral traits, businesses can't answer basic questions that drive strategy: which customers are price-sensitive versus loyalty-driven, what products should be bundled or promoted together, and which deals will actually resonate with a given customer. The goal was to take raw, unlabeled customer data and produce actionable, interpretable customer segments, along with concrete recommendations for what to sell, to whom, and how, using unsupervised learning at scale.

Loaded the raw dataset into a Spark DataFrame, dropped 24 rows with missing income values out of 2,240, and engineered features including total number of children, customer tenure, total campaigns accepted, computed age and age group, total spend across product categories, and cleaned marital status and education categories, writing the cleaned dataset back out as a distributed CSV. Validated the engineered features with Spark SQL queries, then converted to pandas to visualize education distribution, spending by education and marital status, spending by age group, and spending by loyalty duration. Built a Spark ML Pipeline combining a VectorAssembler (numeric features), StandardScaler (standardization), StringIndexer + OneHotEncoder (education and marital status), and a final VectorAssembler to merge scaled and encoded features into a single feature vector. Trained K-Means models for K = 2–10 and used three methods together — the elbow method (WSSSE), silhouette scores, and 2D PCA visualizations — to pick the final value: the elbow graph pointed to K = 6 and the silhouette score was marginally higher at K = 5, but the PCA projection at K = 5 produced messy, overlapping clusters, so K = 3 was chosen instead, since its PCA visualization showed the cleanest separation and its silhouette score (0.3136) was only about 0.01 below the K = 5 maximum. Aggregated average income, age, spend by category, purchase channel, deal usage, and education/marital status distribution per cluster via Spark SQL to build an interpretable persona for each segment, then built three complementary recommendation engines on top of the cluster assignments.

K-Means with K = 3 achieved a silhouette score of 0.3136 and produced three well-separated, interpretable customer segments: Balanced Mid-Spenders (avg. income $57,993, avg. spend $730, deal-interested and engaged online), Budget-Conscious Deal Seekers (avg. income $35,402, avg. spend $99, highest deal usage and most price-sensitive), and Affluent Digital Loyalists (avg. income $77,466, avg. spend $1,422, highest campaign acceptance and lowest deal reliance). Association rule mining with FP-Growth returned confidence scores between 0.80 and 1.0 across all clusters, with wine and meat consistently emerging as the core products driving co-purchase behavior, a direct signal for inventory prioritization. All three recommendation engines produced cluster-specific, actionable outputs: Balanced Mid-Spenders map to wines, meat, and gold with online bundle/BOGO deals; Budget-Conscious Deal Seekers map to sweets and meat with in-store coupons and family packs; Affluent Digital Loyalists map to wines, fish, and gold with loyalty rewards and email promos. The project demonstrates a full, reproducible pipeline for turning raw transactional and demographic data into segmentation-driven marketing strategy, moving beyond a single clustering output to three distinct, business-ready recommendation mechanisms. Planned next steps include a dashboard that lets a business enter new customer data, automatically assign it to a cluster, and surface the corresponding marketing and inventory recommendations in real time.

outcome: Chose K = 3 over the elbow method's suggested K = 6 by weighing three validation methods together — the PCA projection was the tiebreaker that a metric alone would have missed.

• 

  Elbow method (WSSSE) across K = 2–10 — the naive elbow points to K = 6, but K = 3 was chosen instead.

• 

  Customer segments visualized by PCA — K = 3 produced the cleanest separation of the values tested.

• PySpark
• MLlib
• K-Means
• FP-Growth
• PCA