Case Study
Music & Machine Learning Pipeline
End-to-end ML pipeline to study how far audio and lyrical content alone can predict music popularity.
Key Metrics
At a Glance
Dataset
1,550 tracks
Multi-source, deduplicated
Embeddings
384 -> 213 dims
PCA (95% variance)
Best Model
Gradient Boosting
Isolation Forest outliers
Test Performance
RMSE ~ 19.7
R² ~ 0.20
Summary
TL;DR
Background
Context
The project investigates whether intrinsic musical content -- audio features and lyrical sentiment -- can meaningfully predict popularity, independent of external effects like marketing or platform exposure. Emphasis on building a pipeline that is reproducible, auditable, and bias-aware.
Architecture
Pipeline Overview
End-to-End Pipeline
Data Sources
Billboard
Apple Music
Spotify
Validation Pipeline
Normalize
Deduplicate
Join
Sentiment
1,550 Tracks
Clean dataset
Feature Eng.
Embeddings
Emotion scores
Models
Lasso
Ridge
GB
Predictions
Popularity score
Data
Dataset Construction & Validation
Combined multiple chart sources (Billboard, Apple Music, Spotify charts, curated playlists). The pipeline applied four validation passes: normalization of artist/title strings, cross-chart deduplication, Spotify feature joins, and multilingual lyric sentiment integration. Result: a clean master dataset of 1,550 tracks.
Features
Feature Engineering
Emotional Features
Lyrics Processing
Embedding & PCA
Visualization
PCA Dimensionality Reduction
Dimensionality Reduction
384-D
Embeddings
213-D
PCA reduced
Analysis
Exploratory Analysis & Insights
Popularity increases slightly with arousal, while happiness alone is a poor predictor -- aligning with prior research linking chart success to high-arousal emotional content rather than positive sentiment alone.
Key finding
Models
Modeling & Evaluation
Evaluated multiple regression models under different outlier-handling strategies. Performance measured using cross-validation and held-out test sets, revealing a consistent generalization gap and highlighting the importance of outlier treatment.
| Model | Outlier Strategy | Test RMSE | Test R² |
|---|---|---|---|
| Lasso | None | ~20.5 | ~0.14 |
| Ridge | None | ~20.8 | ~0.11 |
| Gradient Boosting | Isolation Forest | ~19.7 | ~0.20 |
Limitations
Bias & Limitations
Known Biases
- Western and English-language overrepresentation in chart data and lyric availability
- Platform bias from Spotify and Billboard reflecting algorithmic curation rather than pure listener preference
- Demographic skew toward younger, high-income users in streaming platforms
These factors explain a significant portion of the remaining prediction error and motivate the inclusion of exposure-based signals in future work.
Contribution
Role
Co-authored the project. Designed and implemented the ingestion and validation pipeline, lyric sentiment extraction, embedding and PCA workflow, model evaluation, and bias analysis.
Future Work
What I'd Do Next
Incorporate exposure-based signals (artist history, playlist placement, release timing) to test how much of the unexplained variance is attributable to external rather than intrinsic factors.