Cash Score: Alternative Credit Scoring

To prepare the transaction data for further analysis and modeling, we enhanced the memo field by adding specific tokens based on transaction characteristics. This token augmentation adds additional context about transaction amounts and dates, helping the model learn more meaningful patterns.

The following steps outline the transformation process:

• Whole Dollar Amount Identification: We added a token <W_D> to the memo field for transactions with whole dollar amounts. This token helps identify transactions that might involve cash withdrawals or ATM transactions, as these are typically in round amounts (e.g., $20, $50).
• Day of Transaction: For each transaction's posting date, we generated a day token in the format <D_day>, where day represents the specific day of the month. This token helps analyze patterns such as end-of-month or beginning-of-month spending habits.
• Month of Transaction: Similarly, we generated a month token in the format <M_month> (where month is the numerical month) to capture seasonal or monthly spending trends.
• Token Augmentation Process: For each row in the dataset, these tokens were concatenated to the memo field using the following transformation:

  memo = memo + whole_dollar_amount_token + day_token + month_token

To prepare the memo field for analysis, we first applied a series of cleaning steps to standardize the text. We transformed all text to lowercase for uniformity, removed any extraneous punctuation and symbols, and stripped out dates, state abbreviations, and recurring text that didn’t contribute to transaction categorization (e.g., “POS withdrawal”). Placeholder values such as multiple X’s were also removed.

To prepare the dataset for modeling, we began by reviewing a sample of unique transaction memos from each category. This step provided insights into common patterns and inconsistencies, helping us determine the scope and focus of our cleaning tasks.

We then applied several transformations to standardize the data and improve its interpretability:

• Date Removal: We used regular expressions (RegEx) to locate and remove dates across entries in the memo column, as they did not contribute to the model's predictive goals and added unnecessary complexity. Placeholder patterns such as "XXXX" were also removed.
• Pattern Recognition: Specific patterns and keywords were identified as indicators of certain transaction categories. For example, "TST" was reliably associated with "Food and Beverages," while "APPLE.COM/BILL" was linked to "General Merchandise." These patterns were flagged to automate future classifications and reduce manual intervention.
• Selective Character Retention: To preserve potentially valuable information, certain characters such as dots ('.') were retained to keep URLs or email addresses intact within the memo field, which could provide clues to transaction categories.
• Transaction Labels: We identified recurring phrases such as "POS Withdrawal," location-specific markers (e.g., “CA 10/27” for state and date), and labels indicating recurring payments. These were removed as they did not contribute to the model’s predictive goals and added unnecessary complexity.
• Text Normalization: All text was converted to lowercase to reduce variability due to case differences.

We transformed text data into numerical features using a TF-IDF vectorizer, allowing the model to analyze term frequency patterns.

• Max Features = 5000: Limited to the top 5,000 most important terms.
• Max DF = 0.95: Ignored terms appearing in more than 95% of documents.
• Min DF = 5: Ignored terms appearing in fewer than 5 documents.

To classify transaction categories based on the memo text field, we first used a Logistic Regression model with TF-IDF vectorization. This method converts raw text data into numerical features, allowing the model to utilize term frequency patterns for classification.

For classification, we configured the Logistic Regression model with:

• solver='saga': Efficient for large datasets and supports L2 regularization.
• max_iter=200: Ensured convergence.
• n_jobs=-1: Utilized all available CPU cores for parallel training.

Accuracy: The model achieved 96.15% accuracy on the test set.

Confusion Matrix

To improve classification performance, we implemented a Random Forest model. While this model introduces greater complexity, its performance was slightly lower than Logistic Regression.

For classification, we configured the Random Forest model with:

• n_estimators=100: Used 100 decision trees.
• max_depth=60: Restricted tree depth to prevent overfitting.
• n_jobs=-1: Utilized all available CPU cores for parallel training.

Accuracy: The Random Forest model achieved 84.28% accuracy on the test set.

Confusion Matrix

FastText is a lightweight and efficient text classification model developed by Facebook AI. We trained this model to categorize transaction memos with high speed and accuracy.

Model Training

The FastText model was trained using the following hyperparameters:

• Epochs: 25
• Learning Rate: 1.0
• Word N-Grams: 2
• Embedding Dimension: 50
• Bucket Size: 200,000

Accuracy: The FastText model achieved an impressive 98.9% accuracy.

Classification Report

The classification report below presents precision, recall, and F1-scores for different transaction categories.

We implemented a Transformer-based model using distilbert-base-uncased to leverage deep learning for text classification. This model was trained to classify transaction memos with contextual embeddings.

Training Configuration

The model was configured with the following hyperparameters:

• Model Type: DistilBERT
• Maximum Length: 128 tokens
• Batch Size: 16
• Learning Rate: 2e-5
• Optimizer: AdamW
• Scheduler: Linear schedule with a warmup ratio of 0.1
• Gradient Clipping: Maximum value of 1.0
• Epochs: 3

Model Performance

The Transformer model was evaluated on 9 transaction categories. After one epoch, it achieved:

• Training Loss: 0.1361
• Validation Loss: 0.0603
• Validation Accuracy: 98.56%
• Evaluation Time: 2 minutes and 43 seconds

While the Transformer model achieved high accuracy, it exhibited higher latency compared to simpler models, requiring longer processing times for both training and evaluation.

We conducted an exploratory data analysis (EDA) to understand patterns in consumer spending, overdrafts, and transaction distributions. This helped us identify meaningful features for predicting credit risk.

• Transaction Patterns: Identified differences in transaction behaviors between delinquent and non-delinquent consumers, such as frequency and types of purchases.
• Seasonal Trends & Payday Effects: Examined how consumer spending fluctuates with seasonal trends, including holiday expenses and end-of-month payday spikes.
• Income Estimation: Estimated consumer income based on recurring transactions such as payroll deposits, rent, and utility payments.
• Impact of Fees & Overdrafts: Analyzed the effects of account fees, buy-now-pay-later (BNPL) transactions, and overdraft occurrences on overall financial stability.

We analyzed balance trends over time to observe how cash flow differs between individuals who are delinquent and those who are not. Our key findings highlight financial stability differences based on account balance trends.

• Delinquent Consumers: Frequently experience negative balances, indicating financial instability and a higher risk of missing payments.
• Non-Delinquent Consumers: Maintain relatively stable balances with fewer occurrences of overdrafts, suggesting better financial health.
• Periodic Fluctuations: Balance trends exhibit cyclical patterns influenced by income deposits and spending habits.

We engineered multiple features relevant to the prediction of delinquency, focusing on balance trends, transaction behaviors, and account types. These features help in assessing financial stability and predicting default risk.

• Balance Features: Negative balance ratio, balance trends, payday effects.
• Transaction-Based Features: Credit vs. debit transaction volume, category-based spending breakdown.
• Temporal Features: Spending frequency over time, accounting for longevity effects.
• Account Types: Features based on the types of accounts a consumer has.
• Overdraft Frequency: The number of times a consumer overdrafted in the past 6 months.
• Spending Volatility: Standard deviation of monthly expenditures.
• Recurring Payments: Identifying transactions like rent, utilities, and subscriptions.

Top predictive features for different models:

The following are the top 10 most important features influencing delinquency prediction, as identified by SHAP values:

• account_type_SAVINGS: Indicates whether a user has a savings account. A savings account is often associated with financial stability, reducing the risk of delinquency.
• balance: Represents the consumer's total account balance. Lower balances correlate with higher delinquency risk, while higher balances suggest financial security.
• ACCOUNT_FEES_median: Measures the median account fees incurred. Frequent fees may indicate financial strain or poor account management.
• LOAN_std (Loan Standard Deviation): Represents the variability in loan-related transactions. High fluctuations may indicate inconsistent repayments and financial instability.
• OVERDRAFT_count: Tracks the number of overdraft occurrences. A higher count strongly correlates with financial distress and increased risk of delinquency.
• REFUND_skewness: Measures the asymmetry in refund transactions. Unusual refund patterns may suggest erratic cash flow or financial instability.
• ACCOUNT_FEES_count: The total number of account fees charged. Higher counts may indicate insufficient funds or excessive financial penalties.
• INVESTMENT_INCOME_std (Investment Income Standard Deviation): Measures fluctuations in investment income. High variation could indicate unreliable financial inflows.
• SELF_TRANSFER_iqr (Interquartile Range of Self-Transfers): Measures variability in self-transfers between accounts. A high IQR may indicate inconsistent cash flow management.
• ATM_CASH_median: The median amount of cash withdrawn from ATMs. Frequent and high cash withdrawals may suggest reliance on liquid cash, which could relate to financial instability.

These features provide critical insights into financial behavior and help the model better assess delinquency risk.

We trained multiple machine learning models to predict delinquency risk, focusing on maximizing performance while maintaining interpretability and efficiency.

• Feature Selection: Used only the top 75 most important features to improve model interpretability and predictive power.
• Scoring Exclusions: Consumers with fewer than two transactions were excluded to ensure meaningful behavioral patterns.

To address the class imbalance issue (delinquent consumers make up only 8.4% of the dataset), we applied:

• SMOTE & SMOTEENN: Synthetic oversampling techniques to balance the dataset.
• Feature Normalization: Standardization of key numerical variables to improve model performance.

To optimize the ROC-AUC score, we trained five primary models:

• XGBoost: High-performance gradient boosting model with extensive hyperparameter tuning.
• LightGBM: Fast and efficient for large datasets, especially with categorical features.
• CatBoost: Optimized for categorical data, reducing the need for preprocessing.
• Balanced Random Forest: Addresses class imbalance by appropriately weighting samples.
• Weighted Ensemble: Combines top models for an optimal balance of accuracy, interpretability, and efficiency.

We evaluated model performance using several key classification metrics:

• ROC-AUC: Measures the model’s ability to differentiate between delinquent and non-delinquent users.
• Precision and Recall: Precision measures the percentage of correctly predicted delinquents, while recall captures the proportion of actual delinquents that were correctly identified.
• F1-Score: Provides a balance between precision and recall, ensuring both false positives and false negatives are considered.
• Training Time: Evaluates how long each model takes to train, an essential factor for scalability.
• Prediction Time: Measures how efficiently models generate predictions for new consumers.

Model	ROC-AUC	Accuracy	Precision	Recall	F1-Score	Train Time (s)	Predict Time (s)
LightGBM	0.8221	0.9031	0.8739	0.9031	0.8814	2.5050	0.000019
Weighted Ensemble	0.8301	0.9006	0.8728	0.9006	0.8813	0.0010	0.000001
XGBoost	0.8232	0.8962	0.8677	0.8962	0.8778	2.0606	0.000006
CatBoost	0.8212	0.8892	0.8707	0.8892	0.8785	3.0342	0.000004
Balanced RF	0.8144	0.8982	0.8703	0.8982	0.8797	26.2355	0.000064
HistGB	0.823	0.913	0.892	0.913	0.901	3.196	0.000021
RUSBoost	0.802	0.829	0.903	0.829	0.859	6.954	0.000012
Random Forest	0.800	0.915	0.893	0.915	0.902	14.364	0.000088
Logistic Regression	0.777	0.743	0.911	0.743	0.803	0.348	0.000003

Our analysis highlights the effectiveness of different machine learning models in predicting delinquency risk. Among them, weighted ensemble demonstrated the highest ROC-AUC score, making it the most effective model for our use case.

• Best Model: Weighted ensemble achieved the best overall performance.
• Cash Scores outperform Credit Scores: Our analysis shows that transaction-based Cash Scores provide better delinquency predictions than traditional Credit Scores.