Credit Risk Assessment

On a sunny day in the middle of the Brazilian semi-arid region, a representative of a financial institution, more specifically Fintech λλ, contacted me by email, requesting a service to build a solution that would improve their customer credit risk assessment model.

Among the areas that make up these companies, one of those that causes the most problems and discomfort is the default rate across customer portfolios. This makes portfolio assessment a mandatory and extremely important activity, since defaults can cause enormous deficits on the balance sheet.

So I went digging and found out that, for these institutions, each customer is part of a credit portfolio, which indicates how likely customers are to be financially able to repay their debt (loans, credit cards and others); this portfolio assessment is reported as a ranking.

Because of this, these institutions are investing more and more in new technologies to develop and improve their assessment systems, always aiming to minimize the risk of default or of non-compliance with the obligations and/or conditions of a loan, and betting increasingly on Machine Learning models.

Countless solutions could be built to address these problems. In this project, I will create an algorithm to identify the probability that a customer will fail to meet their financial obligations. Note that this whole story is a fiction I created, and this work is one of the hands-on projects of Carlos Melo's Data Science na Prática course.

To start the work, I needed to evaluate the database provided by the institution. It is worth noting that these assessments are usually done when the customer applies for the card (typically at the first contact with the institution); however, we will use an existing database (Dataset). This dataset has 45,000 entries and 43 columns and is basically a CSV file.

In this project, it is extremely important to understand the meaning of each variable used in the work, which can be seen below:

• id - anonymous identifier, a unique value per customer.
• target_default - the target variable, which we will use to analyze the risk of default.
• The columns score_3, score_4 and score_5 are numeric, while score_1 and score_2 are encoded in some way. We will have to check later whether they have a number of classes that can be converted into useful information.
• There are other variables with some kind of encoding, such as reason, state, zip, channel, job_name and real_state, which will also need deeper analysis to determine whether any information can be extracted from them.
• profile_tags - contains a dictionary with a label assigned to each customer.
• target_fraud - would be the target variable for another model, where the goal would be fraud detection.
• lat_lon - stored as a string containing a tuple with the coordinates.

As the first phase of any project, I will start with exploratory data analysis, in this order: missing values, data types, unique values, data balance and descriptive statistics.

Running this calculation, we can see that some columns, such as target_fraud, last_amount_borrowed, last_borrowed_in_months, ok_since and external_data_provider_credit_checks_last_2_year, have more than half of their data missing. The columns external_data_provider_credit_checks_last_year, credit_limit and n_issues have between 25% and 34% of their values missing. In addition, the unique value count shows that the columns external_data_provider_credit_checks_last_2_year and channel have a single possible value, so they cannot contribute to the model and can be dropped.

Next, when I analyzed the balance of our target variable target_default, the proportion of defaulters was False 77.95% and True 14.80%, so balancing will be needed later on.

Finally, from the main descriptive statistics shown, we can highlight a few observations:

• The column external_data_provider_credit_checks_last_2_year has minimum, maximum and standard deviation values all equal to zero.
• The column reported_income has inf values, which will interfere with the analysis and the model. We will replace np.inf values with np.nan to work with the data.
• The column external_data_provider_email_seen_before has a minimum value of -999, which is odd given the other information. After a closer look, I concluded that these values are outliers or were improperly handled. We will replace values equal to -999 with np.nan.

With most of the exploratory analysis done, the next step is to carry out the manipulations needed to build the best model, so we can now start preprocessing the data.

To do so, I performed the following cleaning steps:

• Replace inf with NaN in the reported_income column;
• Drop the columns ids, target_fraud, external_data_provider_credit_checks_last_2_year, channel and profile_phone_number;
• Replace -999 in external_data_provider_email_seen_before with NaN;
• Drop columns with no apparent information or that require further research;
• Drop the entries where target_default is NaN.

With this first round of preprocessing done by analyzing the variables, I could then start preprocessing the entries. This dataset has a lot of missing and null data, but we do not have much information about the reasons behind these problems. To address this, I will replace the missing values (NaN) with zero for variables that have no reference, such as 'last_amount_borrowed', 'last_borrowed_in_months' and 'n_issues'. For the rest, I will use the median for numeric variables and the mode for categorical ones.

With all of this preprocessing done, the dataset looked like this:

This left us with 41,741 entries and 25 columns. Note that most of the columns are encrypted, so I will use the LabelEncoder and GetDummies algorithms to transform the categorical data into numerical data, splitting out those with more than two categories.

The result was a dataset like this:

As we saw earlier, the target data is heavily imbalanced, so some manipulation was needed, resulting in a boolean split of False 35,080 and True 35,025.

With all the manipulations done, I searched the literature and looked at other data scientists' work to find which model best fits this type of problem. Through a systematic search, I found that the best model for this problem, or rather the most commonly used, is RandomForest (RF). Then, to identify the best RF model, I created a validation function that uses the Recall metric, establishing a general baseline of how much the model could err and thereby confirming that it would be the best learning model to use.

#Função de validação

def valid(X, y, model, quite = False):

  X = np.array(X)
  y = np.array(y)
  pipeline = make_pipeline(StandardScaler(), model)
  scores = cross_val_score(pipeline, X, y, scoring="recall")

  if quite == False:
     print("Recall: {:.3f} (+/- {:.3f})".format(scores.mean(), scores.std()))

  return scores.mean()

rfc = RandomForestClassifier()
score_baseline = valid(X_treino_balanceado, y_treino_balanceado, rfc)

This yielded Recall: 0.967 (+/- 0.003), which allowed me to proceed with the RF algorithm and start testing with GridSearchCV to identify the best parameters, arriving at the following final model.

forest = RandomForestClassifier(criterion='entropy', n_estimators=100)
forest.fit(X_treino_balanceado, y_treino_balanceado)

RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='entropy', max_depth=None, max_features='auto',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=100,
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=0, warm_start=False)

With the model ready, I could then make predictions and compare them with the results of the test set, obtaining an excellent result, as you can see below:

              precision    recall  f1-score   support

           0       0.99      0.96      0.97      8772
           1       0.96      0.99      0.97      8755

    accuracy                           0.97     17527
   macro avg       0.97      0.97      0.97     17527
weighted avg       0.97      0.97      0.97     17527

Área sobre a curva ASC: 0.971373390177927

The confusion matrix shows 96% True Positives and 99% True Negatives.

In this project, I carried out several procedures to build a credit risk prediction model, which required a number of manipulations, from dropping columns to balancing and standardization. The result was extremely satisfactory for the client and could be put into production. However, there is still a gap: trying other learning models for this task to determine whether RandomForest is in fact the best for this problem. You can see the full project in the notebook.

For Fintech λλ, the result was indeed very satisfactory, since it could improve their assessment service. We can see this with a simple example: for a client with a credit portfolio of 1 million reais, my model could ensure that the company would receive 990 thousand from the loans, erring in only 1% of cases. This makes the company's financial risk acceptable, as it would keep its credit portfolio in a very healthy state, with a default rate well below the national averages.

Did you like this kind of structure for the blog articles? Send me your feedback on my social media or by email, and if you enjoyed it, share it to support the work. :)