Handling Missing Data - Example - Part 3 (Non-numeric Values)

Loan Amount (Revisit)

A cursory look at the Loan Amount column shows that most values are numeric, but some values are non-numeric. For example in one place 2000 is written as Two Thousand.

To handle non-numeric values in the LoanAmount column, such as "Two Thousand", you would typically need to:

Identify all the non-numeric entries.
Convert them to a numeric format.
Replace the original non-numeric entries with their numeric equivalents.

Here's a step-by-step guide on how to write the code to do this:

Identify Non-numeric Entries: Use regular expressions or pd.to_numeric with errors='coerce' to flag non-numeric entries.
Map Non-numeric to Numeric: Create a mapping of words to numbers. For English number words, you can use the word2number package, which can convert number words like "Two Thousand" into numeric values.
Replace Non-numeric Entries: Use apply to replace the non-numeric entries with their numeric equivalents.

Here's a Python code snippet that shows this. First, you would need to install the word2number package if it's not already installed:

1conda install -c conda-forge word2number
2or 
3pip install word2number
4
5

1from word2number import w2n
2
3# Function to convert non-numeric loan amounts to numeric
4def convert_to_numeric(value):
5    try:
6        # This will convert numeric strings to integers and ignore already numeric values
7        return pd.to_numeric(value)
8    except ValueError:
9        try:
10            # This will convert written numbers to numeric values
11            return w2n.word_to_num(value)
12        except ValueError:
13            # If conversion fails, return a default value or raise an error
14            return None
15
16# Apply the function to the 'LoanAmount' column
17loan_data_cleaned['LoanAmount'] = loan_data_cleaned['LoanAmount'].apply(convert_to_numeric)
18
19# Check for any None values which indicate failed conversions
20failed_conversions = loan_data_cleaned[loan_data_cleaned['LoanAmount'].isnull()]
21print("Failed conversions:\n", failed_conversions[['CustomerName', 'LoanAmount']])
22
23

Our Loan Amount data now has only numeric values.

Premium Content

This tutorial is a part of the course Data Manipulation Using Pandas - Part 1. This is a premium course. The purchase options for the course are provided below. With this course, you get access to complete course content, source code, practical exercises, and all resources that are a part of the course.

Lifetime Premium Membership

$250

$179

Get unlimited access to all courses and premium content

Join Premium

Individual Course

Purchase this course individually

Purchase Course View Course Details

What's Included:

Complete access to course content and updates

All downloadable resources

Interactive course quizzes

Practice exercises and sample code

Ad-free learning experience

Learn

Resources

Handling Missing Data - Example - Part 3 (Non-numeric Values)

Handling Missing Data - Example - Part 2

Handling Missing Data - Example - Part 4

Topics

Course Downloads

Course Resources

eBook - Data Manipulation Using Pandas

Data and Jupyter Notebooks