「机器学习系列4」数据预处理-缺失值处理

数据预处理-缺失值处理

import pandas as pd
import numpy  as np
df = pd.DataFrame([\
                   ['frank', 'M',    np.nan], \
                   ['mary' , np.nan, np.nan], \
                   ['tom'  , 'M',    35], \
                   ['ted'  , 'M',    33], \
                   ['jean' , np.nan, 21], \
                   ['lisa' , 'F',    20]])
df.columns = ['name', 'gender', 'age']

1.统计数据集缺失情况

df['gender'].isnull()
df.isnull().sum()

2.缺失值处理方法1：删除remove missing values:dropna

?df.dropna

Signature: df.dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)
Docstring: Remove missing values.

Parameters #参数
----------
axis : {0 or 'index', 1 or 'columns'}, default 0  #按行或者按列删除元素
Determine if rows or columns which contain missing values are removed.

how : {'any', 'all'}, default 'any'
Determine if row or column is removed from DataFrame, when we have at least one NA or all NA.
    * 'any' : If any NA values are present, drop that row or column.#行/列有一个缺失值就删除
    * 'all' : If all values are NA, drop that row or column.#行/列都是缺失值才删除

thresh : int, optional # 整型，可选
    Require that many non-NA values.#删除 【缺失值个数>=N】 的某行/某列

subset : array-like, optional #列表，可选
	 Define in which columns to look for missing values.#从哪列寻找缺失值

inplace : bool, default False
    If True, do operation inplace and return None.

3.缺失值处理方法2：填充fill missing values:fillna

?df.fillna

Signature:df.fillna(value=None,method=None,axis=None,inplace=False,limit=None,downcast=None) 
Docstring:Fill NA/NaN values using the specified method.

method : {'backfill', 'bfill', 'pad', 'ffill', None}, default None
    Method to use for filling holes in reindexed Series
    pad / ffill: propagate last valid observation forward to next valid #用前一行的值填充
    backfill / bfill: use next valid observation to fill gap.#用后一行的值填充

df.fillna(0) #缺失值全部填充为0
df['age'].fillna(df['age'].mean()) # 平均值填充
df['age'].fillna(df.groupby('gender')['age'].transform('mean'), inplace=True) #分组平均值填充
df.fillna(method="pad")

4.数据预处理流程样例

#4.1 读取数据
import pandas
df = pandas.read_csv('data/house_data.csv')
#4.2 查看数据前几行/后几行
df.head(6)
df.tail(6)
#4.3 删除第一列重复索引
del df['Unnamed: 0']
#4.4 数据统一：'暂无资料' 也是缺失值
import numpy as np
df.loc[df['物 业 费'] == '暂无资料','物 业 费'] = np.NAN
# 4.5 查看数据信息、类型、描述性统计
df.info()
df.columns
df.dtypes
df.describe()
# 4.6 统计缺失值情况
df.isnull().sum()
df.isnull().sum() / df.count()
# 4.7 处理缺失值
df = df.drop('参考月供', axis = 1)
df = df.dropna(axis=1, how='all')
df['物 业 费'] =df['物 业 费'].fillna(0)
df['總價'] = df['總價'].fillna(df['建筑面积'] * (df['總價'] / df['建筑面积']).mean())
# 4.8 保存处理结果
df.to_csv('house_final.csv', index_label=False)