本次分享的项目来自 Kaggle 的经典赛题：房价预测。分为数据分析和数据挖掘两部分介绍。本篇为数据分析篇。

赛题解读

比赛概述

影响房价的因素有很多，在本题的数据集中有 79 个变量几乎描述了爱荷华州艾姆斯 (Ames, Iowa) 住宅的方方面面，要求预测最终的房价。

技术栈

• 特征工程 (Creative feature engineering)
• 回归模型 (Advanced regression techniques like random forest and
  gradient boosting)

最终目标

预测出每间房屋的价格，对于测试集中的每一个Id，给出变量SalePrice相应的值。

提交格式

Id,SalePrice
1461,169000.1
1462,187724.1233
1463,175221
etc.

数据分析

数据描述

首先我们导入数据并查看：

train_df = pd.read_csv('./input/train.csv', index_col=0)
test_df = pd.read_csv('./input/test.csv', index_col=0)

train_df.head()

我们可以看到有 80 列，也就是有 79 个特征。

接下来将训练集和测试集合并在一起，这么做是为了进行数据预处理的时候更加方便，让测试集和训练集的特征变换为相同的格式，等预处理进行完之后，再把他们分隔开。

我们知道SalePrice作为我们的训练目标，只出现在训练集中，不出现在测试集，因此我们需要把这一列拿出来再进行合并。在拿出这一列前，我们先来观察它，看看它长什么样子，也就是查看它的分布。

prices = DataFrame({'price': train_df['SalePrice'], 'log(price+1)': np.log1p(train_df['SalePrice'])})
prices.hist()

因为label本身并不平滑，为了我们分类器的学习更加准确，我们需要首先把label给平滑化（正态化）。我在这里使用的是log1p, 也就是 log(x+1)。要注意的是我们这一步把数据平滑化了，在最后算结果的时候，还要把预测到的平滑数据给变回去，那么log1p()的反函数就是expm1()，后面用到时再具体细说。

然后我们把这一列拿出来：

y_train = np.log1p(train_df.pop('SalePrice'))

y_train.head()

有

Id
1    12.247699
2    12.109016
3    12.317171
4    11.849405
5    12.429220
Name: SalePrice, dtype: float64

这时，y_train就是SalePrice那一列。

然后我们把两个数据集合并起来：

df = pd.concat((train_df, test_df), axis=0)

查看shape:

df.shape

(2919, 79)

df就是我们合并之后的DataFrame。

数据预处理

根据 kaggle 给出的说明，有以下特征及其说明：

SalePrice - the property's sale price in dollars. This is the target variable that you're trying to predict.
MSSubClass: The building class
MSZoning: The general zoning classification
LotFrontage: Linear feet of street connected to property
LotArea: Lot size in square feet
Street: Type of road access
Alley: Type of alley access
LotShape: General shape of property
LandContour: Flatness of the property
Utilities: Type of utilities available
LotConfig: Lot configuration
LandSlope: Slope of property
Neighborhood: Physical locations within Ames city limits
Condition1: Proximity to main road or railroad
Condition2: Proximity to main road or railroad (if a second is present)
BldgType: Type of dwelling
HouseStyle: Style of dwelling
OverallQual: Overall material and finish quality
OverallCond: Overall condition rating
YearBuilt: Original construction date
YearRemodAdd: Remodel date
RoofStyle: Type of roof
RoofMatl: Roof material
Exterior1st: Exterior covering on house
Exterior2nd: Exterior covering on house (if more than one material)
MasVnrType: Masonry veneer type
MasVnrArea: Masonry veneer area in square feet
ExterQual: Exterior material quality
ExterCond: Present condition of the material on the exterior
Foundation: Type of foundation
BsmtQual: Height of the basement
BsmtCond: General condition of the basement
BsmtExposure: Walkout or garden level basement walls
BsmtFinType1: Quality of basement finished area
BsmtFinSF1: Type 1 finished square feet
BsmtFinType2: Quality of second finished area (if present)
BsmtFinSF2: Type 2 finished square feet
BsmtUnfSF: Unfinished square feet of basement area
TotalBsmtSF: Total square feet of basement area
Heating: Type of heating
HeatingQC: Heating quality and condition
CentralAir: Central air conditioning
Electrical: Electrical system
1stFlrSF: First Floor square feet
2ndFlrSF: Second floor square feet
LowQualFinSF: Low quality finished square feet (all floors)
GrLivArea: Above grade (ground) living area square feet
BsmtFullBath: Basement full bathrooms
BsmtHalfBath: Basement half bathrooms
FullBath: Full bathrooms above grade
HalfBath: Half baths above grade
Bedroom: Number of bedrooms above basement level
Kitchen: Number of kitchens
KitchenQual: Kitchen quality
TotRmsAbvGrd: Total rooms above grade (does not include bathrooms)
Functional: Home functionality rating
Fireplaces: Number of fireplaces
FireplaceQu: Fireplace quality
GarageType: Garage location
GarageYrBlt: Year garage was built
GarageFinish: Interior finish of the garage
GarageCars: Size of garage in car capacity
GarageArea: Size of garage in square feet
GarageQual: Garage quality
GarageCond: Garage condition
PavedDrive: Paved driveway
WoodDeckSF: Wood deck area in square feet
OpenPorchSF: Open porch area in square feet
EnclosedPorch: Enclosed porch area in square feet
3SsnPorch: Three season porch area in square feet
ScreenPorch: Screen porch area in square feet
PoolArea: Pool area in square feet
PoolQC: Pool quality
Fence: Fence quality
MiscFeature: Miscellaneous feature not covered in other categories
MiscVal: $Value of miscellaneous feature
MoSold: Month Sold
YrSold: Year Sold
SaleType: Type of sale
SaleCondition: Condition of sale

接下来我们对特征进行分析。上述列出了一个目标变量SalePrice和 79 个特征，数量较多，这一步的特征分析是为了之后的特征工程做准备。

我们来查看哪些特征存在缺失值：

print(pd.isnull(df).sum())

这样并不方便观察，我们先查看缺失值最多的 10 个特征：

df.isnull().sum().sort_values(ascending=False).head(10)

为了更清楚的表示，我们用缺失率来考察缺失情况：

df_na = (df.isnull().sum() / len(df)) * 100
df_na = df_na.drop(df_na[df_na == 0].index).sort_values(ascending=False)
missing_data = pd.DataFrame({'缺失率': df_na})
missing_data.head(10)

对其进行可视化：

f, ax = plt.subplots(figsize=(15,12))
plt.xticks(rotation='90')
sns.barplot(x=df_na.index, y=df_na)
plt.xlabel('Features', fontsize=15)
plt.ylabel('Percent of missing values', fontsize=15)
plt.title('Percent missing data by feature', fontsize=15)

我们可以看到PoolQC、MiscFeature、Alley、Fence、FireplaceQu 等特征存在大量缺失，LotFrontage 有 16.7% 的缺失率，GarageType、GarageFinish、GarageQual 和 GarageCond等缺失率相近，这些特征有的是 category 数据，有的是 numerical 数据，对它们的缺失值如何处理，将在关于特征工程的部分给出。

最后，我们对每个特征进行相关性分析，查看热力图：

corrmat = train_df.corr()
plt.subplots(figsize=(15,12))
sns.heatmap(corrmat, vmax=0.9, square=True)

我们看到有些特征相关性大，容易造成过拟合现象，因此需要进行剔除。在下一篇的数据挖掘篇我们来对这些特征进行处理并训练模型。

不足之处，欢迎指正。