“特征选择是选择用于模型构建的相关特征的子集的过程”，或者换句话说，选择最重要的特征。

在正常情况下，领域知识起着重要作用，我们可以选择我们认为最重要的特征。例如，在预测房价时，卧室和面积通常被认为是重要的。不幸的是，在Do not Overfit II竞赛（https://www.kaggle.com/c/dont-overfit-ii/data）中，领域知识的使用是不可能的，因为我们有一个二元目标和300个连续变量，这迫使我们尝试特征选择技术。

简介

通常，我们将特征选择和降维组合在一起使用。虽然这两种方法都用于减少数据集中的特征数量，但存在很大不同。

特征选择只是选择和排除给定的特征而不改变它们。

降维是将特征转换为较低维度。

在本文中，我们将探索以下特征选择和降维技术：

特征选择

• 删除缺少值的特征
• 删除方差较小的特征
• 删除高度相关的特征
• 单变量特征选择
• 递归特征消除
• 使用SelectFromModel选择特征

维度降低

• PCA

加载数据

导入必须的Python库

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

设置默认绘图参数

%matplotlib inline
plt.rcParams['figure.figsize'] = [20.0, 7.0]
plt.rcParams.update({'font.size': 22})
sns.set_palette('viridis')
sns.set_style('white')
sns.set_context('talk', font_scale=0.8)

加载机器学习数据集

train = pd.read_csv('../input/train.csv')
test = pd.read_csv('../input/test.csv')
print('Train Shape: ', train.shape)
print('Test Shape: ', test.shape)
train.head()

Train Shape: (250, 302)

Test Shape: (19750, 301)

使用seaborns countplot来显示机器学习数据集中问题的分布

fig, ax = plt.subplots()
g = sns.countplot(train.target, palette='viridis')
g.set_xticklabels(['0', '1'])
g.set_yticklabels([])
# function to show values on bars
def show_values_on_bars(axs):
 def _show_on_single_plot(ax): 
 for p in ax.patches:
 _x = p.get_x() + p.get_width() / 2
 _y = p.get_y() + p.get_height()
 value = '{:.0f}'.format(p.get_height())
 ax.text(_x, _y, value, ha="center") 
 if isinstance(axs, np.ndarray):
 for idx, ax in np.ndenumerate(axs):
 _show_on_single_plot(ax)
 else:
 _show_on_single_plot(axs)
show_values_on_bars(ax)
sns.despine(left=True, bottom=True)
plt.xlabel('')
plt.ylabel('')
plt.title('Distribution of Target', fontsize=30)
plt.tick_params(axis='x', which='major', labelsize=15)
plt.show()

基线模型

我们将使用逻辑回归作为基线模型。我们首先将数据分为测试集和训练集，并进行了缩放:

# prepare for modeling
X_train_df = train.drop(['id', 'target'], axis=1)
y_train = train['target']
X_test = test.drop(['id'], axis=1)
# scaling data
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train_df)
X_test = scaler.transform(X_test)
lr = LogisticRegression(solver='liblinear')
rfc = RandomForestClassifier(n_estimators=100)
lr_scores = cross_val_score(lr,
 X_train,
 y_train,
 cv=5,
 scoring='roc_auc')
rfc_scores = cross_val_score(rfc, X_train, y_train, cv=5, scoring='roc_auc')
print('LR Scores: ', lr_scores)
print('RFC Scores: ', rfc_scores)

LR Scores: [0.80729167 0.71875 0.734375 0.80034722 0.66319444]

RFC Scores: [0.66753472 0.61371528 0.69618056 0.63715278 0.65104167]

检查是最重要的特征

# checking which are the most important features
feature_importance = rfc.fit(X_train, y_train).feature_importances_
# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
sorted_idx = sorted_idx[-20:-1:1]
pos = np.arange(sorted_idx.shape[0]) + .5
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X_train_df.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Feature Importance', fontsize=30)
plt.tick_params(axis='x', which='major', labelsize=15)
sns.despine(left=True, bottom=True)
plt.show()

从交叉验证分数的变化可以看出，模型存在过拟合现象。我们可以尝试通过特征选择来提高这些分数。

删除有缺失值的特征

检查缺失值是任何机器学习问题的第一步。然后我们可以删除超过我们定义的阈值的列。

train.isnull().any().any()

False

数据集没有缺失值，因此在此步骤中没有要删除的特征。

删除低方差的特征

在sklearn的特征选择模块中，我们可以找到VarianceThreshold。它删除方差不满足某个阈值的所有特征。默认情况下，它删除了方差为零的特征，或所有样本值相同的特征。

from sklearn import feature_selection
sel = feature_selection.VarianceThreshold()
train_variance = sel.fit_transform(train)
train_variance.shape

(250, 302)

我们可以从上面看到，所有列中都没有相同值的特征，因此我们没有要删除的特征。

删除高度相关的特征

高度相关或共线性的特征可能导致过度拟合。

当一对变量高度相关时，我们可以删除一个变量来减少维度，而不会损失太多信息。我们应该保留哪一个呢?与目标相关性更高的那个。

让我们来探索我们的特征之间的相关性:

# find correlations to target
corr_matrix = train.corr().abs()
print(corr_matrix['target'].sort_values(ascending=False).head(10))

这里我们看到了与目标变量高度相关的特性。特征33与目标相关性最高，但相关值仅为0.37，仅为弱相关。

我们还可以检查特征与其他特征之间的相关性。下面我们可以看到一个相关矩阵。看起来我们所有的特征都不是高度相关的。

# Select upper triangle of correlation matrix
matrix = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
sns.heatmap(matrix)
plt.show;

相关矩阵

让我们尝试删除相关值大于0.5的特征：

# Find index of feature columns with high correlation
to_drop = [column for column in matrix.columns if any(matrix[column] > 0.50)]
print('Columns to drop: ' , (len(to_drop)))

Columns to drop: 0

从上面的相关矩阵可以看出，数据集中没有高度相关的特征。最高的相关性仅为0.37。

单变量特征选择

单变量特征选择是基于单变量统计检验选择最优特征。

我们可以使用sklearn的SelectKBest来选择一些要保留的特征。这种方法使用统计测试来选择与目标相关性最高的特征。这里我们将保留前100个特征。

# feature extraction
k_best = feature_selection.SelectKBest(score_func=feature_selection.f_classif, k=100)
# fit on train set
fit = k_best.fit(X_train, y_train)
# transform train set
univariate_features = fit.transform(X_train)
# checking which are the most important features
feature_importance = rfc.fit(univariate_features, y_train).feature_importances_
# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
sorted_idx = sorted_idx[-20:-1:1]
pos = np.arange(sorted_idx.shape[0]) + .5
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X_train_df.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Feature Importance', fontsize=30)
plt.tick_params(axis='x', which='major', labelsize=15)
sns.despine(left=True, bottom=True)
plt.show()

交叉验证分数比上面的基线有所提高，但是我们仍然可以看到分数的变化，这表明过度拟合。

递归特性消除

递归特征选择通过消除最不重要的特征来实现。它进行递归，直到达到指定数量的特征为止。递归消除可以用于通过coef_或feature_importances_为特征分配权重的任何模型。

在这里，我们将使用随机森林选择100个最好的特征:

# feature extraction
rfe = feature_selection.RFE(lr, n_features_to_select=100)
# fit on train set
fit = rfe.fit(X_train, y_train)
# transform train set
recursive_features = fit.transform(X_train)
lr = LogisticRegression(solver='liblinear')
rfc = RandomForestClassifier(n_estimators=10)
lr_scores = cross_val_score(lr, recursive_features, y_train, cv=5, scoring='roc_auc')
rfc_scores = cross_val_score(rfc, recursive_features, y_train, cv=5, scoring='roc_auc')
print('LR Scores: ', lr_scores)
print('RFC Scores: ', rfc_scores)

LR Scores: [0.99826389 0.99652778 0.984375 1. 0.99652778]

RFC Scores: [0.63368056 0.72569444 0.66666667 0.77430556 0.59895833]

# checking which are the most important features
feature_importance = rfc.fit(recursive_features, y_train).feature_importances_
# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
sorted_idx = sorted_idx[-20:-1:1]
pos = np.arange(sorted_idx.shape[0]) + .5
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X_train_df.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Feature Importance', fontsize=30)
plt.tick_params(axis='x', which='major', labelsize=15)
sns.despine(left=True, bottom=True)
plt.show()

使用SelectFromModel选择特征

与递归特征选择一样，sklearn的SelectFromModel与任何具有coef_或featureimportances属性的估计器一起使用。它删除低于设置阈值的特征。

# feature extraction
select_model = feature_selection.SelectFromModel(lr)
# fit on train set
fit = select_model.fit(X_train, y_train)
# transform train set
model_features = fit.transform(X_train)
lr = LogisticRegression(solver='liblinear')
rfc = RandomForestClassifier(n_estimators=100)
lr_scores = cross_val_score(lr, model_features, y_train, cv=5, scoring='roc_auc')
rfc_scores = cross_val_score(rfc, model_features, y_train, cv=5, scoring='roc_auc')
print('LR Scores: ', lr_scores)
print('RFC Scores: ', rfc_scores)

LR Scores: [0.984375 0.99479167 0.97222222 0.99305556 0.99305556]

RFC Scores: [0.70659722 0.80729167 0.76475694 0.84461806 0.77170139]

# checking which are the most important features
feature_importance = rfc.fit(model_features, y_train).feature_importances_
# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
sorted_idx = sorted_idx[-20:-1:1]
pos = np.arange(sorted_idx.shape[0]) + .5
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X_train_df.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Feature Importance', fontsize=30)
plt.tick_params(axis='x', which='major', labelsize=15)
sns.despine(left=True, bottom=True)
plt.show()

PCA

主成分分析(PCA)是一种降维技术，它将数据投影到较低的维度空间。PCA在许多情况下都是有用的，但在多重共线性或预测函数需要解释的情况下，就不需要优先考虑了。

这里我们将使用PCA，保持90%的方差:

from sklearn.decomposition import PCA
# pca - keep 90% of variance
pca = PCA(0.90)
principal_components = pca.fit_transform(X_train)
principal_df = pd.DataFrame(data = principal_components)
principal_df.shape

(250, 139)

lr = LogisticRegression(solver='liblinear')
rfc = RandomForestClassifier(n_estimators=100)
lr_scores = cross_val_score(lr, principal_df, y_train, cv=5, scoring='roc_auc')
rfc_scores = cross_val_score(rfc, principal_df, y_train, cv=5, scoring='roc_auc')
print('LR Scores: ', lr_scores)
print('RFC Scores: ', rfc_scores)

LR Scores: [0.80902778 0.703125 0.734375 0.80555556 0.66145833]

RFC Scores: [0.60503472 0.703125 0.69878472 0.56597222 0.72916667]

# pca keep 75% of variance
pca = PCA(0.75)
principal_components = pca.fit_transform(X_train)
principal_df = pd.DataFrame(data = principal_components)
principal_df.shape

(250, 93)

lr = LogisticRegression(solver='liblinear')
rfc = RandomForestClassifier(n_estimators=100)
lr_scores = cross_val_score(lr, principal_df, y_train, cv=5, scoring='roc_auc')
rfc_scores = cross_val_score(rfc, principal_df, y_train, cv=5, scoring='roc_auc')
print('LR Scores: ', lr_scores)
print('RFC Scores: ', rfc_scores)

LR Scores: [0.72048611 0.60069444 0.68402778 0.71006944 0.61284722]

RFC Scores: [0.61545139 0.71440972 0.57465278 0.59722222 0.640625 ]

# checking which are the most important features
feature_importance = rfc.fit(principal_df, y_train).feature_importances_
# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
sorted_idx = sorted_idx[-20:-1:1]
pos = np.arange(sorted_idx.shape[0]) + .5
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X_train_df.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Feature Importance', fontsize=30)
plt.tick_params(axis='x', which='major', labelsize=15)
sns.despine(left=True, bottom=True)
plt.show()

# feature extraction
rfe = feature_selection.RFE(lr, n_features_to_select=100)
# fit on train set
fit = rfe.fit(X_train, y_train)
# transform train set
recursive_X_train = fit.transform(X_train)
recursive_X_test = fit.transform(X_test)
lr = LogisticRegression(C=1, class_weight={1:0.6, 0:0.4}, penalty='l1', solver='liblinear')
lr_scores = cross_val_score(lr, recursive_X_train, y_train, cv=5, scoring='roc_auc')
lr_scores.mean()

0.9059027777777778

predictions = lr.fit(recursive_X_train, y_train).predict_proba(recursive_X_test)
submission = pd.read_csv('../input/sample_submission.csv')
submission['target'] = predictions
submission.to_csv('submission.csv', index=False)
submission.head()

结论

特征选择是任何机器学习过程的重要组成部分。在本文中，我们探索了几种有助于提高模型性能的特征选择和降维方法。