py2exe打包oss2和pycryptodome失败问题排查

背景：
最近用python开发一个程序，程序需求在没有安装python的电脑上运行。对比python的打包exe工具之后我选择py2exe(py2exe官方已经不更新，由第三方人员开发维护)。

在使用过程中发现py2exe打包后出现一些文件库丢失，因此特意记录下。

简述：
py2exe是一个Python Distutils扩展，它将Python脚本转换为可执行的Windows程序，无需安装Python即可运行。

这里主要是介绍使用py2exe打包出错的排查思路，py2exe的用法请参考py2exe官网: http://www.py2exe.org/

测试环境：

系统：win10 x64
python3.6.3

依懒:

oss2==2.8.0
py2exe==0.9.3.2  # 下载地址：https://github.com/albertosottile/py2exe/releases
# pycryptodome==3.8.2 安装oss2是会自动下载安装, 安装后的目录名称是Crypto

测试代码app.py

import oss2


class OSSHandler:
    def __init__(self):
        self.endpoint = "endpoint"
        self.auth = oss2.Auth("access_key_id", "access_key_secret")
        self.bucket = oss2.Bucket(self.auth, self.endpoint, "bucket_name")

    def iterator(self):
        """
        遍历bucket文件
        :return:
        """
        for object_info in oss2.ObjectIterator(bucket=self.bucket):
            print(object_info.key)


def main():
    OSSHandler().iterator()


if __name__ == "__main__":
    main()

打包代码setup.py, 注：console要使用终端运行，否则会一闪而过

import py2exe
from distutils.core import setup

py2exe_options = {
    "dist_dir": "dist",
    "compressed": 1,
    "optimize": 2,
    "ascii": 0,
}

setup(
    name='oss',
    version='0.1.0',
    description="win tool",
    options={'py2exe': py2exe_options},
    # 注：console要使用终端运行，否则会一闪而过
    console=[{
        # windows = [{
        "script": "app.py",
    }],
    zipfile="lib/shared.lib",
    data_files=[],
)

打包命令

python setup.py py2exe

报错
运行dist文件夹app.exe得到错误, 发现找不到模块

Traceback (most recent call last):
  File "app.py", line 1, in <module>
    import oss2
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "oss2\__init__.pyc", line 3, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "oss2\models.pyc", line 10, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "oss2\utils.pyc", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "Crypto\Cipher\__init__.pyc", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "Crypto\Cipher\_mode_ecb.pyc", line 47, in <module>
  File "Crypto\Util\_raw_api.pyc", line 300, in load_pycryptodome_raw_lib
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.cp36-win_amd64.pyd': [WinError 126] 找不到指定的模块。, Trying '_raw_ecb.pyd': [WinError 126] 找不到指定的模块。

查看distlibshared.lib文件，用解压工具可以解压查看, 发现py2exe工具没有把*.pyd文件打包进去，导致模块查找失败

修改
根据上面的报错信息可以看到模块调用文件位置是"CryptoUtil_raw_api.pyc", line 300

try:
    filename = basename + ext
    # 通过调试发现pycryptodome_filename函数查到不到*.pyd导致出错
    return load_lib(pycryptodome_filename(dir_comps, filename),
                    cdecl)
except OSError as exp:
    attempts.append("Trying '%s': %s" % (filename, str(exp)))

查看pycryptodome_filename函数CryptoUtil_file_system.py

util_lib, _ = os.path.split(os.path.abspath(__file__))
root_lib = os.path.join(util_lib, "..")
# 添加打印信息，查看模块加载位置
print("root_lib: ", root_lib)

return os.path.join(root_lib, *dir_comps)

添加打印信息，查看模块加载位置, 发现目录是shared.lib下面，但.pyd没有打包进去， 所以只要让程序加载到.pyd就可以了。

root_lib:  D:\code\dist\lib\shared.lib\Crypto\Util\..

修改*.pyd加载目录, 使root_lib加载目录distlib, 注意：此方法会影响未打包的oss和pycryptodome正常使用

# util_lib, _ = os.path.split(os.path.abspath(__file__))
# root_lib = os.path.join(util_lib, "..")
util_lib = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
root_lib = os.path.join(util_lib, "Crypto")
return os.path.join(root_lib, *dir_comps)

重新打包, 复制Crypto文件夹到distlib下，Libsite-packagesCrypto文件夹只需要保留*.pyd文件，其他的可以删除

发现没有报模块加载不到，而是报*.json文件加载不到

参考上面的方法， 修改Libsite-packagesaliyunsdkcoreutils_init_.py文件

# base_dir = os.path.dirname(os.path.abspath(aliyunsdkcore.__file__))
base_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(aliyunsdkcore.__file__))))

复制Libsite-packagesaliyunsdkcoredata到distlib下,最终目录

运行
至此程序就可能正常运行了

通过上面的思路，打包py2exe遇到文件加载不到时，可以通过console模式查找文件加载的目录，然后进行相应的修改。