{ "cells": [ { "cell_type": "markdown", "id": "3b57686b-7ac8-4897-bf76-3d982b1ff8da", "metadata": {}, "source": [ "

本科生《深度学习》课程
实验报告

\n", "
\n", "
课程名称:深度学习
\n", "
实验题目:Pytorch基本操作
\n", "
学号:21281280
\n", "
姓名:柯劲帆
\n", "
班级:物联网2101班
\n", "
指导老师:张淳杰
\n", "
报告日期:2023年10月9日
\n", "
" ] }, { "cell_type": "markdown", "id": "e24aa17e-faf9-4d69-9eae-43159116b56f", "metadata": {}, "source": [ "实验环境：\n", "- OS：Ubuntu 22.04 内核版本 6.2.0-34-generic\n", "- CPU：12th Gen Intel(R) Core(TM) i7-12700H\n", "- GPU：NVIDIA GeForce RTX 3070 Ti Laptop\n", "- conda: miniconda 23.9.0\n", "- python：3.10.13\n", "- pytorch：2.1.0" ] }, { "cell_type": "code", "execution_count": 1, "id": "a4e12268-bad4-44c4-92d5-883624d93e25", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import torch\n", "from torch.autograd import Variable\n", "from torch.utils.data import Dataset, DataLoader\n", "from torch import nn\n", "from torchvision import datasets, transforms" ] }, { "cell_type": "markdown", "id": "cc7f0ce5-d613-425b-807c-78115632cd80", "metadata": {}, "source": [ "引用相关库。" ] }, { "cell_type": "markdown", "id": "59a43d35-56ac-4ade-995d-1c6fcbcd1262", "metadata": {}, "source": [ "# 一、Pytorch基本操作考察\n", "## 题目2\n", "**使用 𝐓𝐞𝐧𝐬𝐨𝐫 初始化一个 𝟏×𝟑 的矩阵 𝑴 和一个 𝟐×𝟏 的矩阵 𝑵，对两矩阵进行减法操作（要求实现三种不同的形式），给出结果并分析三种方式的不同（如果出现报错，分析报错的原因），同时需要指出在计算过程中发生了什么。**" ] }, { "cell_type": "code", "execution_count": 2, "id": "79ea46db-cf49-436c-9b5b-c6562d0da9e2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "方法1的结果:\n", "tensor([[-3, -2, -1],\n", " [-4, -3, -2]])\n", "方法2的结果:\n", "tensor([[-3, -2, -1],\n", " [-4, -3, -2]])\n", "方法3的结果:\n", "tensor([[-3, -2, -1],\n", " [-4, -3, -2]])\n" ] } ], "source": [ "A = torch.tensor([[1, 2, 3]])\n", "\n", "B = torch.tensor([[4],\n", " [5]])\n", "\n", "# 方法1: 使用PyTorch的减法操作符\n", "result1 = A - B\n", "\n", "# 方法2: 使用PyTorch的sub函数\n", "result2 = torch.sub(A, B)\n", "\n", "# 方法3: 手动实现广播机制并作差\n", "def mysub(a:torch.Tensor, b:torch.Tensor):\n", " if not (\n", " (a.size(0) == 1 and b.size(1) == 1) \n", " or \n", " (a.size(1) == 1 and b.size(0) == 1)\n", " ):\n", " raise ValueError(\"输入的张量大小无法满足广播机制的条件。\")\n", " else:\n", " target_shape = torch.Size([max(A.size(0), B.size(0)), max(A.size(1), B.size(1))])\n", " A_broadcasted = A.expand(target_shape)\n", " B_broadcasted = B.expand(target_shape)\n", " result = torch.zeros(target_shape, dtype=torch.int64).to(device=A_broadcasted.device)\n", " for i in range(target_shape[0]):\n", " for j in range(target_shape[1]):\n", " result[i, j] = A_broadcasted[i, j] - B_broadcasted[i, j]\n", " return result\n", "\n", "result3 = mysub(A, B)\n", "\n", "print(\"方法1的结果:\")\n", "print(result1)\n", "print(\"方法2的结果:\")\n", "print(result2)\n", "print(\"方法3的结果:\")\n", "print(result3)" ] }, { "cell_type": "markdown", "id": "2489a3ad-f6ff-4561-bb26-e02654090b98", "metadata": {}, "source": [ "## 题目2\n", "1. **利用Tensor创建两个大小分别3*2和4*2的随机数矩阵P和Q，要求服从均值为0，标准差0.01为的正态分布；**\n", "2. **对第二步得到的矩阵Q进行形状变换得到Q的转置Q^T；**\n", "3. **对上述得到的矩阵P和矩阵Q^T求矩阵相乘。**" ] }, { "cell_type": "code", "execution_count": 3, "id": "41e4ee02-1d05-4101-b3f0-477bac0277fb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "矩阵 P:\n", "tensor([[ 0.0053, 0.0013],\n", " [-0.0086, 0.0136],\n", " [-0.0013, 0.0176]])\n", "矩阵 Q:\n", "tensor([[ 0.0044, 0.0014],\n", " [ 0.0147, 0.0078],\n", " [-0.0002, -0.0023],\n", " [ 0.0001, -0.0011]])\n", "矩阵 QT:\n", "tensor([[ 0.0044, 0.0147, -0.0002, 0.0001],\n", " [ 0.0014, 0.0078, -0.0023, -0.0011]])\n", "矩阵相乘的结果:\n", "tensor([[ 2.4953e-05, 8.7463e-05, -3.8665e-06, -8.9576e-07],\n", " [-1.9514e-05, -2.0557e-05, -2.9649e-05, -1.5913e-05],\n", " [ 1.8189e-05, 1.1834e-04, -4.0097e-05, -1.9608e-05]])\n" ] } ], "source": [ "mean = 0\n", "stddev = 0.01\n", "\n", "P = torch.normal(mean=mean, std=stddev, size=(3, 2))\n", "Q = torch.normal(mean=mean, std=stddev, size=(4, 2))\n", "\n", "print(\"矩阵 P:\")\n", "print(P)\n", "print(\"矩阵 Q:\")\n", "print(Q)\n", "\n", "# 对矩阵Q进行转置操作，得到矩阵Q的转置Q^T\n", "QT = Q.T\n", "print(\"矩阵 QT:\")\n", "print(QT)\n", "\n", "# 计算矩阵P和矩阵Q^T的矩阵相乘\n", "result = torch.matmul(P, QT)\n", "print(\"矩阵相乘的结果:\")\n", "print(result)" ] }, { "cell_type": "markdown", "id": "cea9cb6d-adde-4e08-b9f2-8c417abf4231", "metadata": {}, "source": [ "## 题目2\n", "**给定公式$ y_3=y_1+y_2=𝑥^2+𝑥^3$，且$x=1$。利用学习所得到的Tensor的相关知识，求$y_3$对$x$的梯度，即$\\frac{dy_3}{dx}$。**" ] }, { "cell_type": "code", "execution_count": 4, "id": "951512cd-d915-4d04-959f-eb99d1971e2d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "梯度(dy_3/dx): 2.0\n" ] } ], "source": [ "x = torch.tensor(1.0, requires_grad=True)\n", "y_1 = x ** 2\n", "with torch.no_grad():\n", " y_2 = x**3\n", "\n", "y3 = y_1 + y_2\n", "\n", "y3.backward()\n", "\n", "print(\"梯度(dy_3/dx): \", x.grad.item())" ] }, { "cell_type": "markdown", "id": "3269dbf6-889a-49eb-8094-1e588e1a6c30", "metadata": {}, "source": [ "# 二、动手实现logistic回归\n", "## 题目1\n", "**要求动手从0实现 logistic 回归（只借助Tensor和Numpy相关的库）在人工构造的数据集上进行训练和测试，并从loss以及训练集上的准确率等多个角度对结果进行分析（可借助nn.BCELoss或nn.BCEWithLogitsLoss作为损失函数，从零实现二元交叉熵为选作）**" ] }, { "cell_type": "markdown", "id": "bcd12aa9-f187-4d88-8c59-af6d16107edb", "metadata": {}, "source": [ "给定预测概率$ \\left( \\hat{y} \\right) $和目标标签$ \\left( y \\right)$（通常是0或1），BCELoss的计算公式如下：\n", "$$\n", " \\text{BCELoss}(\\hat{y}, y) = -\\frac{1}{N} \\sum_{i=1}^{N} \\left(y_i \\cdot \\log(\\hat{y}_i) + (1 - y_i) \\cdot \\log(1 - \\hat{y}_i)\\right) \n", "$$\n", "其中，$\\left( N \\right) $是样本数量，$\\left( \\hat{y}_i \\right) $表示模型的预测概率向量中的第$ \\left( i \\right) $个元素，$\\left( y_i \\right) $表示实际的目标标签中的第$ \\left( i \\right) $个元素。在二分类问题中，$\\left( y_i \\right) $通常是0或1。这个公式表示对所有样本的二分类交叉熵损失进行了求和并取平均。\n", "\n", "因此BCELoss的手动实现如下。" ] }, { "cell_type": "code", "execution_count": 5, "id": "e31b86ec-4114-48dd-8d73-fe4e0686419a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "输入：\n", "tensor([0.6900])\n", "标签：\n", "tensor([1.])\n", "My_BCELoss损失值: 0.37110066413879395\n", "nn.BCELoss损失值: 0.37110066413879395\n" ] } ], "source": [ "class My_BCELoss:\n", " def __call__(self, prediction: torch.Tensor, target: torch.Tensor):\n", " loss = -torch.mean(target * torch.log(prediction) + (1 - target) * torch.log(1 - prediction))\n", " return loss\n", "\n", "\n", "# 测试\n", "prediction = torch.sigmoid(torch.tensor([0.8]))\n", "target = torch.tensor([1.0])\n", "print(f\"输入：\\n{prediction}\")\n", "print(f\"标签：\\n{target}\")\n", "\n", "my_bce_loss = My_BCELoss()\n", "my_loss = my_bce_loss(prediction, target)\n", "print(\"My_BCELoss损失值:\", my_loss.item())\n", "\n", "nn_bce_loss = nn.BCELoss()\n", "nn_loss = nn_bce_loss(prediction, target)\n", "print(\"nn.BCELoss损失值:\", nn_loss.item())" ] }, { "cell_type": "markdown", "id": "345b0300-8808-4c43-9bf9-05a7e6e1f5af", "metadata": {}, "source": [ "Optimizer的实现较为简单。\n", "\n", "主要实现：\n", "- 传入参数：`__init__()`\n", "- 对传入的参数进行更新：`step()`\n", "- 清空传入参数存储的梯度：`zero_grad()`" ] }, { "cell_type": "code", "execution_count": 6, "id": "0297066c-9fc1-448d-bdcb-29a6f1519117", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "y.backward()之后，x的梯度: 2.0\n", "optimizer_test.step()之后，x的值: 0.800000011920929\n", "optimizer_test.zero_grad()之后，x的梯度: 0.0\n" ] } ], "source": [ "class My_optimizer:\n", " def __init__(self, params: list[torch.Tensor], lr: float):\n", " self.params = params\n", " self.lr = lr\n", "\n", " def step(self):\n", " for param in self.params:\n", " param.data = param.data - self.lr * param.grad.data\n", "\n", " def zero_grad(self):\n", " for param in self.params:\n", " if param.grad is not None:\n", " param.grad.data.zero_()\n", "\n", "\n", "# 测试\n", "x = torch.tensor(1.0, requires_grad=True)\n", "y = x ** 2\n", "optimizer_test = My_optimizer([x], lr=0.1)\n", "\n", "y.backward()\n", "print(\"y.backward()之后，x的梯度: \", x.grad.item())\n", "\n", "optimizer_test.step()\n", "print(\"optimizer_test.step()之后，x的值: \", x.item())\n", "\n", "optimizer_test.zero_grad()\n", "print(\"optimizer_test.zero_grad()之后，x的梯度: \", x.grad.item())" ] }, { "cell_type": "markdown", "id": "6ab83528-a88b-4d66-b0c9-b1315cf75c22", "metadata": {}, "source": [ "线性层主要有一个权重（weight)和一个偏置（bias）。\n", "线性层的数学公式如下：\n", "$$\n", "x:=x \\times weight^T+bias\n", "$$\n", "因此代码实现如下：" ] }, { "cell_type": "code", "execution_count": 7, "id": "8e18695a-d8c5-4f77-8b5c-de40d9240fb9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "输入：\n", "tensor([[1.],\n", " [2.]], requires_grad=True)\n", "权重：\n", "tensor([[-1.0980],\n", " [-0.5413],\n", " [ 1.5884]], requires_grad=True)\n", "偏置：\n", "tensor([-1.1733], requires_grad=True)\n", "输出：\n", "tensor([[-2.2713, -1.7146, 0.4151],\n", " [-3.3692, -2.2559, 2.0036]], grad_fn=)\n" ] } ], "source": [ "class My_Linear:\n", " def __init__(self, input_feature: int, output_feature: int):\n", " self.weight = torch.randn((output_feature, input_feature), requires_grad=True, dtype=torch.float32)\n", " self.bias = torch.randn(1, requires_grad=True, dtype=torch.float32)\n", " self.params = [self.weight, self.bias]\n", "\n", " def __call__(self, x: torch.Tensor):\n", " return self.forward(x)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = torch.matmul(x, self.weight.T) + self.bias\n", " return x\n", "\n", " def to(self, device: str):\n", " for param in self.params:\n", " param.data = param.data.to(device=device)\n", " return self\n", "\n", " def parameters(self):\n", " return self.params\n", "\n", " \n", "# 测试\n", "linear_test = My_Linear(1, 3)\n", "x = torch.tensor([[1.], [2.]], requires_grad=True)\n", "print(f\"输入：\\n{x}\")\n", "print(f\"权重：\\n{linear_test.weight}\")\n", "print(f\"偏置：\\n{linear_test.bias}\")\n", "y = linear_test(x)\n", "print(f\"输出：\\n{y}\")" ] }, { "cell_type": "markdown", "id": "5ff813cc-c1f0-4c73-a3e8-d6796ef5d366", "metadata": {}, "source": [ "手动实现logistic回归模型。\n", "\n", "模型很简单，主要由一个线性层和一个sigmoid层组成。\n", "\n", "Sigmoid函数（又称为 Logistic函数）是一种常用的激活函数，通常用于神经网络的输出层或隐藏层，其作用是将输入的实数值压缩到一个范围在0和1之间的数值。" ] }, { "cell_type": "code", "execution_count": 8, "id": "e7de7e4b-a084-4793-812e-46e8550ecd8d", "metadata": {}, "outputs": [], "source": [ "class Model_2_1():\n", " def __init__(self):\n", " self.linear = My_Linear(1, 1)\n", " self.params = self.linear.params\n", "\n", " def __call__(self, x):\n", " return self.forward(x)\n", "\n", " def forward(self, x):\n", " x = self.linear(x)\n", " x = torch.sigmoid(x)\n", " return x\n", "\n", " def to(self, device: str):\n", " for param in self.params:\n", " param.data = param.data.to(device=device)\n", " return self\n", "\n", " def parameters(self):\n", " return self.params" ] }, { "cell_type": "markdown", "id": "e14acea9-e5ef-4c24-aea9-329647224ce1", "metadata": {}, "source": [ "人工随机构造数据集。\n", "\n", "这里我遇到了比较大的问题。因为数据构建不合适，会导致后面的训练出现梯度爆炸。\n", "\n", "我采用随机产生数据后归一化的方法，即\n", "$$\n", "\\hat{x} = \\frac{x - \\text{min}_x}{\\text{max}_x - \\text{min}_x} \n", "$$\n", "将数据控制在合适的区间。\n", "\n", "我的y设置为$4-3\\times x + noise$，noise为随机噪声。\n", "\n", "生成完x和y后进行归一化处理，并写好DataLoader访问数据集的接口`__getitem__()`。" ] }, { "cell_type": "code", "execution_count": 9, "id": "c39fbafb-62e4-4b8c-9d65-6718d25f2970", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "测试数据集大小：100\n", "测试数据集第0对数据：\n", "x_0 = 0.5531462811708403\n", "y_0 = 0.42036701080526284\n" ] } ], "source": [ "class My_Dataset(Dataset):\n", " def __init__(self, data_size=1000000):\n", " np.random.seed(0)\n", " x = 2 * np.random.rand(data_size, 1)\n", " noise = 0.2 * np.random.randn(data_size, 1)\n", " y = 4 - 3 * x + noise\n", " self.min_x, self.max_x = np.min(x), np.max(x)\n", " min_y, max_y = np.min(y), np.max(y)\n", " x = (x - self.min_x) / (self.max_x - self.min_x)\n", " y = (y - min_y) / (max_y - min_y)\n", " self.data = [[x[i][0], y[i][0]] for i in range(x.shape[0])]\n", "\n", " def __len__(self):\n", " return len(self.data)\n", "\n", " def __getitem__(self, index):\n", " x, y = self.data[index]\n", " return x, y\n", "\n", "\n", "# 测试\n", "dataset_test = My_Dataset(data_size=100)\n", "dataset_size = len(dataset_test)\n", "print(f\"测试数据集大小：{dataset_size}\")\n", "x0, y0 = dataset_test[0]\n", "print(f\"测试数据集第0对数据：\")\n", "print(f\"x_0 = {x0}\")\n", "print(f\"y_0 = {y0}\")" ] }, { "cell_type": "markdown", "id": "957a76a2-b306-47a8-912e-8fbf00cdfd42", "metadata": {}, "source": [ "训练Logistic回归模型。\n", "进行如下步骤：\n", "1. 初始化超参数\n", "2. 获取数据集\n", "3. 初始化模型\n", "4. 定义损失函数和优化器\n", "5. 训练\n", " 1. 从训练dataloader中获取批量数据\n", " 2. 传入模型\n", " 3. 使用损失函数计算与ground_truth的损失\n", " 4. 使用优化器进行反向传播\n", " 5. 循环以上步骤\n", "6. 测试\n", " 1. 设置测试数据\n", " 2. 传入模型\n", " 3. 得到预测值" ] }, { "cell_type": "code", "execution_count": 10, "id": "5612661e-2809-4d46-96c2-33ee9f44116d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/10, Loss: 680.9198314547539, Acc: 0.9677169744703272\n", "Epoch 2/10, Loss: 677.2582936882973, Acc: 0.9985965700887113\n", "Epoch 3/10, Loss: 677.1911396384239, Acc: 0.9993738265049104\n", "Epoch 4/10, Loss: 677.1777537465096, Acc: 0.9995470920810262\n", "Epoch 5/10, Loss: 677.1745615005493, Acc: 0.9998228389835642\n", "Epoch 6/10, Loss: 677.1743944883347, Acc: 0.9999690339979311\n", "Epoch 7/10, Loss: 677.1735371947289, Acc: 0.9998205132243208\n", "Epoch 8/10, Loss: 677.1737813353539, Acc: 0.999798559017381\n", "Epoch 9/10, Loss: 677.1740361452103, Acc: 0.9998672931901137\n", "Epoch 10/10, Loss: 677.1736125349998, Acc: 0.9997257713704987\n", "Model weights: -0.0006128809181973338, bias: 0.023128816857933998\n", "Prediction for test data: 0.505628764629364\n" ] } ], "source": [ "learning_rate = 5e-2\n", "num_epochs = 10\n", "batch_size = 1024\n", "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "dataset = My_Dataset()\n", "dataloader = DataLoader(\n", " dataset=dataset, batch_size=batch_size, shuffle=True, num_workers=5, pin_memory=True\n", ")\n", "\n", "model = Model_2_1().to(device)\n", "criterion = My_BCELoss()\n", "optimizer = My_optimizer(model.parameters(), lr=learning_rate)\n", "\n", "for epoch in range(num_epochs):\n", " total_epoch_loss = 0\n", " total_epoch_pred = 0\n", " total_epoch_target = 0\n", " for x, targets in dataloader:\n", " optimizer.zero_grad()\n", " \n", " x = x.to(device).to(dtype=torch.float32)\n", " targets = targets.to(device).to(dtype=torch.float32)\n", " \n", " x = x.unsqueeze(1)\n", " y_pred = model(x)\n", " loss = criterion(y_pred, targets)\n", " total_epoch_loss += loss.item()\n", " total_epoch_target += targets.sum().item()\n", " total_epoch_pred += y_pred.sum().item()\n", "\n", " loss.backward()\n", " optimizer.step()\n", "\n", " print(\n", " f\"Epoch {epoch + 1}/{num_epochs}, Loss: {total_epoch_loss}, Acc: {1 - abs(total_epoch_pred - total_epoch_target) / total_epoch_target}\"\n", " )\n", "\n", "with torch.no_grad():\n", " test_data = (np.array([[2]]) - dataset.min_x) / (dataset.max_x - dataset.min_x)\n", " test_data = Variable(\n", " torch.tensor(test_data, dtype=torch.float32), requires_grad=False\n", " ).to(device)\n", " predicted = model(test_data).to(\"cpu\")\n", " print(\n", " f\"Model weights: {model.linear.weight.item()}, bias: {model.linear.bias.item()}\"\n", " )\n", " print(f\"Prediction for test data: {predicted.item()}\")" ] }, { "cell_type": "markdown", "id": "9e416582-a30d-4084-acc6-6e05f80a6aff", "metadata": {}, "source": [ "## 题目2\n", "**利用 torch.nn 实现 logistic 回归在人工构造的数据集上进行训练和测试，并对结果进行分析，并从loss以及训练集上的准确率等多个角度对结果进行分析**" ] }, { "cell_type": "markdown", "id": "0460d125-7d03-44fe-845c-c4d13792e241", "metadata": {}, "source": [ "使用torch.nn实现模型。\n", "\n", "将之前的Model_2_1中的手动实现函数改为torch.nn内置函数即可，再加上继承nn.Module以使用torch.nn内置模型模板特性。" ] }, { "cell_type": "code", "execution_count": 11, "id": "fa121afd-a1af-4193-9b54-68041e0ed068", "metadata": {}, "outputs": [], "source": [ "class Model_2_2(nn.Module):\n", " def __init__(self):\n", " super(Model_2_2, self).__init__()\n", " self.linear = nn.Linear(1, 1, dtype=torch.float64)\n", "\n", " def forward(self, x):\n", " x = self.linear(x)\n", " x = torch.sigmoid(x)\n", " return x" ] }, { "cell_type": "markdown", "id": "176eee7e-4e3d-470e-8af2-8761bca039f8", "metadata": {}, "source": [ "训练与测试过程与之前手动实现的几乎一致。仅有少量涉及数据类型（dtype）的代码需要更改以适应torch.nn的内置函数要求。" ] }, { "cell_type": "code", "execution_count": 12, "id": "93b0fdb6-be8b-4663-b59e-05ed19a9ea09", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/10, Loss: 600.8090852049173, Acc: 0.9945839732715815\n", "Epoch 2/10, Loss: 565.9542879898308, Acc: 0.9999073566261442\n", "Epoch 3/10, Loss: 565.9275637627202, Acc: 0.9999969933728429\n", "Epoch 4/10, Loss: 565.927609191542, Acc: 0.9999961959888584\n", "Epoch 5/10, Loss: 565.928202885308, Acc: 0.9999953721249991\n", "Epoch 6/10, Loss: 565.9323843971484, Acc: 0.9999969051674709\n", "Epoch 7/10, Loss: 565.9298919086365, Acc: 0.9999935973983517\n", "Epoch 8/10, Loss: 565.9299267993255, Acc: 0.9999985970973472\n", "Epoch 9/10, Loss: 565.9306044380719, Acc: 0.9999947955797296\n", "Epoch 10/10, Loss: 565.9329843268798, Acc: 0.9999973784035556\n", "Model weights: -3.7066140776793373, bias: 1.8709382558479912\n", "Prediction for test data: 0.13756338580653613\n" ] } ], "source": [ "learning_rate = 1e-2\n", "num_epochs = 10\n", "batch_size = 1024\n", "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "dataset = My_Dataset()\n", "dataloader = DataLoader(\n", " dataset=dataset, batch_size=batch_size, shuffle=True, num_workers=5, pin_memory=True\n", ")\n", "\n", "model = Model_2_2().to(device)\n", "criterion = nn.BCELoss()\n", "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n", "\n", "for epoch in range(num_epochs):\n", " total_epoch_loss = 0\n", " total_epoch_pred = 0\n", " total_epoch_target = 0\n", " for x, targets in dataloader:\n", " optimizer.zero_grad()\n", "\n", " x = x.to(device)\n", " targets = targets.to(device)\n", "\n", " x = x.unsqueeze(1)\n", " targets = targets.unsqueeze(1)\n", " y_pred = model(x)\n", " loss = criterion(y_pred, targets)\n", " total_epoch_loss += loss.item()\n", " total_epoch_target += targets.sum().item()\n", " total_epoch_pred += y_pred.sum().item()\n", "\n", " loss.backward()\n", " optimizer.step()\n", "\n", " print(\n", " f\"Epoch {epoch + 1}/{num_epochs}, Loss: {total_epoch_loss}, Acc: {1 - abs(total_epoch_pred - total_epoch_target) / total_epoch_target}\"\n", " )\n", "\n", "with torch.no_grad():\n", " test_data = (np.array([[2]]) - dataset.min_x) / (dataset.max_x - dataset.min_x)\n", " test_data = Variable(\n", " torch.tensor(test_data, dtype=torch.float64), requires_grad=False\n", " ).to(device)\n", " predicted = model(test_data).to(\"cpu\")\n", " print(\n", " f\"Model weights: {model.linear.weight.item()}, bias: {model.linear.bias.item()}\"\n", " )\n", " print(f\"Prediction for test data: {predicted.item()}\")\n" ] }, { "cell_type": "markdown", "id": "e6bff679-f8d2-46cc-bdcb-82af7dab38b3", "metadata": {}, "source": [ "对比发现，使用torch.nn的内置损失函数和优化器，正确率提升更快。\n", "\n", "但是为什么相同分布的数据集训练出的权重和偏置，以及预测结果存在较大差别，这个问题的原因还有待我探究。" ] }, { "cell_type": "markdown", "id": "ef41d7fa-c2bf-4024-833b-60af0a87043a", "metadata": {}, "source": [ "# 三、动手实现softmax回归\n", "\n", "## 问题1\n", "\n", "**要求动手从0实现softmax回归（只借助Tensor和Numpy相关的库）在Fashion-MNIST数据集上进行训练和测试，并从loss、训练集以及测试集上的准确率等多个角度对结果进行分析（要求从零实现交叉熵损失函数）**" ] }, { "cell_type": "markdown", "id": "3c356760-75a8-4814-ba69-73b270396a4e", "metadata": {}, "source": [ "手动实现nn.one_hot()。\n", "\n", "one-hot向量用于消除线性标签值所映射的类别的非线性。\n", "\n", "one-hot向量是使用一个长度为分类数量的数组表示标签值，其中有且仅有1个值为为1，该值的下标为标签值；其余为0。\n", "\n", "原理很简单，步骤如下：\n", "1. 初始化全零的张量，大小为（标签数量，分类数量）；\n", "2. 将标签值映射到全零张量的\\[下标，标签值\\]中，将该位置为1；\n", "3. 返回修改后的张量，即是ont-hot向量。" ] }, { "cell_type": "code", "execution_count": 13, "id": "e605f1b0-1d32-410f-bddf-402a85ccc9ff", "metadata": {}, "outputs": [], "source": [ "def my_one_hot(indices: torch.Tensor, num_classes: int):\n", " one_hot_tensor = torch.zeros(len(indices), num_classes).to(indices.device)\n", " one_hot_tensor.scatter_(1, indices.view(-1, 1), 1)\n", " return one_hot_tensor" ] }, { "cell_type": "markdown", "id": "902603a6-bfb9-4ce3-bd0d-b00cebb1d3cb", "metadata": {}, "source": [ "手动实现CrossEntropyLoss。\n", "\n", "CrossEntropyLoss由一个log_softmax和一个nll_loss组成。\n", "\n", "softmax的数学表达式如下：\n", "$$\n", "\\text{softmax}(y_i) = \\frac{e^{y_i - \\text{max}(y)}}{\\sum_{j=1}^{N} e^{y_j - \\text{max}(y)}} \n", "$$\n", "log_softmax即为$\\log\\left(softmax\\left(y\\right)\\right)$。\n", "\n", "CrossEntropyLoss的数学表达式如下：\n", "$$\n", "\\text{CrossEntropyLoss}(y, \\hat{y}) = -\\frac{1}{N} \\sum_{i=1}^{N} \\hat{y}_i \\cdot \\log(\\text{softmax}(y_i)) \n", "$$\n", "\n", "故代码如下：" ] }, { "cell_type": "code", "execution_count": 14, "id": "759a3bb2-b5f4-4ea5-a2d7-15f0c4cdd14b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "输入：\n", "tensor([[-1.2914, 0.4715, -0.0432, 1.7427, -1.9236],\n", " [ 0.5361, -0.7551, -0.6810, 1.0945, 0.6135],\n", " [-1.3398, -0.0026, -1.6066, -0.4659, -1.6076]], requires_grad=True)\n", "标签：\n", "tensor([[1., 0., 0., 0., 0.],\n", " [0., 0., 0., 0., 1.],\n", " [0., 0., 0., 0., 1.]])\n", "My_CrossEntropyLoss损失值: 2.4310648441314697\n", "nn.CrossEntropyLoss损失值: 2.4310646057128906\n" ] } ], "source": [ "class My_CrossEntropyLoss:\n", " def __call__(self, predictions: torch.Tensor, targets: torch.Tensor):\n", " max_values = torch.max(predictions, dim=1, keepdim=True).values\n", " exp_values = torch.exp(predictions - max_values)\n", " softmax_output = exp_values / torch.sum(exp_values, dim=1, keepdim=True)\n", " log_probs = torch.log(softmax_output)\n", " \n", " nll_loss = -torch.sum(targets * log_probs, dim=1)\n", " average_loss = torch.mean(nll_loss)\n", " return average_loss\n", "\n", " \n", "# 测试\n", "input = torch.randn(3, 5, requires_grad=True)\n", "target = torch.randn(3, 5).softmax(dim=1).argmax(1)\n", "target = torch.nn.functional.one_hot(target, num_classes=5).to(dtype=torch.float32)\n", "print(f\"输入：\\n{input}\")\n", "print(f\"标签：\\n{target}\")\n", "\n", "my_crossentropyloss = My_CrossEntropyLoss()\n", "my_loss = my_crossentropyloss(input, target)\n", "print(\"My_CrossEntropyLoss损失值:\", my_loss.item())\n", "\n", "nn_crossentropyloss = nn.CrossEntropyLoss()\n", "nn_loss = nn_crossentropyloss(input, target)\n", "print(\"nn.CrossEntropyLoss损失值:\", nn_loss.item())" ] }, { "cell_type": "markdown", "id": "dbf78501-f5be-4008-986c-d331d531491f", "metadata": {}, "source": [ "手动实现Flatten。\n", "\n", "原理很简单，就是把多维的张量拉直成一个向量。" ] }, { "cell_type": "code", "execution_count": 15, "id": "74322629-8325-4823-b80f-f28182d577c1", "metadata": {}, "outputs": [], "source": [ "class My_Flatten:\n", " def __call__(self, x: torch.Tensor):\n", " return self.forward(x)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = x.view(x.shape[0], -1)\n", " return x" ] }, { "cell_type": "markdown", "id": "35aee905-ae37-4faa-a7f1-a04cd8579f78", "metadata": {}, "source": [ "手动实现softmax回归模型。\n", "\n", "模型很简单，主要由一个Flatten层和一个线性层组成。\n", "\n", "Flatten层主要用于将2维的图像展开，直接作为1维的特征量输入网络。" ] }, { "cell_type": "code", "execution_count": 16, "id": "bb31a75e-464c-4b94-b927-b219a765e35d", "metadata": {}, "outputs": [], "source": [ "class Model_3_1:\n", " def __init__(self, num_classes):\n", " self.flatten = My_Flatten()\n", " self.linear = My_Linear(28 * 28, num_classes)\n", " self.params = self.linear.params\n", "\n", " def __call__(self, x: torch.Tensor):\n", " return self.forward(x)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = self.flatten(x)\n", " x = self.linear(x)\n", " return x\n", "\n", " def to(self, device: str):\n", " for param in self.params:\n", " param.data = param.data.to(device=device)\n", " return self\n", "\n", " def parameters(self):\n", " return self.params" ] }, { "cell_type": "markdown", "id": "17e686d1-9c9a-4727-8fdc-9990d348c523", "metadata": {}, "source": [ "训练与测试过程与之前手动实现的几乎一致。由于数据集的变化，对应超参数也进行了调整。\n", "\n", "数据集也使用了现成的FashionMNIST数据集，且划分了训练集和测试集。\n", "\n", "FashionMNIST数据集直接调用API获取。数据集的image为28*28的单通道灰白图片，label为单个数值标签。\n", "\n", "训练softmax回归模型。\n", "进行如下步骤：\n", "1. 初始化超参数\n", "2. 获取数据集\n", "3. 初始化模型\n", "4. 定义损失函数和优化器\n", "5. 训练\n", " 1. 从训练dataloader中获取批量数据\n", " 2. 传入模型\n", " 3. 使用损失函数计算与ground_truth的损失\n", " 4. 使用优化器进行反向传播\n", " 5. 循环以上步骤\n", "6. 测试\n", " 1. 从测试dataloader中获取批量数据\n", " 2. 传入模型\n", " 3. 将预测值与ground_truth进行比较，得出正确率\n", " 4. 对整个训练集统计正确率，从而分析训练效果" ] }, { "cell_type": "code", "execution_count": 17, "id": "d816dae1-5fbe-4c29-9597-19d66b5eb6b4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 2/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 3/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 4/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 5/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 6/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 7/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 8/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 9/10, Loss: nan, Acc: 0.09999999403953552\n", "Epoch 10/10, Loss: nan, Acc: 0.09999999403953552\n" ] } ], "source": [ "learning_rate = 5e-3\n", "num_epochs = 10\n", "batch_size = 4096\n", "num_classes = 10\n", "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "transform = transforms.Compose(\n", " [\n", " transforms.ToTensor(),\n", " transforms.Normalize((0.5,), (0.5,)),\n", " ]\n", ")\n", "train_dataset = datasets.FashionMNIST(root=\"./dataset\", train=True, transform=transform, download=True)\n", "test_dataset = datasets.FashionMNIST(root=\"./dataset\", train=False, transform=transform, download=True)\n", "train_loader = DataLoader(\n", " dataset=train_dataset, batch_size=batch_size,\n", " shuffle=True, num_workers=4, pin_memory=True,\n", ")\n", "test_loader = DataLoader(\n", " dataset=test_dataset, batch_size=batch_size,\n", " shuffle=True, num_workers=4, pin_memory=True,\n", ")\n", "\n", "model = Model_3_1(num_classes).to(device)\n", "criterion = My_CrossEntropyLoss()\n", "optimizer = My_optimizer(model.parameters(), lr=learning_rate)\n", "\n", "for epoch in range(num_epochs):\n", " total_epoch_loss = 0\n", " for images, targets in train_loader:\n", " optimizer.zero_grad()\n", "\n", " images = images.to(device)\n", " targets = targets.to(device).to(dtype=torch.long)\n", "\n", " one_hot_targets = (\n", " my_one_hot(targets, num_classes=num_classes)\n", " .to(device)\n", " .to(dtype=torch.long)\n", " )\n", "\n", " outputs = model(images)\n", " loss = criterion(outputs, one_hot_targets)\n", " total_epoch_loss += loss\n", "\n", " loss.backward()\n", " optimizer.step()\n", "\n", " total_acc = 0\n", " with torch.no_grad():\n", " for image, targets in test_loader:\n", " image = image.to(device)\n", " targets = targets.to(device)\n", " outputs = model(image)\n", " total_acc += (outputs.argmax(1) == targets).sum()\n", " print(\n", " f\"Epoch {epoch + 1}/{num_epochs}, Loss: {total_epoch_loss}, Acc: {total_acc / len(test_dataset)}\"\n", " )" ] }, { "cell_type": "markdown", "id": "a49d0165-aeb7-48c0-9b67-956bb08cb356", "metadata": {}, "source": [ "这里发现梯度爆炸。暂时无法解决。" ] }, { "cell_type": "markdown", "id": "3ef5240f-8a11-4678-bfce-f1cbc7e71b77", "metadata": {}, "source": [ "## 问题2\n", "\n", "**利用torch.nn实现softmax回归在Fashion-MNIST数据集上进行训练和测试，并从loss，训练集以及测试集上的准确率等多个角度对结果进行分析**" ] }, { "cell_type": "markdown", "id": "5c4a88c6-637e-4af5-bed5-f644685dcabc", "metadata": {}, "source": [ "使用torch.nn实现模型。\n", "\n", "将之前的Model_3_1中的手动实现函数改为torch.nn内置函数即可，再加上继承nn.Module以使用torch.nn内置模型模板特性。" ] }, { "cell_type": "code", "execution_count": 18, "id": "0163b9f7-1019-429c-8c29-06436d0a4c98", "metadata": {}, "outputs": [], "source": [ "class Model_3_2(nn.Module):\n", " def __init__(self, num_classes):\n", " super(Model_3_2, self).__init__()\n", " self.flatten = nn.Flatten()\n", " self.linear = nn.Linear(28 * 28, num_classes)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = self.flatten(x)\n", " x = self.linear(x)\n", " return x" ] }, { "cell_type": "markdown", "id": "6e765ad7-c1c6-4166-bd7f-361666bd4016", "metadata": {}, "source": [ "训练与测试过程与之前手动实现的几乎一致。" ] }, { "cell_type": "code", "execution_count": 19, "id": "a58a23e1-368c-430a-ad62-0e256dff564d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/10, Loss: 15.148970603942871, Acc: 0.7520999908447266\n", "Epoch 2/10, Loss: 9.012335777282715, Acc: 0.7996999621391296\n", "Epoch 3/10, Loss: 7.9114227294921875, Acc: 0.8095999956130981\n", "Epoch 4/10, Loss: 7.427404403686523, Acc: 0.8215999603271484\n", "Epoch 5/10, Loss: 7.084254264831543, Acc: 0.8277999758720398\n", "Epoch 6/10, Loss: 6.885956287384033, Acc: 0.8274999856948853\n", "Epoch 7/10, Loss: 6.808426380157471, Acc: 0.8327999711036682\n", "Epoch 8/10, Loss: 6.647855758666992, Acc: 0.8323000073432922\n", "Epoch 9/10, Loss: 6.560361862182617, Acc: 0.8317999839782715\n", "Epoch 10/10, Loss: 6.5211310386657715, Acc: 0.8349999785423279\n" ] } ], "source": [ "learning_rate = 5e-3\n", "num_epochs = 10\n", "batch_size = 4096\n", "num_classes = 10\n", "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "transform = transforms.Compose(\n", " [\n", " transforms.ToTensor(),\n", " transforms.Normalize((0.5,), (0.5,)),\n", " ]\n", ")\n", "train_dataset = datasets.FashionMNIST(\n", " root=\"./dataset\", train=True, transform=transform, download=True\n", ")\n", "test_dataset = datasets.FashionMNIST(\n", " root=\"./dataset\", train=False, transform=transform, download=True\n", ")\n", "train_loader = DataLoader(\n", " dataset=train_dataset,\n", " batch_size=batch_size,\n", " shuffle=True,\n", " num_workers=4,\n", " pin_memory=True,\n", ")\n", "test_loader = DataLoader(\n", " dataset=test_dataset,\n", " batch_size=batch_size,\n", " shuffle=True,\n", " num_workers=4,\n", " pin_memory=True,\n", ")\n", "\n", "model = Model_3_2(num_classes).to(device)\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n", "\n", "for epoch in range(num_epochs):\n", " total_epoch_loss = 0\n", " model.train()\n", " for images, targets in train_loader:\n", " optimizer.zero_grad()\n", "\n", " images = images.to(device)\n", " targets = targets.to(device)\n", "\n", " one_hot_targets = (\n", " torch.nn.functional.one_hot(targets, num_classes=num_classes)\n", " .to(device)\n", " .to(dtype=torch.float32)\n", " )\n", "\n", " outputs = model(images)\n", " loss = criterion(outputs, one_hot_targets)\n", " total_epoch_loss += loss\n", "\n", " loss.backward()\n", " optimizer.step()\n", "\n", " model.eval()\n", " total_acc = 0\n", " with torch.no_grad():\n", " for image, targets in test_loader:\n", " image = image.to(device)\n", " targets = targets.to(device)\n", " outputs = model(image)\n", " total_acc += (outputs.argmax(1) == targets).sum()\n", " print(\n", " f\"Epoch {epoch + 1}/{num_epochs}, Loss: {total_epoch_loss}, Acc: {total_acc / len(test_dataset)}\"\n", " )" ] }, { "cell_type": "markdown", "id": "59555b67-1650-4e1a-a98e-7906878bf3d0", "metadata": {}, "source": [ "与手动实现的softmax回归相比较，nn.CrossEntropyLoss比手动实现的My_CrossEntropyLoss更加稳定，没有出现梯度爆炸的情况。" ] }, { "cell_type": "markdown", "id": "f40431f2-e77b-4ead-81a3-ff6451a8e452", "metadata": {}, "source": [ "**实验心得体会**\n", "\n", "通过完成本次Pytorch基本操作实验，让我对Pytorch框架有了更加深入的理解。我接触深度学习主要是在大语言模型领域，比较熟悉微调大模型，但是涉及到底层的深度学习知识，我还有很多短板和不足。这次实验对我这方面的锻炼让我收获良多。\n", "\n", "首先是数据集的设置。如果数据没有进行归一化，很容易出现梯度爆炸。这是在我以前直接使用图片数据集的经历中没有遇到过的问题。\n", "\n", "在实现logistic回归模型时，通过手动实现各个组件如优化器、线性层等，让我对这些模块的工作原理有了更清晰的认识。尤其是在实现广播机制时，需要充分理解张量操作的维度变换规律。而使用Pytorch内置模块进行实现时，通过继承nn.Module可以自动获得許多功能，使代码更加简洁。\n", "\n", "在实现softmax回归时，则遇到了更大的困难。手动实现的模型很容易出现梯度爆炸的问题，而使用Pytorch内置的损失函数和优化器则可以稳定训练。这让我意识到了选择合适的优化方法的重要性。另外，Pytorch强大的自动微分机制也是构建深度神经网络的重要基础。\n", "\n", "通过这个实验，让我对Pytorch框架有了更加直观的感受，也让我看到了仅靠基础模块搭建复杂模型的难点所在。这些经验对我后续使用Pytorch构建数据集模型会很有帮助。" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 5 }