Laboratory Task 4

Laboratory Task 4#

Genheylou Felisilda - DS4A

Instruction: Train a linear regression model in PyTorch using a regression dataset. Use the following parameters.

  • Criterion: MSE Loss

  • Fully Connected Layers x 2

  • Batch Size: 8

  • Optimizer: SGD

  • Epoch: 1000

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from sklearn.datasets import load_diabetes
import numpy as np

# Load the diabetes dataset
X, y = load_diabetes(return_X_y=True)

# convert to PyTorch tensors

X_tensor = torch.FloatTensor(X)
y_tensor = torch.FloatTensor(y)

# create dataloader with batch size of 8

dataset = TensorDataset(X_tensor, y_tensor)
dataloader = DataLoader(dataset, batch_size=8, shuffle=True)

# define model with 2 fully connected layers

class LinearRegressionModel(nn.Module):
    def __init__(self, input_size):
        super(LinearRegressionModel, self).__init__()
        self.fc1 = nn.Linear(input_size, 64)  # First FC layer
        self.fc2 = nn.Linear(64, 1)           # Second FC layer
        
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return x

model = LinearRegressionModel(X.shape[1])

# define MSE loss and SGD optimizer

criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)

# train for 1000 epochs

num_epochs = 1000

for epoch in range(num_epochs):
    for batch_X, batch_y in dataloader:
        # Zero gradients
        optimizer.zero_grad()
        
        # Forward pass
        outputs = model(batch_X)
        loss = criterion(outputs.squeeze(), batch_y)
        
        # Backward pass and optimize
        loss.backward()
        optimizer.step()
    
    # Print progress every 100 epochs
    if (epoch + 1) % 100 == 0:
        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

print("Training completed!")

Epoch [100/1000], Loss: 3673.6592

Epoch [200/1000], Loss: 4432.0562

Epoch [300/1000], Loss: 2132.8162

Epoch [400/1000], Loss: 7818.8320

Epoch [500/1000], Loss: 16554.6445

Epoch [600/1000], Loss: 22812.7422

Epoch [700/1000], Loss: 4699.1050

Epoch [800/1000], Loss: 20.9809

Epoch [900/1000], Loss: 1151.0487

Epoch [1000/1000], Loss: 8544.7725
Training completed!

Loss is fluctuating significantly The loss jumps up and down across epochs rather than steadily decreasing. This indicates that the network is not converging properly and the learning process is unstable.

Potential causes of instability:

  • Randomness of data processing

  • Learning rate might be too high, causing the weights to overshoot the minimum.

  • The network architecture or initialization may not be suitable for the data.

  • The dataset might be small or noisy, making it hard for the network to find a consistent pattern.

  • Training is not effective: After 1000 epochs, the loss is still very high, which implies the network has not learned to approximate the target outputs.