lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-05-29 17:32:19 +02:00
parent f85d5a1513
commit a093015d3c
2 changed files with 702 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Programs/Regression/Logreg.txt
+313
View File
@@ -0,0 +1,313 @@
From: Morten Hjorth-Jensen <morten.hjorth-jensen@fys.uio.no>
Subject: Logreg
Date: May 29, 2025 at 11:43:36 AM GMT+2
To: Morten Hjorth-Jensen <morten.hjorth-jensen@fys.uio.no>


Logistic Regression from Scratch




Logistic regression is a foundational binary classification technique that models the probability of a sample belonging to class 1 using the sigmoid (logistic) function. In binary classification, it outputs a probability  between 0 and 1, typically thresholded at 0.5 to decide class labels . For multiclass problems (more than two classes), we generalize logistic regression via the softmax function (also known as multinomial logistic regression), which produces a probability distribution over all classes . In both cases, training involves minimizing the cross-entropy loss (also called log-loss) using gradient descent . Accuracy is commonly used to evaluate classification performance (the fraction of correctly predicted samples) , alongside the cross-entropy value to measure model fit.


Our goal is to build an object-oriented Python implementation of logistic regression without scikit-learn, using only standard libraries like NumPy and CSV. The implementation will support both binary and multiclass modes, use gradient descent for training, and include methods for prediction and evaluation. It will also include synthetic data generation (to test the model) and functionality to export predictions and true labels to CSV.




Design and Features




• Class Structure: A LogisticRegression class encapsulates model parameters (weights, bias) and methods for training (fit) and prediction (predict, predict_proba). The design handles binary vs multiclass internally by checking how many unique target labels are present.
• Probability Functions: For binary classification we use the sigmoid function. For multiclass, we use the softmax function, defined as \hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}, which produces a valid probability distribution over classes .
• Training with Gradient Descent: We implement batch gradient descent on the cross-entropy loss. For binary logistic regression the loss is
-\frac{1}{N}\sum_i [y_i\log(p_i) + (1-y_i)\log(1-p_i)]
where p_i=\sigma(\mathbf{w}^T\mathbf{x}i) . For multiclass, the loss (categorical cross-entropy) for one-hot true labels y and predicted probabilities \hat{y} is
-\frac{1}{N}\sum_i \sum{k=1}^C y_{ik}\log(\hat{y}_{ik})
. We compute gradients of these losses w.r.t. the weights and iteratively update the model.
• Prediction: The predict_proba method returns probabilities (sigmoid or softmax). The predict method applies a threshold (0.5) for binary or takes argmax for multiclass.
• Metrics: We include functions for accuracy and cross-entropy loss. Accuracy is defined as the fraction of correct predictions . We compute binary cross-entropy or categorical cross-entropy to quantify model performance on data.
• Synthetic Data: We provide functions to generate synthetic datasets: (a) Binary data with two Gaussian clusters for two classes; (b) Multiclass data with several clusters, one per class. This allows testing the model.
• CSV Export: Using Python’s csv module, we can save the true labels and predicted labels (or probabilities) to CSV files for external analysis.




These components together form a clean, modular implementation. The following sections detail the code with explanations.




LogisticRegression Class Implementation




Below is the LogisticRegression class. It initializes parameters (learning rate, epochs, etc.), and defines private methods for adding an intercept term, sigmoid, and softmax. The fit method detects binary vs multiclass and runs gradient descent accordingly, updating weights. The predict_prob and predict methods compute probabilities and class labels.
import numpy as np


class LogisticRegression:
    """
    Logistic Regression for binary and multiclass classification.
    """
    def __init__(self, lr=0.01, epochs=1000, fit_intercept=True, verbose=False):
        self.lr = lr                  # Learning rate for gradient descent
        self.epochs = epochs          # Number of iterations
        self.fit_intercept = fit_intercept  # Whether to add intercept (bias)
        self.verbose = verbose        # Print loss during training if True
        self.weights = None
        self.multi_class = False      # Will be determined at fit time


    def _add_intercept(self, X):
        """Add intercept term (column of ones) to feature matrix."""
        intercept = np.ones((X.shape[0], 1))
        return np.concatenate((intercept, X), axis=1)


    def _sigmoid(self, z):
        """Sigmoid function for binary logistic."""
        return 1 / (1 + np.exp(-z))


    def _softmax(self, Z):
        """Softmax function for multiclass logistic."""
        exp_Z = np.exp(Z - np.max(Z, axis=1, keepdims=True))
        return exp_Z / np.sum(exp_Z, axis=1, keepdims=True)


    def fit(self, X, y):
        """
        Train the logistic regression model using gradient descent.
        Supports binary (sigmoid) and multiclass (softmax) based on y.
        """
        X = np.array(X)
        y = np.array(y)
        n_samples, n_features = X.shape


        # Add intercept if needed
        if self.fit_intercept:
            X = self._add_intercept(X)
            n_features += 1


        # Determine classes and mode (binary vs multiclass)
        unique_classes = np.unique(y)
        if len(unique_classes) > 2:
            self.multi_class = True
        else:
            self.multi_class = False


        # ----- Multiclass case -----
        if self.multi_class:
            n_classes = len(unique_classes)
            # Map original labels to 0...n_classes-1
            class_to_index = {c: idx for idx, c in enumerate(unique_classes)}
            y_indices = np.array([class_to_index[c] for c in y])
            # Initialize weight matrix (features x classes)
            self.weights = np.zeros((n_features, n_classes))


            # One-hot encode y
            Y_onehot = np.zeros((n_samples, n_classes))
            Y_onehot[np.arange(n_samples), y_indices] = 1


            # Gradient descent
            for epoch in range(self.epochs):
                scores = X.dot(self.weights)          # Linear scores (n_samples x n_classes)
                probs = self._softmax(scores)        # Probabilities (n_samples x n_classes)
                # Compute gradient (features x classes)
                gradient = (1 / n_samples) * X.T.dot(probs - Y_onehot)
                # Update weights
                self.weights -= self.lr * gradient


                if self.verbose and epoch % 100 == 0:
                    # Compute current loss (categorical cross-entropy)
                    loss = -np.sum(Y_onehot * np.log(probs + 1e-15)) / n_samples
                    print(f"[Epoch {epoch}] Multiclass loss: {loss:.4f}")


        # ----- Binary case -----
        else:
            # Convert y to 0/1 if not already
            if not np.array_equal(unique_classes, [0, 1]):
                # Map the two classes to 0 and 1
                class0, class1 = unique_classes
                y_binary = np.where(y == class1, 1, 0)
            else:
                y_binary = y.copy().astype(int)


            # Initialize weights vector (features,)
            self.weights = np.zeros(n_features)


            # Gradient descent
            for epoch in range(self.epochs):
                linear_model = X.dot(self.weights)     # (n_samples,)
                probs = self._sigmoid(linear_model)   # (n_samples,)
                # Gradient for binary cross-entropy
                gradient = (1 / n_samples) * X.T.dot(probs - y_binary)
                self.weights -= self.lr * gradient


                if self.verbose and epoch % 100 == 0:
                    # Compute binary cross-entropy loss
                    loss = -np.mean(
                        y_binary * np.log(probs + 1e-15) + 
                        (1 - y_binary) * np.log(1 - probs + 1e-15)
                    )
                    print(f"[Epoch {epoch}] Binary loss: {loss:.4f}")


    def predict_prob(self, X):
        """
        Compute probability estimates. Returns a 1D array for binary or
        a 2D array (n_samples x n_classes) for multiclass.
        """
        X = np.array(X)
        # Add intercept if the model used it
        if self.fit_intercept:
            X = self._add_intercept(X)
        scores = X.dot(self.weights)
        if self.multi_class:
            return self._softmax(scores)
        else:
            return self._sigmoid(scores)


    def predict(self, X):
        """
        Predict class labels for samples in X.
        Returns integer class labels (0,1 for binary, or 0...C-1 for multiclass).
        """
        probs = self.predict_prob(X)
        if self.multi_class:
            # Choose class with highest probability
            return np.argmax(probs, axis=1)
        else:
            # Threshold at 0.5 for binary
            return (probs >= 0.5).astype(int)
The class implements the sigmoid and softmax internally. During fit(), we check the number of classes: if more than 2, we set self.multi_class=True and perform multinomial logistic regression. We one-hot encode the target vector and update a weight matrix with softmax probabilities. Otherwise, we do standard binary logistic regression, converting labels to 0/1 if needed and updating a weight vector. In both cases we use batch gradient descent on the cross-entropy loss (we add a small epsilon 1e-15 to logs for numerical stability). Progress (loss) can be printed if verbose=True.




Evaluation Metrics




We define helper functions for accuracy and cross-entropy loss. Accuracy is the fraction of correct predictions . For loss, we compute the appropriate cross-entropy:
def accuracy_score(y_true, y_pred):
    """Accuracy = (# correct predictions) / (total samples)."""
    y_true = np.array(y_true)
    y_pred = np.array(y_pred)
    return np.mean(y_true == y_pred)


def binary_cross_entropy(y_true, y_prob):
    """
    Binary cross-entropy loss.
    y_true: true binary labels (0 or 1), y_prob: predicted probabilities for class 1.
    """
    y_true = np.array(y_true)
    y_prob = np.clip(np.array(y_prob), 1e-15, 1-1e-15)
    return -np.mean(y_true * np.log(y_prob) + (1 - y_true) * np.log(1 - y_prob))


def categorical_cross_entropy(y_true, y_prob):
    """
    Categorical cross-entropy loss for multiclass.
    y_true: true labels (0...C-1), y_prob: array of predicted probabilities (n_samples x C).
    """
    y_true = np.array(y_true, dtype=int)
    y_prob = np.clip(np.array(y_prob), 1e-15, 1-1e-15)
    # One-hot encode true labels
    n_samples, n_classes = y_prob.shape
    one_hot = np.zeros_like(y_prob)
    one_hot[np.arange(n_samples), y_true] = 1
    # Compute cross-entropy
    loss_vec = -np.sum(one_hot * np.log(y_prob), axis=1)
    return np.mean(loss_vec)
The binary cross-entropy matches the formula  , and categorical cross-entropy aligns with the standard softmax loss . We clip probabilities to avoid log(0). Accuracy is straightforward (correct/total) .




Synthetic Data Generation




To test the model, we generate synthetic datasets:


• Binary classification data: Create two Gaussian clusters in 2D. For example, class 0 around mean [-2,-2] and class 1 around [2,2].
• Multiclass data: Create several Gaussian clusters (one per class) spread out in feature space.




The code below demonstrates simple generators:
import numpy as np


def generate_binary_data(n_samples=100, n_features=2, random_state=None):
    """
    Generate synthetic binary classification data.
    Returns (X, y) where X is (n_samples x n_features), y in {0,1}.
    """
    rng = np.random.RandomState(random_state)
    # Half samples for class 0, half for class 1
    n0 = n_samples // 2
    n1 = n_samples - n0
    # Class 0 around mean -2, class 1 around +2
    mean0 = -2 * np.ones(n_features)
    mean1 =  2 * np.ones(n_features)
    X0 = rng.randn(n0, n_features) + mean0
    X1 = rng.randn(n1, n_features) + mean1
    X = np.vstack((X0, X1))
    y = np.array([0]*n0 + [1]*n1)
    return X, y


def generate_multiclass_data(n_samples=150, n_features=2, n_classes=3, random_state=None):
    """
    Generate synthetic multiclass data with n_classes Gaussian clusters.
    """
    rng = np.random.RandomState(random_state)
    X = []
    y = []
    samples_per_class = n_samples // n_classes
    for cls in range(n_classes):
        # Random cluster center for each class
        center = rng.uniform(-5, 5, size=n_features)
        Xi = rng.randn(samples_per_class, n_features) + center
        yi = [cls] * samples_per_class
        X.append(Xi)
        y.extend(yi)
    X = np.vstack(X)
    y = np.array(y)
    return X, y
These functions use NumPy to create normally-distributed points. We fix cluster centers to separate classes. Each class’s points are in a distinct region, making the classification problem learnable by logistic regression.




Demo: Training and Evaluation




Finally, we test the implementation on synthetic data. We train on both binary and multiclass cases, then evaluate accuracy and loss, and export results to CSV.
# Generate and test on binary data
X_bin, y_bin = generate_binary_data(n_samples=200, n_features=2, random_state=42)
model_bin = LogisticRegression(lr=0.1, epochs=1000)
model_bin.fit(X_bin, y_bin)
y_prob_bin = model_bin.predict_prob(X_bin)      # probabilities for class 1
y_pred_bin = model_bin.predict(X_bin)           # predicted classes 0 or 1


acc_bin = accuracy_score(y_bin, y_pred_bin)
loss_bin = binary_cross_entropy(y_bin, y_prob_bin)
print(f"Binary Classification - Accuracy: {acc_bin:.2f}, Cross-Entropy Loss: {loss_bin:.2f}")
For multiclass:
# Generate and test on multiclass data
X_multi, y_multi = generate_multiclass_data(n_samples=300, n_features=2, n_classes=3, random_state=1)
model_multi = LogisticRegression(lr=0.1, epochs=1000)
model_multi.fit(X_multi, y_multi)
y_prob_multi = model_multi.predict_prob(X_multi)     # (n_samples x 3) probabilities
y_pred_multi = model_multi.predict(X_multi)          # predicted labels 0,1,2


acc_multi = accuracy_score(y_multi, y_pred_multi)
loss_multi = categorical_cross_entropy(y_multi, y_prob_multi)
print(f"Multiclass Classification - Accuracy: {acc_multi:.2f}, Cross-Entropy Loss: {loss_multi:.2f}")
These print statements show how well the model fits the training data.




CSV Export




To save predictions and true labels, we use Python’s csv module. For example:
import csv


# Export binary results
with open('binary_results.csv', mode='w', newline='') as f:
    writer = csv.writer(f)
    writer.writerow(["TrueLabel", "PredictedLabel"])
    for true, pred in zip(y_bin, y_pred_bin):
        writer.writerow([true, pred])


# Export multiclass results
with open('multiclass_results.csv', mode='w', newline='') as f:
    writer = csv.writer(f)
    writer.writerow(["TrueLabel", "PredictedLabel"])
    for true, pred in zip(y_multi, y_pred_multi):
        writer.writerow([true, pred])
This writes two CSV files with columns for true and predicted labels. One can later analyze or plot these results externally.




Summary




The above implementation provides a clear, object-oriented logistic regression model in pure Python. It handles both binary and multiclass cases by using sigmoid and softmax functions respectively. Training uses batch gradient descent to minimize cross-entropy loss . We include methods for prediction, accuracy calculation , and loss computation, as well as utilities for generating synthetic data and exporting results. This modular design can be extended (e.g., adding regularization or optimization improvements) but already demonstrates the key mechanics of logistic regression end-to-end.


References: Logistic regression concepts and loss functions ; accuracy metric .
Morten Hjorth-Jensen, Michigan State University and University of Oslo, Norway. http://mhjgit.github.io/info/doc/web
doc/Programs/SupportVM/svm.py
+389
View File
@@ -0,0 +1,389 @@
import numpy as np
import csv
class SVM:
def __init__(self, C=1.0, lr=0.001, epochs=1000,
kernel='linear', degree=3, gamma=None, coef0=1, random_features=100):
"""
SVM classifier with linear, polynomial, or (approximate) RBF kernel.
Trains via stochastic subgradient descent on hinge loss.
"""
self.C = C # Regularization parameter
self.lr = lr # Learning rate
self.epochs = epochs # Number of epochs for training
self.kernel = kernel # 'linear', 'poly', or 'rbf'
self.degree = degree # Degree for polynomial kernel
self.coef0 = coef0 # Offset for polynomial kernel (not used in feature expansion here)
self.gamma = gamma # Kernel scale for RBF or polynomial (default 1/n_features)
self.random_features = random_features # Number of random Fourier features for RBF
self.w = None # Weight vector (in feature space)
self.b = 0.0 # Bias term
# For RBF approximation: random projection parameters
self.W = None
self.b_rff = None
def _poly_features(self, X):
"""
Explicit polynomial feature expansion up to self.degree.
E.g., degree=2 for features [x1, x2] gives [x1, x2, x1^2, x1*x2, x2^2].
"""
from itertools import combinations_with_replacement
n_samples, n_features = X.shape
features = []
for deg in range(1, self.degree+1):
for combo in combinations_with_replacement(range(n_features), deg):
feat = np.ones(n_samples)
for i in combo:
feat *= X[:, i]
features.append(feat)
if len(features) > 0:
return np.vstack(features).T
else:
return np.zeros((n_samples, 0))
def _rbf_features(self, X):
"""
Random Fourier feature mapping for RBF kernel approximation:
phi(x) = sqrt(2/D) * [cos(w_i^T x + b_i)]_i.
"""
if self.gamma is None:
self.gamma = 1.0 / X.shape[1]
D = self.random_features
if self.W is None:
self.W = np.sqrt(2 * self.gamma) * np.random.randn(D, X.shape[1])
if self.b_rff is None:
self.b_rff = 2 * np.pi * np.random.rand(D)
# Compute RFF features
return np.sqrt(2.0 / D) * np.cos(np.dot(X, self.W.T) + self.b_rff)
def fit(self, X, y):
"""
Train the SVM. y should be encoded as {-1, +1}.
Uses subgradient descent on the hinge loss (primal form).
"""
X = np.array(X, dtype=float)
y = np.array(y, dtype=float)
# Feature mapping based on kernel choice
if self.kernel == 'linear':
X_train = X.copy()
elif self.kernel == 'poly':
X_train = self._poly_features(X)
elif self.kernel == 'rbf':
X_train = self._rbf_features(X)
else:
raise ValueError("Unsupported kernel type")
n_samples, n_features = X_train.shape
# Initialize weights and bias
self.w = np.zeros(n_features)
self.b = 0.0
# Ensure labels are -1 or +1
y_signed = np.where(y <= 0, -1, 1)
# Stochastic subgradient descent
for epoch in range(self.epochs):
for i in range(n_samples):
xi, yi = X_train[i], y_signed[i]
margin = yi * (np.dot(self.w, xi) + self.b)
if margin < 1:
# Hinge loss is active: update gradient includes -C*y*x
grad_w = self.w - self.C * yi * xi
grad_b = -self.C * yi
else:
# No loss: only regularizer gradient
grad_w = self.w
grad_b = 0.0
# Update parameters
self.w -= self.lr * grad_w
self.b -= self.lr * grad_b
def decision_function(self, X):
"""
Compute the raw decision score (w·x + b) for samples X.
"""
X = np.array(X, dtype=float)
if self.kernel == 'linear':
X_eval = X
elif self.kernel == 'poly':
X_eval = self._poly_features(X)
elif self.kernel == 'rbf':
X_eval = self._rbf_features(X)
else:
raise ValueError("Unsupported kernel type")
return np.dot(X_eval, self.w) + self.b
def predict(self, X):
"""
Predict class labels {-1, +1} for samples in X.
"""
scores = self.decision_function(X)
return np.where(scores >= 0, 1, -1)
def predict_proba(self, X):
"""
(Approximate) probability estimates for binary classification.
Uses a logistic function on the decision scores.
Returns shape (n_samples, 2) with columns [P(y=-1), P(y=+1)].
"""
scores = self.decision_function(X)
# Sigmoid for positive class
p_pos = 1.0 / (1.0 + np.exp(-scores))
p_neg = 1.0 - p_pos
return np.vstack([p_neg, p_pos]).T
class MultiClassSVM:
def __init__(self, C=1.0, lr=0.001, epochs=1000,
kernel='linear', degree=3, gamma=None, coef0=1):
"""
One-vs-Rest multiclass SVM. Trains one binary SVM per class.
"""
self.C = C
self.lr = lr
self.epochs = epochs
self.kernel = kernel
self.degree = degree
self.gamma = gamma
self.coef0 = coef0
self.classes_ = None
self.models = {}
def fit(self, X, y):
"""
Fit one SVM for each class.
"""
X = np.array(X, dtype=float)
y = np.array(y)
self.classes_ = np.unique(y)
for cls in self.classes_:
# Prepare binary labels: +1 for current class, -1 for all others
y_binary = np.where(y == cls, 1, -1)
svm = SVM(C=self.C, lr=self.lr, epochs=self.epochs,
kernel=self.kernel, degree=self.degree,
gamma=self.gamma, coef0=self.coef0)
svm.fit(X, y_binary)
self.models[cls] = svm
def predict(self, X):
"""
Predict multiclass labels by picking the class with highest decision score.
"""
X = np.array(X, dtype=float)
# Compute decision score for each class model
scores = np.vstack([self.models[cls].decision_function(X) for cls in self.classes_]).T
# Choose class index with max score
idx = np.argmax(scores, axis=1)
return self.classes_[idx]
def predict_proba(self, X):
"""
Return (approximate) class probabilities via softmax on the decision scores.
"""
X = np.array(X, dtype=float)
scores = np.vstack([self.models[cls].decision_function(X) for cls in self.classes_]).T
# Softmax for probabilities
exp_scores = np.exp(scores - np.max(scores, axis=1, keepdims=True))
probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)
return probs
class SVR:
def __init__(self, C=1.0, lr=0.001, epochs=1000, epsilon=0.1,
kernel='linear', degree=3, gamma=None, coef0=1, random_features=100):
self.C = C
self.lr = lr
self.epochs = epochs
self.epsilon = epsilon
self.kernel = kernel
self.degree = degree
self.gamma = gamma
self.coef0 = coef0
self.random_features = random_features
self.w = None
self.b = 0.0
self.W = None
self.b_rff = None
def _poly_features(self, X):
from itertools import combinations_with_replacement
n_samples, n_features = X.shape
features = []
for deg in range(1, self.degree+1):
for combo in combinations_with_replacement(range(n_features), deg):
feat = np.ones(n_samples)
for i in combo:
feat *= X[:, i]
features.append(feat)
if len(features) > 0:
return np.vstack(features).T
else:
return np.zeros((n_samples, 0))
def _rbf_features(self, X):
if self.gamma is None:
self.gamma = 1.0 / X.shape[1]
D = self.random_features
if self.W is None:
self.W = np.sqrt(2 * self.gamma) * np.random.randn(D, X.shape[1])
if self.b_rff is None:
self.b_rff = 2 * np.pi * np.random.rand(D)
return np.sqrt(2.0 / D) * np.cos(np.dot(X, self.W.T) + self.b_rff)
def fit(self, X, y):
"""
Train the SVR model. Uses subgradient descent on the ε-insensitive loss.
"""
X = np.array(X, dtype=float)
y = np.array(y, dtype=float)
# Feature mapping for kernel
if self.kernel == 'linear':
X_train = X.copy()
elif self.kernel == 'poly':
X_train = self._poly_features(X)
elif self.kernel == 'rbf':
X_train = self._rbf_features(X)
else:
raise ValueError("Unsupported kernel type")
n_samples, n_features = X_train.shape
self.w = np.zeros(n_features)
self.b = 0.0
# Training loop
for epoch in range(self.epochs):
for i in range(n_samples):
xi, yi = X_train[i], y[i]
f = np.dot(self.w, xi) + self.b
if yi - f > self.epsilon:
# Prediction too low: push up
grad_w = self.w - self.C * xi
grad_b = -self.C
elif f - yi > self.epsilon:
# Prediction too high: push down
grad_w = self.w + self.C * xi
grad_b = self.C
else:
# Within ε: only regularization gradient
grad_w = self.w
grad_b = 0.0
self.w -= self.lr * grad_w
self.b -= self.lr * grad_b
def predict(self, X):
"""
Predict continuous values for regression.
"""
X = np.array(X, dtype=float)
if self.kernel == 'linear':
X_eval = X
elif self.kernel == 'poly':
X_eval = self._poly_features(X)
elif self.kernel == 'rbf':
X_eval = self._rbf_features(X)
else:
raise ValueError("Unsupported kernel type")
return np.dot(X_eval, self.w) + self.b
# Evaluation metrics and utilities
def accuracy_score(y_true, y_pred):
"""Classification accuracy."""
y_true = np.array(y_true)
y_pred = np.array(y_pred)
return np.mean(y_true == y_pred)
def mean_squared_error(y_true, y_pred):
"""Mean squared error for regression."""
y_true = np.array(y_true)
y_pred = np.array(y_pred)
return np.mean((y_true - y_pred)**2)
def save_predictions_to_csv(y_true, y_pred, filename):
"""Save true vs. predicted values to a CSV file."""
with open(filename, mode='w', newline='') as f:
writer = csv.writer(f)
writer.writerow(['True', 'Predicted'])
for true, pred in zip(y_true, y_pred):
writer.writerow([true, pred])
# Synthetic data generation
def generate_classification_data(binary=True, n_samples=100, n_features=2, n_classes=2, random_state=None):
"""
Generate synthetic classification data:
- Binary: two Gaussian clusters.
- Multiclass: 'n_classes' clusters with increasing means.
"""
rng = np.random.RandomState(random_state)
if binary and n_classes == 2:
mean1 = rng.randn(n_features) - 1
mean2 = rng.randn(n_features) + 1
cov = np.eye(n_features) * 0.2
X1 = rng.multivariate_normal(mean1, cov, size=n_samples//2)
X2 = rng.multivariate_normal(mean2, cov, size=n_samples//2)
X = np.vstack([X1, X2])
y = np.hstack([np.zeros(n_samples//2), np.ones(n_samples//2)])
else:
X = []
y = []
for cls in range(n_classes):
center = rng.randn(n_features) + cls * 2.0
cov = np.eye(n_features) * 0.3
Xc = rng.multivariate_normal(center, cov, size=n_samples//n_classes)
yc = np.full(n_samples//n_classes, cls)
X.append(Xc); y.append(yc)
X = np.vstack(X)
y = np.hstack(y)
return X, y
def generate_regression_data(n_samples=100, n_features=1, noise=0.1, random_state=None):
"""
Generate synthetic regression data: y = sin(x) + noise.
"""
rng = np.random.RandomState(random_state)
X = np.linspace(-5, 5, n_samples).reshape(-1, n_features)
y = np.sin(X).ravel() + noise * rng.randn(n_samples)
return X, y
# --- Binary Classification Demo ---
# Generate binary data
X_bin, y_bin = generate_classification_data(binary=True, n_samples=200, n_features=2, n_classes=2, random_state=1)
y_bin_signed = np.where(y_bin == 0, -1, 1) # Encode labels as -1/+1
svm = SVM(C=1.0, lr=0.01, epochs=500, kernel='linear')
svm.fit(X_bin, y_bin_signed)
pred_bin = svm.predict(X_bin)
acc_bin = accuracy_score(y_bin_signed, pred_bin)
print("Binary SVM accuracy:", acc_bin)
# (Optional) Save predictions
save_predictions_to_csv(y_bin_signed, pred_bin, "binary_svm_results.csv")
# --- Multiclass Classification Demo ---
# Generate 3-class data
X_multi, y_multi = generate_classification_data(binary=False, n_samples=300, n_features=2, n_classes=3, random_state=42)
mc_svm = MultiClassSVM(C=1.0, lr=0.01, epochs=500, kernel='linear')
mc_svm.fit(X_multi, y_multi)
pred_multi = mc_svm.predict(X_multi)
acc_multi = accuracy_score(y_multi, pred_multi)
print("Multiclass SVM accuracy:", acc_multi)
# Show probability estimates for first few samples (softmaxed scores)
probs_multi = mc_svm.predict_proba(X_multi[:5])
print("Class probabilities (first 5):\n", probs_multi)
# --- Regression Demo ---
# Generate regression data
X_reg, y_reg = generate_regression_data(n_samples=100, noise=0.2, random_state=0)
# Linear SVR
svr = SVR(C=1.0, lr=0.01, epochs=1000, epsilon=0.1, kernel='linear')
svr.fit(X_reg, y_reg)
pred_reg = svr.predict(X_reg)
mse = mean_squared_error(y_reg, pred_reg)
print("SVR Mean Squared Error:", mse)