No R packages are permitted for use in this assignment.

  1. In the previous assignment you were asked to provide pseudocode for how to generate synthetic data from the statistical model posited for your project. The motivation was described as: synthetic data will be helpful for evaluating an estimation procedure and algorithm. Describe how you will design a simulation study based on your synthetic data to evaluate the estimation procedure and algorithm that you proposed a few weeks ago.
  2. In the simple linear regression model \(y_i = \beta_0 + x_i\beta_1 + u_i\) for \(i \in \{1, \ldots, n\}\), show that \(\beta_0\) is estimable by finding a vector \(a\) and scalar \(c\) such that \(E(c + a'y) = \beta_0\).
  3. Prove that if \(\lambda^{(1)'}\beta, \ldots, \lambda^{(k)'}\beta\) are estimable, then so is
    \[ \sum_{j=1}^{k} d_j \lambda^{(j)'}\beta, \]
    for any scalar constants \(d_1, \ldots, d_k\).
  4. Show that if the least squares estimator \(\lambda'\hat{\beta}\) is the same for all solutions \(\hat{\beta}\) to the normal equations, then \(\lambda'\beta\) is estimable.
  5. Consider the model \(Y_{ij} = \mu + \alpha_i + \beta_i x_{ij} + U_{ij}\), for \(i \in \{1, \ldots, n\}\) and \(j \in \{1 \ldots, m\}\). Further, assume that \(\sum_{j=1}^{m}(x_{ij} - \bar{x}_{i\cdot})^2 > 0\) for all \(i \in \{1, \ldots, n\}\). Derive the necessary and sufficient conditions for an estimable function \(\lambda'\gamma\), where \(\gamma := (\mu, \alpha_1, \ldots, \alpha_n, \beta_1, \ldots, \beta_n)'\).
  6. The problem of least squares regression can be understood as a special case of the more general problem of ridge regression. For an \(n\)-dimensional column vector \(y\) and an \(n \times p\) design matrix \(X\), the problem of ridge regression is to solve for the parameter vector \(b\) that minimizes
    \[ a\|b\|_2^2 + \|y - Xb\|_2^2, \]
    where \(a \geq 0\) is fixed.
    1. Derive a closed-form expression of the ridge regression solution.
    2. Assume that \(X\) has full column rank, and suppose that \(y\) is an observed instance of the random vector \(Y = X\beta + U\), where \(\beta \in \mathbb{R}^p\) is fixed and \(U\) satisfies the Gauss-Markov assumptions. Under what condition(s) is the ridge regression solution the best linear unbiased estimator (BLUE) for any \(\beta\)?