No

Even if you could explain what all of thise words mean, i still think i could only second HR.

But i might ask, what is the difference between a high dimensional matrix and a tensor?

CSteinhardt could. I don't know if he would.

I hope he'll stumble on this thread and decide he's bored enough to spare some time on it...

It sounds like the question is incomplete, anyway. Are you asking why the inverse of a covariance matrix "isn't [unbiased]," or something else?

Your question is rather incomplete, so it needs to be clarified substantially before anyone can help you (if there is anyone to help since this is rather a specific subfield of statistics and econometrics).

I do have some resources that may be helpful but they are rather dense and I suspect way above what you are asking for, but here ya go:

ftp://ftp.aefweb.net/WorkingPapers/w516.pdf

http://www-stat.wharton.upenn.edu/~tcai/paper/Covariance-Survey.pdf

https://arxiv.org/pdf/1105.4292.pdf

Measure of precision vs measure of dispersion. The more precise the more biased.

I think the question is rather complete. There exists a problem with evaluating high-dimensional covariance matrices. First of all, if the number of parameters is greater than the number of observations the matrix is singular (for reason I still am trying to understand, but at least for this I have material). There is however another problem - you sometimes need an inverse of the covariance matrix, for example in choosing optimal asset bundle. However, even though covariance matrix is not biased, its inverse is [biased], and the more dimensional the model, the bigger the bias. So, I would like to find some literature on the subject, because I would like to see the maths behind it.

@Zultar, thank you, this help a bit. The first paper is actually one I have to base my assignment on, the other ones look like they might help me, I'll get down to reading as soon as I get home.

So first, i'm learning here. A matrix is just a collection of numbers, higher dimensions is just more numbers. Covarianve is a relationship between numbers - so if a random selection of house prices going up is related to a random selection of incomes going ip, the covariances (between these two? Variables) would be positive. If the price of rice going up is relate to the price of meat sropping, then the covariance would be negative.

Thus a matrix could have the co-variances between house prices and income as the first element, the house prces and meat prices as the second, and the next row would be income and rice prices, followed by income and meat prives.

Right? (Fyi i believe that if you can't explain a thing, you don't understand it... And you haven't explained the problem very clearly but then you're only asking because you don't understand it)

You are quite right in most of what you wrote. But then it is a bit complicated. A covariance matrix is a matrix of covariances between all the variables in some multi-variate model, let's say the price of some 10 assets. Such matrix would be 10x10 matrix, with element in i-th row and j-th column being the covariange between i-th and j-th asset (Such matrix is symmetrical, because the covariance between i-th and j-th is the same as between j-th and i-th, and on the diagonal it has the variance of i-th asset).

So you have your assets, and you want to see if their prices are covariate. So you look at the prices for 100 days and count the covariance between each asset and put it all into a matrix. But that's not really good enough, because what you observed was just one, 100-day long realisation of a random process that generated the prices. So what you observed is just a sample, and you want to make some conclusions concerning the "population", so the real relations between the prices of the assets. The sample covariance matrix is an unbiased estimator, so whatever matrix you calculated on your sample you have no reason to believe it's actually smaller or larger then the true covariance. However, especially when you are dealing with a high-dimensional problem, so in this case you'd have relatively many assets to be evaluated with respect to number of observations, you run into some problems.

First of all, the covariance estimator has a HUGE variance, so even though you have no reason to believe that the true covariances are smaller or larger than what you calculated, you know that you are probably nowhere near the true covariance matrix.
Secondly, if number of assets is larger than the number of days you observed the pice the covariance matrix is singular, and you won't be able to calculate the inverse (the determinant of the matrix will be 0) - this is for reasons I have to find out yet.
But then if the number of assets is smaller, but still pretty large compared to the number of observations you end up with a highly biased inverse of the covariance matrix. If I understand correctly the numbers in the inverse tend to be larger than the true values.

I might have mixed up some stuff above though, I am just learning about it and I do find it pretty challenging to be honest.

And to be absolutely honest I haven't explained the problem because it is extremely specific, and I thought that whoever would be able to help me would have no problem understanding the question

Though I do realize that perhaps it might have been better to include some explanation.

Much appreciation of the explaination. I think i understood that much better than any wikipedia article (or article from a typical journal/text book)

I'm now wondering about generating this for assets on the market in eve online... I think there are huge amounts of data available. Hmm, could be an interesting little project.

I'm not sure I completely understand your question, but assuming you have a proper covariance function, shouldn't the covariance matrix be positive semi-definite and therefore non-singular, no matter the number of variables or the number of timepoints?

I honestly have such huge gaps in knowledge when it comes to matrices, but isn't it the case that the "semi-ness" in positive semi-definite is what gives the possibility for the matrix to be singular?

I honestly don't know how to explain this any better because I only now start to understand this, but it makes a bit of sense:
If you have the matrix Y of your data, that is p variables over n observation, its rank is at most min(p,n), so it does kind of make sense that when you calculate the covariance matrix its rank would be at most min(p,n). Nevertheless I still need to fully understand why. And since we have a situation where p>n, the covariance matrix is the size pxp, it has the rank of n which is smaller, it is not of full rank, it is singular.

But that isn't even the big problem that I have to solve - the bias of the inverse is what really puzzles me.

O yeeeaaaah, it makes sense once you consider the formula for the convariance matrix where you sum n times the difference between the vector of the i_th observation and the mean vector multiplied by the transpose of said difference. And if you add two matrices the rank of the sum is at most the sum of their ranks, so you end up with a matrix of rank at most n (because you sum n matrixes). Makes sense.

Still struggling with the bias though.

I'm still struggling to understand how to work out the covariance of the ith and jth item.

"First of all, if the number of parameters is greater than the number of observations the matrix is singular (for reason I still am trying to understand, but at least for this I have material). "

a singular matrix is a matrix that is not invertible (as in she has no twin by inversion therefore the name). a matrix is not invertible when she is not full rank. (it's a theorem). all of this only makes sense if we are talking about nxn-matrices. yet all covariance matrices are. such a matrix is not full rank if she does not contain at least n linearly independent column vectors. (another theorem)

now think about how your covariance matrix would be generated. and here it becomes hairy and your question incomplete. I can assume what you mean but it could be something completely different. Eg.: are the covariance matrices a categorical model or a continuous, gaussian or non-gaussian, errors or the actual variables, generated by some specific process or not...

@peter: the semi part is what actually makes this a problem.

@yassem: there is no possibility to invert a non-fullrank matrix. hence no inverse. hence the way you phrased the problem really makes no sense.

Yet there is something else: you can fall back to a a so-called generalised inverse. there exist many different forms of it but they follow the general idea:

by construction we know that C is psd. => it's smallest eigenvalue l >= 0
by construction we know that C is not full rank => card{l =0} >= 1
by construction we know C = SDS^(-1) for some matrix S (eigenvalue decomposition), where D is diagonal and contains all eigenvalues of C
NOW we easily see that we can address the problem of eigenvalues by adding an identity matrix M multiplied a sufficiently large factor a to obtain the positive definite matrix C* = S(D+aM)S^(-1).
Now we can invert the matrix C* and use it for our statistical or other stuff.
But the introduction of the aM-term introduced a bias. This bias works differently on each dimension, depending on the absolute value of the eigenvalues.

As the inverse of the covariance matrix (in this case of the "regularised" C) is called the precision matrix is usually called precision matrix this term is here not the actual inverse of C (which does not exist) but of some biased C* and hence biased as well. I hope that helps to get your mind on the right spot. More I couldn't help without more information (and even this one was only under assumption of plausibility).

Comments on covariance matrix singularity:
http://stats.stackexchange.com/questions/70899/what-correlation-makes-a-matrix-singular-and-what-are-implications-of-singularit