color-color diagram about descqa HOT 4 CLOSED

@nsevilla great, looking forward to your PR.

I think it would be better that this test is a standalone test in DESCQA. For one, as you mentioned, there are potential validation data sets and criteria that we can implement, if at a later time we deem that it is important that the catalogs have certain reasonable color-color distribution. In addition, the DESCQA framework allows running several tests at once, so there's no need to put many tests into one.

from descqa.

@nsevilla : I would say that you're guaranteed a horrible chi-squared. The errors in the distribution should be dominated by counting statistics and N is very large.

Your proposed statistic would also be very sensitive to small shifts in the overall colors of objects in data vs. simulations (as that will shift objects from bin-to-bin) as well as the choice of binning. I would suggest that we would want to do something that is unbinned and that applies shifts to align distributions before comparison. In 1D, that is all well-defined, but that is not true in 2D (e.g. there is not a true 2D analog of the K-S test).

from descqa.

Sounds good to get me started @yymao . It runs under the descqa already in my version at NERSC as I said in the hack day. Will tidy it up and make a PR.

Concerning a possible test (I know it's not required any more), a possibility would be binning the SDSS 2D data, creating some jackknife errors in each bin, and check statistical compatibility... I think it is going to be far from a good chi2 in general, but nonetheless, maybe to check quantitatively the impact of any changes in the catalogs (this process can be a bit computationally expensive though but could be optional).

But, if only plots are required, wouldn't this be better part of the general suite of tests done by readiness_test.py for instance?

from descqa.

closed by #88

from descqa.