Issue with deduplication alogrithm in Thanos about thanos HOT 4 OPEN

The query in the UI uses [1m] ~ from the sample it feels like you have 30s scrape frequency. Does it also happen with [5m]? I wrote a qucik test with your given inputs and the result series looks somewhat like:

					samples: []sample{
						{t: 1713668216000, f: 9389.87},
						{t: 1713668224000, f: 9389.94},
						{t: 1713668254000, f: 9389.95},
						{t: 1713668284000, f: 9390.02},
						{t: 1713668314000, f: 9390.03},
						{t: 1713668344000, f: 9390.33},
						{t: 1713668374000, f: 9390.83},
						{t: 1713668404000, f: 9391.13},
						{t: 1713668434000, f: 9391.38},
						{t: 1713668464000, f: 9391.61},
						{t: 1713668494000, f: 9393.53}},
                                         },

from thanos.

It looks like all samples are there and do have proper 30s scrape interval between them; it could be that your 1m windows are aligned in a way that only one sample is contained in the window which would break rate. I think this is an issue with too small window, but the deduplication result looks somewhat correct to me except that we have one sample too much at the beginning

from thanos.

We have evidence about dedup logic bug as well, here are the proof:

First graph is data points missing when getting the results from thanos receiver with replicationFactor == 3, and because we are rolling update the receiver pods, so 1 copy was absent for sure, however after check Deduplication, it still have dips:

No Dedup

With Dedup

After the data got compact and returned from store, the results become correct with no dips:

from thanos.

seems related to this issue: #981

from thanos.