Unique items only (no duplicates) about persist-queue HOT 11 CLOSED

Make sense to create a set like queue for your use case.

How is your data? Is it hashable?

from persist-queue.

from persist-queue.

Did you use the file based queue or sqlite based queue?

On my first thought, storing the hash value in sqlite as a column is possible, we can check the uniqueness via SQL, but for filed based queue, no idea for now.

Do you have any idea?

from persist-queue.

from persist-queue.

from persist-queue.

That's a good news.

A set like queue can be useful in many cases. I do support adding this:

• it can be a new queue, say it's SetQueue, we can optimize it on demand
• either, doing as you mentioned.

Although, I prefer the first one, the second option is also welcome.
Do you mind sending out the PR for either solution?

Thanks
Peter

from persist-queue.

This works for me but can't fully test right now.

class UniqueQ(persistqueue.SQLiteQueue):
    _TABLE_NAME = 'unique_queue'
    _SQL_CREATE = ('CREATE TABLE IF NOT EXISTS {table_name} ('
                   '{key_column} INTEGER PRIMARY KEY AUTOINCREMENT, '
                   'data BLOB, timestamp FLOAT, UNIQUE (data))')

    def put(self, item):
      obj = pickle.dumps(item)
      try:
        self._insert_into(obj, _time.time())
      except sqlite3.IntegrityError:
        #print("UNIQUE constraint failed")
        pass
      else:
        self.total += 1
        self.put_event.set()

from persist-queue.

Looks fine for me, but need to add some test to cover this.

To create PR, please follow below steps:

1. fork this project.
2. clone your forked project to local.
3. create a new branch, say unique_queue, then begin to make changes and commit.
4. git push origin unique_queue.
5. then your can create the PR based your branch by clicking the button New pull request on this page.

from persist-queue.

@delica1 Do you meet any obstacle for the patch:)

from persist-queue.

from persist-queue.

Fixed in PR: #41

from persist-queue.