Posted By

magicrebirth on 04/07/11


Tagged

list python remove duplicates


Versions (?)

Remove duplicates from a list


 / Published in: Django
 

URL: http://www.peterbe.com/plog/uniqifiers-benchmark

Not only is it really really fast; it's also order preserving and supports an optional transform function

  1. def remove_duplicates(seq, idfun=None):
  2. # order preserving
  3. if idfun is None:
  4. def idfun(x): return x
  5. seen = {}
  6. result = []
  7. for item in seq:
  8. marker = idfun(item)
  9. # in old Python versions:
  10. # if seen.has_key(marker)
  11. # but in new ones:
  12. if marker in seen: continue
  13. seen[marker] = 1
  14. result.append(item)
  15. return result
  16.  
  17.  
  18.  
  19. >>> a=list('ABeeE')
  20. >>> f5(a)
  21. ['A','B','e','E']
  22. >>> f5(a, lambda x: x.lower())
  23. ['A','B','e']

Report this snippet  

You need to login to post a comment.