From: John Goerzen Date: Tue, 23 Aug 2005 07:15:09 +0000 (+0100) Subject: [319117] Unroll loop to speed performance on large folders X-Git-Tag: DEBIAN_offlineimap_4.0.11~5 X-Git-Url: https://code.delx.au/offlineimap/commitdiff_plain/0841e03a4c498b12568f859596a0b83b4b1bb6ef [319117] Unroll loop to speed performance on large folders From: "Nikita V. Youshchenko" I trued to use offlineimap and found that while being quite fast on small folders, it takes up to several minutes (of 100% busy CPU and almost no network traffic) to sync a folder with 2000+ messages. While looking into the code, I found why this happens. In folder/Base.py, in method BaseFolder.syncmessagesto_copy(), dest.getmessagelist() is called inside a loop, while being a loop invariant. Similar thing happens in BaseFolder.syncmessagesto_delete() for self.getmessagelist(). This causes quadratic complexity over folder size. Moving these calls out of loops make large folder sync fast (several seconds instead of several minutes for folder with 2000 messages on 700MHz P3). --- diff --git a/offlineimap/folder/Base.py b/offlineimap/folder/Base.py index 8d1bbae..50d9ccb 100644 --- a/offlineimap/folder/Base.py +++ b/offlineimap/folder/Base.py @@ -288,10 +288,11 @@ class BaseFolder: them to dest.""" threads = [] + dest_messagelist = dest.getmessagelist() for uid in self.getmessagelist().keys(): if uid < 0: # Ignore messages that pass 1 missed. continue - if not uid in dest.getmessagelist(): + if not uid in dest_messagelist: if self.suggeststhreads(): self.waitforthread() thread = InstanceLimitedThread(\ @@ -314,10 +315,11 @@ class BaseFolder: Look for message present in dest but not in self. If any, delete them.""" deletelist = [] + self_messagelist = self.getmessagelist() for uid in dest.getmessagelist().keys(): if uid < 0: continue - if not uid in self.getmessagelist(): + if not uid in self_messagelist: deletelist.append(uid) if len(deletelist): UIBase.getglobalui().deletingmessages(deletelist, applyto)