Use SQL Lite databases for LocalStatus (instead of flat files)
The aim of this patch is to:
a) reduce memory footprint on large mailboxes
b) reduce CPU usage maintaining LocalStatus
c) scale better with large mailboxes
Amazingly enough it does pretty much do this.
The additional patch (updateflags) commits less often, making something
like "mark as read" 100,000 messages locally complete way faster than
committing on every flag update.
Performance:
- Tested on a single 411MB Maildir with 12,701 messages.
- Filesystems:
- from remote XFS on raid1 (both disks on same IDE controller)
- to local XFS
- both are real file systems and well aged.
- Connection to IMAP server was via 802.11g directly to the imapd. The
only time the network was remotely saturated in bandwidth was when
syncing large mail messages.
- imapd was courier-imap.
- MaxConnections for the remote repository was 3.
- NOTE: total sync time was no different between the unpatched and
patched versions, suggesting that threads for a single mailbox does
nothing to improve performance.
- preauthtunnel was ssh
- ui was TTYUI.
Disclaimer: I am no python programmer... I hack clustered database
internals... newbie python mistakes welcomed to be pointed out.
Unpatched offlineimap took:
For the initial sync (all messages):
real 9m35.872s
user 4m20.196s
sys 0m39.158s
With resident memory going from 22MB to 30MB to 40MB.
With CPU usage climing from ~35% to ~70%
The second sync (doing nothing) took:
real 0m6.846s
user 0m2.940s
sys 0m0.724s
WITH THE PATCH::
----------------
With this patch, offlineimap took:
real 9m26.239s
user 0m43.595s
sys 0m20.113s
Real time saved: 10seconds (insignificant)
CPU time saved: 3min 40seconds user! and 19seconds of system!
For the second (doing nothing) sync:
1st run:
real 0m11.403s
user 0m4.408s
sys 0m1.320s
2nd run:
real 0m6.893s
user 0m4.260s
sys 0m1.392s
So it looks like the null sync may take a bit more real time and a bit
more CPU. IMHO this is an acceptable hit.
With the update flags patch (also included), marking ~13,000 messages as
read in localstatus takes essentially no time.
I also experimented with bulk committing new mail messages to
LocalStatus (every 10 and every 100 messages). It was possible to shave
~30 and ~60 seconds off the initial sync respectively. Although I don't
think the possibility of duplicate messages is really worth it at this
stage. I think there are other (better) ways to improve initial sync
performance than this.
I have been using this patch since I first mailed it back in March - and
am very happy with it (and have had no problems).