John Goerzen [Mon, 3 Mar 2008 14:22:44 +0000 (08:22 -0600)]
Fix performance for SSL
Added WrappedIMAP4_SSL class to help fix up performance of SSL
Standard imaplib.py is really bad with this, since it reads one
character at a time.
Reported by Aaron Kaplan at
http://lists.complete.org/offlineimap@complete.org/2008/01/msg00012.html.gz
He wrote:
I just noticed that the version of offlineimap I've been using
(3.99.17) is well over four years old. How time flies. I haven't
had any problems with it, but out of curiosity I decided to pull in
5.99.2 from the fedora repository. It turns out to take
consistently over twice as long as the old version to sync the same
account. Is this expected?
He tracked it down at
http://lists.complete.org/offlineimap@complete.org/2008/02/msg00012.html.gz
The following changeset is the one responsible for the difference in
speed I was noticing between the imaplib.py that was packaged with
older versions of offlineimap and the one that comes with python:
* /offlineimap/head: changeset 169
More optimizations -- this time fix readline() to not work
character-by-character!
John Goerzen [Mon, 3 Mar 2008 04:25:05 +0000 (22:25 -0600)]
Fix handling of servers that return UIDs in some FETCH responses
closes #22
from pistore in OfflineIMAP #22:
When an IMAP flag update is performed for multiple messages, some IMAP
servers (e.g. Exchange) return the UID attribute only for some of the
FETCH untagged responses, as shown in the following log:
Function IMAPFolder.processmessagesflags is able to manage the servers
which return the UID and the servers which do not return it, but is
not able to deal with the mixed behavior shown above.
The problem is that the fragment of function
IMAPFolder.processmessagesflags that handles the responses with UID
attribute uses variable flags to store the list of flags of the
message in the IMAP format ("flags = attributehashFLAGS?"), while the
fragment that handles the responses without UID expects variable
"flags" to contain the list of modified flags passed to the function
in Maildir format ("self.messagelist[uid]flags?.append(flag)").
As a consequence, the wrong list of flags is used for the messages
without UID, leading to the addition of "strange" flags to the Maildir
messages:
Syncing messages IMAP[INBOX] -> Maildir[.]
Adding flags to 4 messages on Maildir[.]
Adding flags e to 4 messages on Maildir[.]
Adding flags d to 4 messages on Maildir[.]
Adding flags ) to 4 messages on Maildir[.]
Adding flags ( to 4 messages on Maildir[.]
Adding flags l to 4 messages on Maildir[.]
Adding flags n to 4 messages on Maildir[.]
Adding flags t to 4 messages on Maildir[.]
Adding flags \ to 4 messages on Maildir[.]
Adding flags D to 4 messages on Maildir[.]
Deleting flags T to 4 messages on Maildir[.]
Adding flags to 4 messages on LocalStatus[.]
Adding flags e to 4 messages on LocalStatus[.]
Adding flags d to 4 messages on LocalStatus[.]
Adding flags ) to 4 messages on LocalStatus[.]
Adding flags ( to 4 messages on LocalStatus[.]
Adding flags l to 4 messages on LocalStatus[.]
Adding flags n to 4 messages on LocalStatus[.]
Adding flags t to 4 messages on LocalStatus[.]
Adding flags \ to 4 messages on LocalStatus[.]
Adding flags D to 4 messages on LocalStatus[.]
Deleting flags T to 4 messages on LocalStatus[.]
Fix: use a different variable to store IMAP flags when managing
messages corresponding to responses with UID attribute, e.g.:
*** IMAP.py.orig Wed Aug 22 18:23:17 2007
--- IMAP.py Wed Aug 22 18:22:38 2007
*************** class IMAPFolder(BaseFolder):
*** 340,348 ****
if not ('UID' in attributehash and 'FLAGS' in
attributehash):
# Compensate for servers that don't return a UID
attribute.
continue
! flags = attributehash['FLAGS']
uid = long(attributehash['UID'])
! self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(flags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not
in the list
--- 340,348 ----
if not ('UID' in attributehash and 'FLAGS' in
attributehash):
# Compensate for servers that don't return a UID
attribute.
continue
! lflags = attributehash['FLAGS']
uid = long(attributehash['UID'])
! self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(lflags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not
in the list
02/03/08 14:04:35 changed by js
* attachment flags-fix.patch added.
Delete 02/03/08 14:05:24 changed by js ΒΆ
Unfortunately I have to fetch some of my mail from an Exchange server
(Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1)
and I can confirm that the analysis of the problem is correct, and the
patch given here fixes the problem.
Looking at the code of the processmessagesflags() method I think it
generally is a bug that the "flags" parameter is reused as a local
variable, since the final "for uid in needupdate:" loop needs the
original value of "flags". This only worked by accident.
I'm attaching a unidiff version of the patch which applies cleanly
against Debian unstable's offlineimap 5.99.4.
Riccardo Murri [Thu, 3 Jan 2008 03:56:55 +0000 (04:56 +0100)]
Add Gmail IMAP special support.
New repository/folder classes to support "real deletion" of messages
thorugh Gmail's IMAP interface: to really delete a message in Gmail,
one has to move it to the Trash folder, rather than EXPUNGE it.
fix behaviour for delete/expunge, when lacking rights
This patch maneuvers around python imaplib's mysterious read-only detection
algorithm and correctly calls the UI's deletetoreadonly(), when trying to
delete/expunge in a mailbox without having the necessary rights.
Vincent Beffara [Sun, 2 Sep 2007 00:43:15 +0000 (01:43 +0100)]
UNDO: Synchronize newly created folders both ways
This involves several changes at different places:
- syncfoldersto() takes statusfolder as an argument, and returns the
list of new folders and the list of folders that should be ingnored,
typically those that were deleted. Warns the user about folders that
are present only on one side and are not synced.
- syncfoldersto() is called both ways, and on folder creation
forgetfolders() is used to rebuild the list and take the new creation
into account. Probably not the most efficient, since it involves
talking to the IMAP server again, but it will rarely be used anyway.
- Locally created folders are treated separately in the synchronization,
namely the local messages are uploaded and then the normal sync still
occurs. If the same folder is created on both sides and contains
messages on both sides, a two-way sync occurs.
John Goerzen [Fri, 19 Oct 2007 00:06:18 +0000 (01:06 +0100)]
Fix Maildir race
fixes deb#439384
From: martin f krafft
Subject: race condition in Maildir writing
The offlineimap Maildir code checks for file existence and then
opens a file. That's open to a race condition. It's better to open
the file and fail if it already exists. The following patch does
this. It catches OSError 17 (file exists) and re-raises all others.
I'll leave it up to you to decide whether this is appropriate.
Vincent Beffara [Sun, 2 Sep 2007 00:43:15 +0000 (01:43 +0100)]
Synchronize newly created folders both ways
This involves several changes at different places:
- syncfoldersto() takes statusfolder as an argument, and returns the
list of new folders and the list of folders that should be ingnored,
typically those that were deleted. Warns the user about folders that
are present only on one side and are not synced.
- syncfoldersto() is called both ways, and on folder creation
forgetfolders() is used to rebuild the list and take the new creation
into account. Probably not the most efficient, since it involves
talking to the IMAP server again, but it will rarely be used anyway.
- Locally created folders are treated separately in the synchronization,
namely the local messages are uploaded and then the normal sync still
occurs. If the same folder is created on both sides and contains
messages on both sides, a two-way sync occurs.
John Goerzen [Fri, 19 Oct 2007 00:00:12 +0000 (01:00 +0100)]
Documented remotehosteval
From:
Ben Kibbey
> > Attached is a patch to enable evaluation of account credentials with the
> > remotehosteval, remoteusereval and remotepasseval configuration options.
> > I needed this because rather than change all my other programs
> > configuration settings when I change, say a password, I store them in a
> > file. So I call a function in pythonfile which parses the credential
> > file and returns the wanted info. Not really very well tested, but not
> > complex either. Offlineimap is great, thanks.
[ this is the doc for that patch which was already applied ]
the locked() method isn't implemented for non-interactive UIs, so
exceptions are thrown on cron jobs. Ubuntu's new apport catches these
and ? well, you get the idea.
John Goerzen [Mon, 1 Oct 2007 21:20:37 +0000 (22:20 +0100)]
Daniel Jacobowitz patches
fixes deb#433732
Date: Sun, 30 Sep 2007 13:54:56 -0400
From: Daniel Jacobowitz <drow@false.org>
To: offlineimap@complete.org
Subject: Assorted patches
Here's the result of a lazy Sunday hacking on offlineimap. Sorry for
not breaking this into multiple patches. They're mostly logically
independent so just ask if that would make a difference.
First, a new -q (quick) option. The quick option means to only update
folders that seem to have had significant changes. For Maildir, any
change to any message UID or flags is significant, because checking
the flags doesn't add a significant cost. For IMAP, only a change to
the total number of messages or a change in the UID of the most recent
message is significant. This should catch everything except for
flags changes.
The difference in bandwidth is astonishing: a quick sync takes 80K
instead of 5.3MB, and 28 seconds instead of 90.
There's a configuration variable that lets you say every tenth sync
should update flags, but let all the intervening ones be lighter.
Second, a fix to the UID validity problems many people have been
reporting with Courier. As discussed in Debian bug #433732, I changed
the UID validity check to use SELECT unless the server complains that
the folder is read-only. This avoids the Courier bug (see the Debian
log for more details). This won't fix existing validity errors, you
need to remove the local status and validity files by hand and resync.
Third, some speedups in Maildir checking. It's still pretty slow
due to a combination of poor performance in os.listdir (never reads
more than 4K of directory entries at a time) and some semaphore that
leads to lots of futex wake operations, but at least this saves
20% or so of the CPU time running offlineimap on a single folder:
Time with quick refresh and md5 in loop: 4.75s user 0.46s system 12%
cpu 41.751 total
Time with quick refresh and md5 out of loop: 4.38s user 0.50s system
14% cpu 34.799 total
Time using string compare to check folder: 4.11s user 0.47s system 13%
cpu 34.788 total
And fourth, some display fixes for Curses.Blinkenlights. I made
warnings more visible, made the new quick sync message cyan, and
made all not explicitly colored messages grey. That last one was
really bugging me. Any time OfflineIMAP printed a warning in
this UI, it had even odds of coming out black on black!
Anyway, I hope these are useful. I'm happy to revise them if you see
a problem.
John Goerzen [Wed, 1 Aug 2007 01:25:05 +0000 (02:25 +0100)]
Additional date validity check
patch from Mike Gerber
Two times today I have found my offlineimap to have died with this same
situation. It appears as if certain types of messages (both spam in my
situation), cause offlineimap to choke. When it does it cannot proceed.
This means that when I run offlineimap, it pulls in messages from some
folders, then it hits the folder with the bad message and dies, leaving
undownloaded mail on the server. The only fix to this problem is to find
the problem message on the server and remove it by hand. This isn't such
a huge deal for me, since I run the server, but other people have to
come to me to ask me to delete these messages, and until I do they
cannot download their email.
I have captured the output by running script during one of these
incidents, this has been attached. Additionally, I have also attach the
problematic message.
The patch seems to work for me, might need some Python wizard and better
testing, though.
IMAP.py: fixed cannot concatenate 'str' and 'list' in assert
r[1] is a list. In case it contains more than one 'str' they are concatenated
using '. '. In processmessagesflags the join is tested, for savemessagesflags I
assume that it is also needed.
David Favro [Thu, 15 Mar 2007 04:41:43 +0000 (05:41 +0100)]
UID validity diagnostics improvement
* Reduced the number of parameters passed to ui.validityproblem() because they were all just method-calls to the folder object, which is already passed as the first parameter (reduction of unnecessary complexity).
* Improved the diagnostic message for an 'UID validity problem' by including the name of the repository in which the folder resides; previously it was not possible to determine from the diagnostic alone on which side the problem was.
David Favro [Thu, 15 Mar 2007 04:39:15 +0000 (05:39 +0100)]
UID validity diagnostics improvement
* Reduced the number of parameters passed to ui.validityproblem() because they were all just method-calls to the folder object, which is already passed as the first parameter (reduction of unnecessary complexity).
John Goerzen [Wed, 14 Mar 2007 01:54:19 +0000 (02:54 +0100)]
Don't leave preauthtunnel zombies with autorefresh
From: Peter Colberg
Hello,
using offlineimap with the preauthtunnel option to start a remote
IMAP daemon via ssh, a zombie process is left behind during the
autorefresh sleep period if not holding the connection open.
Above behaviour is a result of using os.popen2 to spawn the tunnel
process, which makes it impossible waiting for the child process
to terminate when shutting down the tunnel.
The patch included below fixes the issue by employing the Popen
class from the subprocess module, which seems to be the preferred
way to spawn processes and connect to their pipes in any case (at
least since python version 2.4.4, which fixes a memory leak in the
subprocess module).
John Goerzen [Thu, 8 Mar 2007 01:59:32 +0000 (02:59 +0100)]
Check all resolved addresses [deb #413030]
From: Mark Brown <broonie@sirena.org.uk>
Currently offlineimap will attempt to connect to the first address
returned by addrinfo() for the remote system and will fail if that fails
even if another result would have worked. This is particularly common
when the remote system supports both IPv4 and IPv6 - a laptop may in
some environments have no routable IPv6 connectivity so if the IPv6
address is returned first the connect will fail even though IPv4 would
have worked.
This is actually a bug in imaplib, a copy of which is included in
offlineimap. This patch fixes the problem by looping over all the
results returned by getaddrinfo(). Unfortunately it mangles the error
reporting slightly since I couldn't work out how to raise an appropriate
exception, though given that that that was a Python backtrace there was
work to do there anyway.