View Issue Details

IDProjectCategoryView StatusLast Update
0003294SOGoActiveSyncpublic2015-09-09 14:14
Reporterangeloxx Assigned Toludovic  
PrioritynormalSeverityminorReproducibilitysometimes
Status resolvedResolutionfixed 
Platform[Server] LinuxOSDebianOS Version8 (Jessie)
Product Version2.3.0 
Fixed in Version2.3.2 
Summary0003294: IMAP LIST loop while ActiveSync a large (many folders) mailbox
Description

I'm testing SOGo (packaged 2.3.0, packaged night build and git) with a existing IMAP server with the test mailbox (mine), about 5Gb of mail and 247 folders.

After initial sync, where the client (Windows Mail or Outlook 2013 with EAS configuration) download the folder list and INBOX content, SOGo start a loop where asks complete folder list (this is a socat log):

2015/07/18 11:06:24.310413 length=41 from=64069 to=64109
1410 list "" "*" return (status (x-guid))
< 2015/07/18 11:06:24.541733 length=8192 from=12246478 to=12254669

  • LIST () "." INBOX\r
  • STATUS INBOX (X-GUID 2c092b130ee8cd54df290000334545e2)\r
  • LIST () "." INBOX.Lavoro\r
  • STATUS INBOX.Lavoro (X-GUID a953f238e0b25855832b0000d53c2401)\r
  • LIST () "." INBOX.Lavoro.Ys\r
    [...]

and then asks the status of special folders (Drafts, Trash, Sent) and receives the requested data:

[...previous list...]

1410 OK List completed.\r

2015/07/18 11:06:25.236833 length=40 from=64112 to=64151
1411 status "INBOX.Drafts" (UIDVALIDITY)< 2015/07/18 11:06:25.237392 length=75 from=12280187 to=12280261

  • STATUS INBOX.Drafts (UIDVALIDITY 1262629219)\r
    1411 OK Status completed.\r
    2015/07/18 11:06:25.237598 length=2 from=64152 to=64153
    \r
    2015/07/18 11:06:25.273935 length=46 from=64154 to=64199
    1412 status "INBOX.Sent Items" (UIDVALIDITY)\r
    < 2015/07/18 11:06:25.276236 length=81 from=12280262 to=12280342
  • STATUS "INBOX.Sent Items" (UIDVALIDITY 1262625493)\r
    1412 OK Status completed.\r
    2015/07/18 11:06:25.276789 length=39 from=64200 to=64238
    1413 status "INBOX.Trash" (UIDVALIDITY)< 2015/07/18 11:06:25.277199 length=74 from=12280343 to=12280416
  • STATUS INBOX.Trash (UIDVALIDITY 1411639996)\r
    1413 OK Status completed.\r
    2015/07/18 11:06:25.277497 length=2 from=64239 to=64240
    \r

This loop continues until the process finish memory and clients (VxMemory sets to 512) and the client is disconnected.

As suggested by the ML User Thomas this is a result of the high number of folders and a sub-optimal way to get the x-guid; he suggest me a the attached patch that is working properly.

TagsNo tags attached.

Activities

angeloxx

angeloxx

2015-07-23 20:21

reporter  

0019-cacheImapFolderGUIDS.patch (3,957 bytes)   
From 14d07663f02af26b66bb7952c178b2a909cae3be Mon Sep 17 00:00:00 2001
From: root <root@poldi.hopto.org>
Date: Tue, 21 Jul 2015 20:53:30 +0200
Subject: [PATCH 19/19] cacheImapFolderGUIDS

---
 ActiveSync/SOGoActiveSyncDispatcher+Sync.m |    4 ++--
 ActiveSync/SOGoActiveSyncDispatcher.h      |    1 +
 ActiveSync/SOGoActiveSyncDispatcher.m      |   26 ++++++++++++++++----------
 3 files changed, 19 insertions(+), 12 deletions(-)

diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
index f91d0cb..52c9377 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
@@ -229,7 +229,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   NSString  *nameInCache;
   
   if (theFolderType == ActiveSyncMailFolder)
-    nameInCache= [[[theCollection mailAccountFolder] imapFolderGUIDs] objectForKey: [theCollection nameInContainer]];
+    nameInCache = [imapFolderGUIDS objectForKey: [theCollection nameInContainer]];
   else
     {
       NSString  *component_name;
@@ -240,7 +240,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
       else
         component_name = @"vtodo";
       
-      nameInCache= [NSString stringWithFormat: @"%@/%@", component_name, [theCollection nameInContainer]];
+      nameInCache = [NSString stringWithFormat: @"%@/%@", component_name, [theCollection nameInContainer]];
     }
   
   return nameInCache;
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.h b/ActiveSync/SOGoActiveSyncDispatcher.h
index 47c9c9c..3a62e80 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.h
+++ b/ActiveSync/SOGoActiveSyncDispatcher.h
@@ -37,6 +37,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 @interface SOGoActiveSyncDispatcher : NSObject
 {
   NSURL *folderTableURL;
+  NSDictionary *imapFolderGUIDS;
   id context;
   BOOL debugOn;
 }
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m
index 40b1b2d..4fc97d8 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher.m
@@ -151,12 +151,14 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
   debugOn = [[SOGoSystemDefaults sharedSystemDefaults] easDebugEnabled];
   folderTableURL = nil;
+  imapFolderGUIDS = nil;
   return self;
 }
 
 - (void) dealloc
 {
   RELEASE(folderTableURL);
+  RELEASE(imapFolderGUIDS);
   [super dealloc];
 }
 
@@ -229,23 +231,27 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
       SOGoMailAccounts *accountsFolder;
       SOGoMailAccount *accountFolder;
       SOGoUserFolder *userFolder;
-      NSDictionary *imapGUIDs;
 
-      userFolder = [[context activeUser] homeFolderInContext: context];
-      accountsFolder = [userFolder lookupName: @"Mail" inContext: context acquire: NO];
-      accountFolder = [accountsFolder lookupName: @"0" inContext: context acquire: NO];
-      
-      // Get the GUID of the IMAP folder
-      imapGUIDs = [accountFolder imapFolderGUIDs];
-      
-      //return [[imapGUIDs allKeysForObject: theIdToTranslate] objectAtIndex: 0];
-      return [[[imapGUIDs allKeysForObject:  [NSString stringWithFormat: @"folder%@", theIdToTranslate]] objectAtIndex: 0] substringFromIndex: 6] ;
+      if (!imapFolderGUIDS)
+        {
+          userFolder = [[context activeUser] homeFolderInContext: context];
+          accountsFolder = [userFolder lookupName: @"Mail" inContext: context acquire: NO];
+          accountFolder = [accountsFolder lookupName: @"0" inContext: context acquire: NO];
+
+          // Get the GUID of the IMAP folder
+          imapFolderGUIDS = [accountFolder imapFolderGUIDs];
+          [imapFolderGUIDS retain];
+
+        }
+
+        return [[[imapFolderGUIDS allKeysForObject:  [NSString stringWithFormat: @"folder%@", theIdToTranslate]] objectAtIndex: 0] substringFromIndex: 6] ;
     }
   
   return theIdToTranslate;
 }
 
 
+
 //
 //
 //
-- 
1.7.10.4

angeloxx

angeloxx

2015-07-27 21:01

reporter   ~0008784

After three days of test (with my existent IMAP mailbox only, 247 folders, 190k messages in 5.2Gb) with OL2013 and Nine for Android I can confirm that the patch works without problems.

Before the patch the Sync process loops with LIST command (about 1010 iterations in 7 minutes) and ends only when VxLimit is reached. Now the same process takes less than 30 seconds.

Notes:

  • No problems deleting or moving mail (after solving the 3293 ticket too)
  • I created also a new folder on IMAP side and OL see that folder after a restart
  • I deleted the new folder using OL but IMAP client (and Filesystem) says that the folder still exists, but I don't know which is the correct behavior in this case
ludovic

ludovic

2015-09-09 14:14

administrator   ~0008897

Patch pushed: https://github.com/inverse-inc/sogo/commit/7c0ba7b72b1ae24e0135e6053de0f50ba1532729

Issue History

Date Modified Username Field Change
2015-07-23 20:21 angeloxx New Issue
2015-07-23 20:21 angeloxx File Added: 0019-cacheImapFolderGUIDS.patch
2015-07-27 21:01 angeloxx Note Added: 0008784
2015-09-09 14:14 ludovic Note Added: 0008897
2015-09-09 14:14 ludovic Status new => resolved
2015-09-09 14:14 ludovic Fixed in Version => 2.3.2
2015-09-09 14:14 ludovic Resolution open => fixed
2015-09-09 14:14 ludovic Assigned To => ludovic