View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0003293 | SOGo | SOPE | public | 2015-07-23 18:59 | 2015-09-09 14:23 |
Reporter | angeloxx | Assigned To | ludovic | ||
Priority | normal | Severity | crash | Reproducibility | always |
Status | resolved | Resolution | fixed | ||
Platform | [Server] Linux | OS | Debian | OS Version | 8 (Jessie) |
Product Version | 2.3.0 | ||||
Fixed in Version | 2.3.2 | ||||
Summary | 0003293: SIGINT interrupt during ActiveSync synchronization and memory leak | ||||
Description | In my installation with sogo-activesync from repo (2.3.0+) or git during the first sync with OL2013 the sogo process crashes with sigint message and filling all available memory. After recompile sogo and sope to obtain debug simbols i can tell that:
and gdb before the hang: So I solved with this:
I hope that this can help. | ||||
Tags | No tags attached. | ||||
The proposed patch probably is not correct, because the procedure that depends on parseVanishedResponseIntoHashMap sometime logs: Jul 24 08:00:02 sogod [6325]: <0x0x555555f077b0[NGImap4Client]> ERROR(-[NGImap4Client _processUnknownCommandParserException:]): catched non-IMAP4 parsing exception NSInvalidArgumentException: NSNumber(instance) does not recognize numberWithUnsignedInt: so I'm looking why this procedure (parseVanishedResponseIntoHashMap) try to allocate all available memory and why this problem is not recurrent in previous tests and with IMAP log I see that: [...] So I think that if the UID FETCH with CHANGEDSINCE fails bacause ENABLE QRESYNC was not sent, sogo works without problems (but probably the sync miss something), if this fetch returns values the parseVanishedResponseIntoHashMap have problems. Now I'm trying to print max and count values to know if the problem is the single NSNumber allocation or a loop of thousands elements. |
|
The problem is not the allocation. I've tryed to track the max and count value: --- a/sope-mime/NGImap4/NGImap4ResponseParser.m
and sometime happend: S[0x555555efb110]: 273 FETCH (UID 2273345937 MODSEQ (9809) FLAGS (\Seen NonJunk)) |
|
The problem starts from the IMAP server's answer and not the SOPE code. UID FETCH's answer (that SOGO does not display) is:
So... the external loop of _parseVanishedResponseIntoHashMap split the answer by "," and after the first range 1-8474 start with the new range: Jul 24 09:50:44 sogod [12094]: <0x0x555555ef5d10[NGImap4ResponseParser]> ANGELOXX: max:8474 count:8470 probably the problem is on IMAP server side? or SOGO can handle this answer correctly? |
|
The cause of the high memory usage is that there is a very high range of vanished mails reported: 247 UID FETCH 1:* (UID) (CHANGEDSINCE 1 VANISHED)\r
The attached patch ignores deleted and vanished mails during initial load. |
|
0001-3293-2.patch (12,606 bytes)
From 172f4fcc803e97c860f151ed8fa6b4aed6976761 Mon Sep 17 00:00:00 2001 From: root <root@poldi.hopto.org> Date: Sun, 26 Jul 2015 12:34:12 +0200 Subject: [PATCH] 3293-2 --- ActiveSync/SOGoActiveSyncDispatcher+Sync.m | 52 ++++++++++++++++++++++++---- ActiveSync/SOGoActiveSyncDispatcher.m | 5 ++- SoObjects/Mailer/SOGoMailFolder.h | 3 +- SoObjects/Mailer/SOGoMailFolder.m | 6 +++- SoObjects/SOGo/SOGoGCSFolder.h | 4 +-- SoObjects/SOGo/SOGoGCSFolder.m | 23 +++++++----- 6 files changed, 73 insertions(+), 20 deletions(-) diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m index 950ee47..47873c3 100644 --- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m +++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m @@ -133,6 +133,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. [[o properties] removeObjectForKey: @"MoreAvailable"]; [[o properties] removeObjectForKey: @"BodyPreferenceType"]; [[o properties] removeObjectForKey: @"SuccessfulMoveItemsOps"]; + [[o properties] removeObjectForKey: @"InitialLoadSequence"]; [[o properties] addEntriesFromDictionary: values]; [o save]; @@ -673,7 +674,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. NSDictionary *component; NSArray *allComponents; - BOOL updated; + BOOL updated, initialLoadInProgress; int deleted, return_count; if (theFolderType == ActiveSyncContactFolder) @@ -683,7 +684,20 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. else component_name = @"vtodo"; - allComponents = [theCollection syncTokenFieldsWithProperties: nil matchingSyncToken: theSyncKey fromDate: theFilterType]; + initialLoadInProgress = NO; + + if ([theSyncKey isEqualToString: @"-1"]) + [folderMetadata setObject: davCollectionTagToStore forKey: @"InitialLoadSequence"]; + + if ([folderMetadata objectForKey: @"InitialLoadSequence"]) + { + if ([theSyncKey intValue] < [[folderMetadata objectForKey: @"InitialLoadSequence"] intValue]) + initialLoadInProgress = YES; + else + [folderMetadata removeObjectForKey: @"InitialLoadSequence"]; + } + + allComponents = [theCollection syncTokenFieldsWithProperties: nil matchingSyncToken: theSyncKey fromDate: theFilterType initialLoad: initialLoadInProgress]; allComponents = [allComponents sortedArrayUsingDescriptors: [NSArray arrayWithObjects: [[NSSortDescriptor alloc] initWithKey: @"c_lastmodified" ascending:YES], nil]]; @@ -847,12 +861,36 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. NSMutableArray *allCacheObjects, *sortedBySequence; SOGoMailObject *mailObject; - NSArray *allMessages; + NSArray *allMessages, *a; + + int j, k, return_count, uidnext, highestmodseq; + BOOL found_in_cache, initialLoadInProgress; + + initialLoadInProgress = NO; - int j, k, return_count; - BOOL found_in_cache; + if ([theSyncKey isEqualToString: @"-1"]) + { + uidnext = highestmodseq = 0; + + a = [[theCollection davCollectionTag] componentsSeparatedByString: @"-"]; + [folderMetadata setObject: [a objectAtIndex: 1] forKey: @"InitialLoadSequence"]; + } + else + { + a = [theSyncKey componentsSeparatedByString: @"-"]; + uidnext = [[a objectAtIndex: 0] intValue]; + highestmodseq = [[a objectAtIndex: 1] intValue]; + } + + if ([folderMetadata objectForKey: @"InitialLoadSequence"]) + { + if (highestmodseq < [[folderMetadata objectForKey: @"InitialLoadSequence"] intValue]) + initialLoadInProgress = YES; + else + [folderMetadata removeObjectForKey: @"InitialLoadSequence"]; + } - allMessages = [theCollection syncTokenFieldsWithProperties: nil matchingSyncToken: theSyncKey fromDate: theFilterType]; + allMessages = [theCollection syncTokenFieldsWithProperties: nil matchingSyncToken: theSyncKey fromDate: theFilterType initialLoad: initialLoadInProgress]; max = [allMessages count]; allCacheObjects = [NSMutableArray array]; @@ -911,7 +949,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. NSString *lastSequence; more_available = YES; - lastSequence = ([[aCacheObject sequence] isEqual: [NSNull null]] ? @"1" : [aCacheObject sequence]); + lastSequence = ([[aCacheObject sequence] isEqual: [NSNull null]] ? [NSString stringWithFormat:@"%d", highestmodseq] : [aCacheObject sequence]); *theLastServerKey = [[NSString alloc] initWithFormat: @"%@-%@", [aCacheObject uid], lastSequence]; //NSLog(@"Reached windowSize - lastUID will be: %@", *theLastServerKey); DESTROY(pool); diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m index 9be2ebe..0f82e71 100644 --- a/ActiveSync/SOGoActiveSyncDispatcher.m +++ b/ActiveSync/SOGoActiveSyncDispatcher.m @@ -938,6 +938,7 @@ static BOOL debugOn = NO; [[o properties] removeObjectForKey: @"MoreAvailable"]; [[o properties] removeObjectForKey: @"BodyPreferenceType"]; [[o properties] removeObjectForKey: @"SuccessfulMoveItemsOps"]; + [[o properties] removeObjectForKey: @"InitialLoadSequence"]; [o save]; command_count++; @@ -1023,6 +1024,7 @@ static BOOL debugOn = NO; [[o properties] removeObjectForKey: @"MoreAvailable"]; [[o properties] removeObjectForKey: @"BodyPreferenceType"]; [[o properties] removeObjectForKey: @"SuccessfulMoveItemsOps"]; + [[o properties] removeObjectForKey: @"InitialLoadSequence"]; } [o save]; @@ -1046,6 +1048,7 @@ static BOOL debugOn = NO; [[o properties] removeObjectForKey: @"MoreAvailable"]; [[o properties] removeObjectForKey: @"BodyPreferenceType"]; [[o properties] removeObjectForKey: @"SuccessfulMoveItemsOps"]; + [[o properties] removeObjectForKey: @"InitialLoadSequence"]; } [o save]; @@ -1204,7 +1207,7 @@ static BOOL debugOn = NO; filter = [NSCalendarDate dateFromFilterType: [[(id)[[allCollections objectAtIndex: j] getElementsByTagName: @"FilterType"] lastObject] textValue]]; syncKey = [[(id)[[allCollections objectAtIndex: j] getElementsByTagName: @"SyncKey"] lastObject] textValue]; - allMessages = [currentCollection syncTokenFieldsWithProperties: nil matchingSyncToken: syncKey fromDate: filter]; + allMessages = [currentCollection syncTokenFieldsWithProperties: nil matchingSyncToken: syncKey fromDate: filter initialLoad: NO]; count = [allMessages count]; diff --git a/SoObjects/Mailer/SOGoMailFolder.h b/SoObjects/Mailer/SOGoMailFolder.h index 84e51d9..8ea5953 100644 --- a/SoObjects/Mailer/SOGoMailFolder.h +++ b/SoObjects/Mailer/SOGoMailFolder.h @@ -98,7 +98,8 @@ - (NSArray *) syncTokenFieldsWithProperties: (NSDictionary *) properties matchingSyncToken: (NSString *) syncToken - fromDate: (NSCalendarDate *) theStartDate; + fromDate: (NSCalendarDate *) theStartDate + initialLoad: (BOOL) initialLoadInProgress; /* flags */ diff --git a/SoObjects/Mailer/SOGoMailFolder.m b/SoObjects/Mailer/SOGoMailFolder.m index c65c6c4..b38e9ee 100644 --- a/SoObjects/Mailer/SOGoMailFolder.m +++ b/SoObjects/Mailer/SOGoMailFolder.m @@ -2090,6 +2090,7 @@ _compareFetchResultsByMODSEQ (id entry1, id entry2, void *data) - (NSArray *) syncTokenFieldsWithProperties: (NSArray *) theProperties matchingSyncToken: (NSString *) theSyncToken fromDate: (NSCalendarDate *) theStartDate + initialLoad: (BOOL) initialLoadInProgress { EOQualifier *searchQualifier; NSMutableArray *allTokens; @@ -2166,6 +2167,9 @@ _compareFetchResultsByMODSEQ (id entry1, id entry2, void *data) for (i = 0; i < [fetchResults count]; i++) { + if ([[[fetchResults objectAtIndex: i] objectForKey: @"flags"] containsObject: @"deleted"] && initialLoadInProgress) + continue; + d = [NSDictionary dictionaryWithObject: ([[[fetchResults objectAtIndex: i] objectForKey: @"flags"] containsObject: @"deleted"]) ? [NSNull null] : [[fetchResults objectAtIndex: i] objectForKey: @"modseq"] forKey: [[[fetchResults objectAtIndex: i] objectForKey: @"uid"] stringValue]]; [allTokens addObject: d]; @@ -2176,7 +2180,7 @@ _compareFetchResultsByMODSEQ (id entry1, id entry2, void *data) if (highestmodseq == 0) highestmodseq = 1; - if (highestmodseq > 0) + if (highestmodseq > 0 && !initialLoadInProgress) { id uid; diff --git a/SoObjects/SOGo/SOGoGCSFolder.h b/SoObjects/SOGo/SOGoGCSFolder.h index 9b8de6b..578bb71 100644 --- a/SoObjects/SOGo/SOGoGCSFolder.h +++ b/SoObjects/SOGo/SOGoGCSFolder.h @@ -128,8 +128,8 @@ - (NSArray *) syncTokenFieldsWithProperties: (NSDictionary *) properties matchingSyncToken: (NSString *) syncToken - fromDate: (NSCalendarDate *) theStartDate; - + fromDate: (NSCalendarDate *) theStartDate + initialLoad: (BOOL) initialLoadInProgress; /* multiget helper */ - (WOResponse *) performMultigetInContext: (WOContext *) queryContext diff --git a/SoObjects/SOGo/SOGoGCSFolder.m b/SoObjects/SOGo/SOGoGCSFolder.m index 9f77359..add26d0 100644 --- a/SoObjects/SOGo/SOGoGCSFolder.m +++ b/SoObjects/SOGo/SOGoGCSFolder.m @@ -1151,6 +1151,7 @@ static NSArray *childRecordFields = nil; - (NSArray *) syncTokenFieldsWithProperties: (NSDictionary *) properties matchingSyncToken: (NSString *) syncToken fromDate: (NSCalendarDate *) theStartDate + initialLoad: (BOOL) initialLoadInProgress { /* TODO: - validation: @@ -1200,13 +1201,17 @@ static NSArray *childRecordFields = nil; mRecords = [NSMutableArray arrayWithArray: [self _fetchFields: fields withQualifier: qualifier ignoreDeleted: YES]]; - qualifier = [EOQualifier qualifierWithQualifierFormat: - @"c_lastmodified > %d and c_deleted == 1", - syncTokenInt]; - fields = [NSMutableArray arrayWithObjects: @"c_name", @"c_lastmodified", @"c_deleted", nil]; - [mRecords addObjectsFromArray: [self _fetchFields: fields - withQualifier: qualifier - ignoreDeleted: NO]]; + if (initialLoadInProgress) + { + qualifier = [EOQualifier qualifierWithQualifierFormat: + @"c_lastmodified > %d and c_deleted == 1", + syncTokenInt]; + fields = [NSMutableArray arrayWithObjects: @"c_name", @"c_lastmodified", @"c_deleted", nil]; + [mRecords addObjectsFromArray: [self _fetchFields: fields + withQualifier: qualifier + ignoreDeleted: NO]]; + } + records = mRecords; } else @@ -1499,7 +1504,9 @@ static NSArray *childRecordFields = nil; properties = [self parseDAVRequestedProperties: propElement]; records = [self syncTokenFieldsWithProperties: properties matchingSyncToken: syncToken - fromDate: nil]; + fromDate: nil + initialLoad: NO]; + [self _appendComponentProperties: [properties allKeys] fromRecords: records matchingSyncToken: [syncToken intValue] -- 1.7.10.4 |
|
I confirm that after many days the patches seems working. I hope that the patch will be merged (and updated) on the main tree. |
|
Hi, tried your patch with patch from 3294 on sogo nightlies from sogo-2.3.1.20150827. SOGoMailFolder(instance) does not recognize syncTokenFieldsWithProperties:matchingSyncToken Patching only with 3294 works. Maybe some code changed during latest nightlies? cheers, |
|
The patch works only on 2.3.0/1 (commit dd9b5e18b6dbc2e21df89208390abb4d0a75c955), days ago I've tried to built against recent nightly release and I confirm that does not works. I hope that the mantainer of SOGo merges these changes in next release to solve the reported problem. |
|
Patch pushed: https://github.com/inverse-inc/sogo/commit/07092828fec4f561602318573eb18b5f1a584eb4 |
|
Date Modified | Username | Field | Change |
---|---|---|---|
2015-07-23 18:59 | angeloxx | New Issue | |
2015-07-24 06:32 | angeloxx | Note Added: 0008767 | |
2015-07-24 07:13 | angeloxx | Note Added: 0008769 | |
2015-07-24 07:54 | angeloxx | Note Added: 0008770 | |
2015-08-02 08:31 | tfu | Note Added: 0008796 | |
2015-08-02 08:32 | tfu | File Added: 0001-3293-2.patch | |
2015-08-19 20:57 | angeloxx | Note Added: 0008835 | |
2015-08-27 16:50 | skasch | Note Added: 0008869 | |
2015-08-27 16:59 | angeloxx | Note Added: 0008870 | |
2015-09-09 14:23 | ludovic | Note Added: 0008898 | |
2015-09-09 14:23 | ludovic | Status | new => resolved |
2015-09-09 14:23 | ludovic | Fixed in Version | => 2.3.2 |
2015-09-09 14:23 | ludovic | Resolution | open => fixed |
2015-09-09 14:23 | ludovic | Assigned To | => ludovic |