View Issue Details

IDProjectCategoryView StatusLast Update
0005620SOGoActiveSyncpublic2023-02-02 19:41
Reporterbgaussen Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformAppleOSiOSOS Version16
Product Version5.7.1 
Summary0005620: EAS server search fails
Description

Running SOGo on a mallow installation, I experience issues with EAS servers searches that don't return any results.
Enabling EAS debug and IMAP logging I extracted the following relevant logs, with a 501 error code answer to EAS request:

mailcowdockerized-sogo-mailcow-1 | Oct 6 17:17:17 90a0cb111729 sogod [55]: <0x0x55aacaef50d0[SOGoActiveSyncDispatcher]> EAS - request for device TK8BPB39TT36L9HIRFF63UHEOG: <?xml version="1.0"?>
mailcowdockerized-sogo-mailcow-1 | Oct 6 17:17:17 90a0cb111729 syslog-ng[9]: Error processing log message: <!DOCTYPE ActiveSync PUBLIC "-//MICROSOFT//DTD ActiveSync//EN" "http://www.microsoft.com/&quot;>
mailcowdockerized-sogo-mailcow-1 | <unknown xmlns="FolderHierarchy:">
mailcowdockerized-sogo-mailcow-1 | <unknown>18C9FEF4-7858-48C4-BF82-BBCBC9BCAF97</unknown>
mailcowdockerized-sogo-mailcow-1 | <unknown>
mailcowdockerized-sogo-mailcow-1 | <unknown>
mailcowdockerized-sogo-mailcow-1 | <unknown>
mailcowdockerized-sogo-mailcow-1 | <Class>Email</Class>
mailcowdockerized-sogo-mailcow-1 | <CollectionId>mail%2F614be7007a976a595f000000c147d4d9</CollectionId>
mailcowdockerized-sogo-mailcow-1 | <unknown>to:"X1234x4321" OR cc:"X1234x4321" OR from:"X1234x4321" OR subject:"X1234x4321" OR "X1234x4321"</unknown>
mailcowdockerized-sogo-mailcow-1 | </unknown>
mailcowdockerized-sogo-mailcow-1 | <unknown>
mailcowdockerized-sogo-mailcow-1 | <unknown>0-99</unknown>
mailcowdockerized-sogo-mailcow-1 | <unknown/>
mailcowdockerized-sogo-mailcow-1 | </unknown>
mailcowdockerized-sogo-mailcow-1 | </unknown>
mailcowdockerized-sogo-mailcow-1 | </unknown>
mailcowdockerized-sogo-mailcow-1 | </unknown>
mailcowdockerized-sogo-mailcow-1 |
mailcowdockerized-sogo-mailcow-1 | DITY 1500157820)
mailcowdockerized-sogo-mailcow-1 | Oct 6 17:17:17 90a0cb111729 2022-10-06 17:17:17.499 sogod[55:55] EXCEPTION: <NSException: 0x55aacae92d80> NAME:NSInvalidArgumentException REASON:-[SOGoActiveSyncDispatcher processunknown:inResponse:]: unrecognized selector sent to instance 0x55aacaef50d0 INFO:(null)
mailcowdockerized-sogo-mailcow-1 | Oct 6 17:17:17 90a0cb111729 sogod [55]: ::ffff:161.105.234.66, ::ffff:161.105.234.66 "POST /SOGo/Microsoft-Server-ActiveSync?User=ben@trez.org&DeviceId=TK8BPB39TT36L9HIRFF63UHEOG&DeviceType=iPhone&Cmd=Find HTTP/1.0" 501 0/220 0.009 - - 2M - 11

Please ask if more or more precise logs needed.

Steps To Reproduce

Search a SOGo mailbox from iOS 16 mail client. Only local results shown.

TagsNo tags attached.

Activities

mayk

mayk

2022-11-06 14:17

reporter   ~0016340

Can confirm. I test on ios 16.x and only local mails (within defined sync range) are returned. Testing this on older ios (version 12) works.
Tested and confirm breakage with ios mail app, outlook ios app and bluemail on ios . All on version 16. Ios mail app , and bluemail on ios 12 work correctly.

There must be a change in ios, as when i test this against another setup with z-push, the behavior is exactly the same.

tfu

tfu

2022-11-11 19:24

reporter   ~0016384

This is not yet implemented. Depends on https://github.com/libwbxml/libwbxml/pull/86.

sebastien

sebastien

2023-01-25 17:55

administrator   ~0016599

Hi @tfu,

Thanks for the feedback, It seems that PR has been merged. Installed the lib but it seems somes changes are needed in the code also :

  • Add processFind:inResponse: in SOGoActiveSyncDispatcher (call processSearch in that)
  • Check Class for name in processSearch for Email
  • Update _qualifierFromMailboxSearchQuery and extract FreeText to get subject, from and text

For those steps it's ok. However the response provided by SOGo seems to be ok but iOS still displays "no results".

Any idea of where to get documentation ?

Here is the search EAS request :

&lt;!DOCTYPE ActiveSync PUBLIC &quot;-//MICROSOFT//DTD ActiveSync//EN&quot; &quot;http://www.microsoft.com/&quot;>
&lt;Find xmlns=&quot;Find:&quot;>
 &lt;SearchId>2C344390-3A86-47B2-8502-FC90F8417645&lt;/SearchId>
 &lt;ExecuteSearch>
  &lt;MailBoxSearchCriterion>
   &lt;Query>
    &lt;Class xmlns=&quot;AirSync:&quot;>Email&lt;/Class>
    &lt;CollectionId xmlns=&quot;AirSync:&quot;>mail%2Fc8a2ed063d135063c3300000c0848020&lt;/CollectionId>
    &lt;FreeText>to:&quot;Test&quot; OR cc:&quot;Test&quot; OR from:&quot;Test&quot; OR subject:&quot;Test&quot; OR &quot;Test&quot;&lt;/FreeText>
   &lt;/Query>
   &lt;Options>
    &lt;Range>0-99&lt;/Range>
    &lt;DeepTraversal/>
   &lt;/Options>
  &lt;/MailBoxSearchCriterion>
 &lt;/ExecuteSearch>
&lt;/Find>

and extract of the response :

&lt;!DOCTYPE ActiveSync PUBLIC &quot;-//MICROSOFT//DTD ActiveSync//EN&quot; &quot;http://www.microsoft.com/&quot;>
&lt;Search xmlns=&quot;Search:&quot;>
 &lt;Status>1&lt;/Status>
 &lt;Response>
  &lt;Store>
   &lt;Status>1&lt;/Status>
   &lt;Result>
    &lt;LongId>mail%2Fc8a2ed063d135063c3300000c0848020+19&lt;/LongId>
    &lt;CollectionId xmlns=&quot;AirSync:&quot;>mail%2Fc8a2ed063d135063c3300000c0848020&lt;/CollectionId>
    &lt;Properties>
     &lt;To xmlns=&quot;Email:&quot;>&quot;s m&quot; &lt;xxx@www.org>&lt;/To>
     &lt;DisplayTo xmlns=&quot;Email:&quot;>s m&lt;/DisplayTo>
     &lt;From xmlns=&quot;Email:&quot;>&quot;s m&quot; &lt;xxx@www.org>&lt;/From>
     &lt;Subject xmlns=&quot;Email:&quot;>coucou &lt;a href=&quot;sqdq&quot;>sdqd&lt;/a>&lt;/Subject>
     &lt;ThreadTopic xmlns=&quot;Email:&quot;>coucou &lt;a href=&quot;sqdq&quot;>sdqd&lt;/a>&lt;/ThreadTopic>
     &lt;DateReceived xmlns=&quot;Email:&quot;>2022-11-14T13:26:33.000Z&lt;/DateReceived>
     &lt;Importance xmlns=&quot;Email:&quot;>1&lt;/Importance>
     &lt;Read xmlns=&quot;Email:&quot;>0&lt;/Read>
     &lt;MessageClass xmlns=&quot;Email:&quot;>IPM.Note&lt;/MessageClass>
     &lt;ContentClass xmlns=&quot;Email:&quot;>urn:content-classes:message&lt;/ContentClass>
     &lt;Reply-To xmlns=&quot;Email:&quot;>s m &lt;xxx@www.org>&lt;/Reply-To>
     &lt;InternetCPID xmlns=&quot;Email:&quot;>65001&lt;/InternetCPID>
     &lt;Flag xmlns=&quot;Email:&quot;>
      &lt;FlagStatus>0&lt;/FlagStatus>
     &lt;/Flag>
     &lt;Categories xmlns=&quot;Email:&quot;/>
     &lt;ConversationId xmlns=&quot;Email2:&quot;>PGUwNjAtNjM3MjQyMDAtMy03MjMzNTQwMEAyNDU1MjQ1Mjk+&lt;/ConversationId>
     &lt;LastVerbExecuted xmlns=&quot;Email2:&quot;>0&lt;/LastVerbExecuted>
     &lt;NativeBodyType xmlns=&quot;AirSyncBase:&quot;>1&lt;/NativeBodyType>
    &lt;/Properties>
   &lt;/Result>
...
   &lt;Range>0-23&lt;/Range>
   &lt;Total>24&lt;/Total>
  &lt;/Store>
 &lt;/Response>
&lt;/Search>
tfu

tfu

2023-02-02 19:41

reporter   ~0016644

I drafted a fix for this some time ago but don't have to time finish it. Maybe it helps.

5620.diff (12,993 bytes)   
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m
index 03b378690..7f9ce6845 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher.m
@@ -341,18 +341,30 @@ void handle_eas_terminate(int signum)
     default:
       {
         SOGoMailAccounts *accountsFolder;
-        SOGoMailFolder *currentFolder;
+        SOGoMailAccount *currentFolder;
         SOGoUserFolder *userFolder;

         userFolder = [[context activeUser] homeFolderInContext: context];
         accountsFolder = [userFolder lookupName: @"Mail"  inContext: context  acquire: NO];
         currentFolder = [accountsFolder lookupName: @"0"  inContext: context  acquire: NO];
-
         collection = [currentFolder lookupName: [NSString stringWithFormat: @"folder%@", theCollectionId]
                                      inContext: context
                                        acquire: NO];
-        if (![(SOGoMailFolder *)collection exists])
-           collection = nil;
+
+        if (![(SOGoMailFolder *)collection exists]) {
+            // If templates folder doesn't exists yet, then create it.
+            if ([theCollectionId isEqualToString: [currentFolder templatesFolderNameInContext: context]]) {
+               if ([(SOGoMailFolder *)[currentFolder templatesFolderInContext: context] create]) {
+                  collection = [currentFolder lookupName: [NSString stringWithFormat: @"folder%@", theCollectionId]
+                                               inContext: context
+                                                 acquire: NO];
+
+                }
+             }
+
+            if (![(SOGoMailFolder *)collection exists])
+              collection = nil;
+          }
       }
     }

@@ -3152,6 +3164,106 @@ void handle_eas_terminate(int signum)
   return nil;
 }

+- (EOQualifier *) _qualifierFromMailboxFindQuery: (id <DOMElement>) theDocumentElement
+{
+  id <DOMElement> freeTextElement;
+  NSMutableArray *qualifiers;
+  EOQualifier *fetchQualifier;
+  NSRange r, r1;
+  NSString *a, *s, *query;
+
+  freeTextElement = [(id)[theDocumentElement getElementsByTagName: @"FreeText"] lastObject];
+  query = [(id)freeTextElement textValue];
+
+  if (!query)
+    return nil;
+
+  //query = @"mmm OR to:xxx";
+  qualifiers = [NSMutableArray array];
+
+  while ([query length] > 0)
+   {
+     r = [query rangeOfString:@":"];
+     r1 = [query rangeOfString:@" "];
+     if (r.location != NSNotFound && (r.location < r1.location  || r1.location == NSNotFound))
+       {
+         a = [query substringToIndex:r.location];
+         query =[query substringFromIndex:r.location+1];
+
+         if ([query hasPrefix:@"\""])
+           {
+             query = [query substringFromIndex:1];
+             r = [query rangeOfString:@"\""];
+             s = [query substringToIndex:r.location];
+             query =[query substringFromIndex:r.location+1]; // strip quote
+           }
+         else
+          {
+            r = [query rangeOfString:@" "];
+            if (r.location == NSNotFound)
+              {
+                s = query;
+                query = @"";
+              }
+           else
+             {
+                s = [query substringToIndex:r.location];
+                query =[query substringFromIndex:r.location];
+              }
+          }
+       }
+     else if ([query hasPrefix: @"OR"])
+       {
+         s = [query substringToIndex:2];
+         query =[query substringFromIndex:2];
+         a = @"";
+       }
+     else if ([query hasPrefix:@"\""])
+       {
+         query = [query substringFromIndex:1];
+         r = [query rangeOfString:@"\""];
+         s = [query substringToIndex:r.location];
+         query =[query substringFromIndex:r.location+1]; // strip quote
+         a = @"text";
+       }
+     else
+       {
+         r = [query rangeOfString:@" "];
+         if (r.location == NSNotFound)
+           {
+             s = query;
+             query = @"";
+          }
+         else
+           {
+             s = [query substringToIndex:r.location];
+             query =[query substringFromIndex:r.location];
+           }
+         a = @"text";
+       }
+
+      if ([query hasPrefix:@" "])
+       {
+         query = [query substringFromIndex: 1];
+       }
+
+     NSLog(@"tfu a: %@", a);
+     NSLog(@"tfu s: %@", s);
+     NSLog(@"tfu query: %@", query);
+     NSLog(@"tfu length: %d", [query length]);
+
+     if ([a length] > 0)
+       {
+         [qualifiers addObject: [EOQualifier qualifierWithQualifierFormat: [NSString stringWithFormat: @"(%@ doesContain: '%@')", a, s]]];
+       }
+   }
+
+  fetchQualifier = [[EOOrQualifier alloc] initWithQualifierArray: qualifiers];
+
+  NSLog(@"tfu qual %@",fetchQualifier);
+   return [fetchQualifier autorelease];
+
+}
 //
 // <!DOCTYPE ActiveSync PUBLIC "-//MICROSOFT//DTD ActiveSync//EN" "http://www.microsoft.com/">
 // <Search xmlns="Search:">
@@ -3213,9 +3325,10 @@ void handle_eas_terminate(int signum)
       return;
     }

-  bodyPreferenceType = [[(id)[[(id)[theDocumentElement getElementsByTagName: @"BodyPreference"] lastObject] getElementsByTagName: @"Type"] lastObject] textValue];
+  bodyPreferenceType = [[(id)[[(id)[theDocumentElement getElementsByTagName: @"BodyPreference"] firstObject] getElementsByTagName: @"Type"] lastObject] textValue];
   [context setObject: bodyPreferenceType  forKey: @"BodyPreferenceType"];
   mimeSupport = [[(id)[theDocumentElement getElementsByTagName: @"MIMESupport"] lastObject] textValue];
+  NSLog(@"tfu mimeSupport %@",mimeSupport);
   [context setObject: mimeSupport  forKey: @"MIMESupport"];

   [context setObject: @"8" forKey: @"MIMETruncation"];
@@ -3334,6 +3447,149 @@ void handle_eas_terminate(int signum)
   [theResponse setContent: d];
 }

+- (void) processFindMailbox: (id <DOMElement>) theDocumentElement
+           inResponse: (WOResponse *) theResponse
+{
+  NSString *folderId, *realCollectionId, *itemId;
+  NSMutableArray *folderIdentifiers;
+  SOGoMailAccounts *accountsFolder;
+  SOGoMailAccount *accountFolder;
+  SOGoMailFolder *currentFolder;
+  SOGoMailObject *mailObject;
+  SOGoUserFolder *userFolder;
+  EOQualifier *qualifier;
+  NSArray *sortedUIDs, *a;
+  NSMutableString *s;
+  NSData *d;
+
+  SOGoMicrosoftActiveSyncFolderType folderType;
+  int i, j, total, begin, startRange, endRange, maxResults, overallTotal;
+
+  overallTotal = 0;
+
+  // We build the qualifier and we launch our search operation
+  qualifier = [self _qualifierFromMailboxFindQuery: [(id)[theDocumentElement getElementsByTagName: @"Query"] lastObject]];
+
+  if (!qualifier)
+    {
+      [theResponse setStatus: 500];
+      return;
+    }
+
+  // We check for the maximum number of results to return.
+  a = [[[(id)[theDocumentElement getElementsByTagName: @"Range"] lastObject] textValue] componentsSeparatedByString: @"-"];
+  startRange = [[a objectAtIndex: 0] intValue];
+  begin = startRange;
+  endRange = [[a objectAtIndex: 1] intValue];
+  maxResults = endRange - startRange;
+
+  if (maxResults == 0)
+    maxResults = endRange = 99;
+
+  // FIXME: support more than one CollectionId tag + DeepTraversal
+  folderId = [[(id)[[(id)[theDocumentElement getElementsByTagName: @"Query"] lastObject] getElementsByTagName: @"CollectionId"] lastObject] textValue];
+  folderIdentifiers = [NSMutableArray array];
+
+  // Android 6 will send search requests with no collection ID - so we search in all folders.
+  // Outlook Mobile App sends search requests with CollectionId=0 - We treat this as an all-folder-search.
+  if (!folderId || [folderId isEqualToString: @"0"])
+    {
+      NSArray *foldersInCache;
+      SOGoCacheGCSObject *o;
+      NSString *prefix;
+
+      o = [SOGoCacheGCSObject objectWithName: @"0" inContainer: nil];
+      [o setObjectType: ActiveSyncFolderCacheObject];
+      [o setTableUrl: folderTableURL];
+
+      foldersInCache = [o cacheEntriesForDeviceId: [context objectForKey: @"DeviceId"] newerThanVersion: -1];
+      prefix = [NSString stringWithFormat: @"/%@+folder", [context objectForKey: @"DeviceId"]];
+
+      for (i = 0; i < [foldersInCache count]; i++)
+        {
+          folderId = [foldersInCache objectAtIndex: i];
+          if ([folderId hasPrefix: prefix])
+            {
+              folderId = [NSString stringWithFormat: @"mail/%@", [folderId substringFromIndex: [prefix length]]];
+              [folderIdentifiers addObject: folderId];
+            }
+        }
+    }
+  else
+    {
+      [folderIdentifiers addObject: folderId];
+    }
+
+  userFolder = [[context activeUser] homeFolderInContext: context];
+  accountsFolder = [userFolder lookupName: @"Mail"  inContext: context  acquire: NO];
+  accountFolder = [accountsFolder lookupName: @"0"  inContext: context  acquire: NO];
+
+  // Prepare the response
+  s = [NSMutableString string];
+  [s appendString: @"<?xml version=\"1.0\" encoding=\"utf-8\"?>"];
+  [s appendString: @"<!DOCTYPE ActiveSync PUBLIC \"-//MICROSOFT//DTD ActiveSync//EN\" \"http://www.microsoft.com/\">"];
+  [s appendString: @"<Find xmlns=\"Find:\">"];
+  [s appendFormat: @"<Status>1</Status>"];
+  [s appendFormat: @"<Response xmlns=\"Find:\">"];
+  [s appendFormat: @"<Store xmlns=\"ItemOperations:\">Mailbox</Store>"];
+  [s appendFormat: @"<Status>1</Status>"];
+
+  for (i = 0; i < [folderIdentifiers count]; i++)
+    {
+      folderId = [folderIdentifiers objectAtIndex: i];
+      realCollectionId = [folderId realCollectionIdWithFolderType: &folderType];
+      realCollectionId = [self globallyUniqueIDToIMAPFolderName: realCollectionId  type: folderType];
+
+      currentFolder = [accountFolder lookupName: [NSString stringWithFormat: @"folder%@", realCollectionId]
+                                      inContext: context
+                                        acquire: NO];
+
+      sortedUIDs = [currentFolder fetchUIDsMatchingQualifier: qualifier
+                                                sortOrdering: @"REVERSE ARRIVAL"
+                                                    threaded: NO];
+      total = [sortedUIDs count];
+      overallTotal+=total;
+
+      if (total < startRange)
+        {
+          begin -= total;
+          continue;;
+        }
+
+      for (j = begin; j < total && maxResults >= 0; j++)
+        {
+          itemId = [[sortedUIDs objectAtIndex: j] stringValue];
+          mailObject = [currentFolder lookupName: itemId  inContext: context  acquire: NO];
+
+          if ([mailObject isKindOfClass: [NSException class]])
+            continue;
+
+          maxResults--;
+
+          [s appendString: @"<Result xmlns=\"Find:\">"];
+          [s appendString: @"<Class>Email</Class>"];
+          [s appendFormat: @"<ServerId>%@</ServerId>",itemId];
+          [s appendFormat: @"<CollectionId xmlns=\"AirSyncBase:\">%@</CollectionId>", folderId];
+          [s appendString: @"<Properties xmlns=\"Find:\">"];
+          [s appendString: [mailObject activeSyncRepresentationInContext: context]];
+          [s appendString: @"</Properties>"];
+          [s appendFormat: @"</Result>"];
+        }
+    }
+
+  if (overallTotal < startRange)
+    overallTotal = 0;
+
+  [s appendFormat: @"<Range xmlns=\"Find:\">%d-%d</Range>",(overallTotal ? startRange : 0), (overallTotal ? endRange - maxResults - 1 : 0)];
+  [s appendFormat: @"<Total xmlns=\"Find:\">%d</Total>", overallTotal];
+  [s appendString: @"</Response>"];
+  [s appendString: @"</Find>"];
+
+  d = [[s dataUsingEncoding: NSUTF8StringEncoding] xml2wbxml];
+
+  [theResponse setContent: d];
+}
+
 //
 // We support EAS Search on the GAL and Mailbox.
 //
@@ -3361,6 +3617,31 @@ void handle_eas_terminate(int signum)
   return;
 }

+- (void) processFind: (id <DOMElement>) theDocumentElement
+            inResponse: (WOResponse *) theResponse
+{
+  id <DOMElement> documentElement;
+
+  if ((documentElement = [(id)[theDocumentElement getElementsByTagName: @"MailBoxSearchCriterion"] lastObject]))
+    {
+      return [self processFindMailbox: theDocumentElement
+                             inResponse: theResponse];
+      //qualifier = [self _qualifierFromMailboxFindQuery: [(id)[theDocumentElement getElementsByTagName: @"Query"] lastObject]];
+
+
+//      return [self processSearchGAL: theDocumentElement
+//                         inResponse: theResponse];
+    }
+  else if ((documentElement = [(id)[theDocumentElement getElementsByTagName: @"GALSearchCriterion"] lastObject]))
+    {
+ //     return [self processSearchMailbox: theDocumentElement
+ //                            inResponse: theResponse];
+    }
+
+  [theResponse setStatus: 500];
+  return;
+}
+
 //
 //
 //
5620.diff (12,993 bytes)   

Issue History

Date Modified Username Field Change
2022-10-07 09:15 bgaussen New Issue
2022-11-06 14:17 mayk Note Added: 0016340
2022-11-11 19:24 tfu Note Added: 0016384
2023-01-25 17:55 sebastien Note Added: 0016599
2023-02-02 19:41 tfu Note Added: 0016644
2023-02-02 19:41 tfu File Added: 5620.diff