View Issue Details

IDProjectCategoryView StatusLast Update
0002667SOGoActiveSyncpublic2014-03-19 15:45
Reportervinyard Assigned Toludovic  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Platform[Client] GoogleOSAndroidOS VersionKit Kat
Product Versionnightly v2 
Target Version2.2.2Fixed in Version2.2.2 
Summary0002667: Decode base64 and quoted-printable encoded multipart emails
Description

Currently multipart emails do not get decoded. Android 4.4.2 activesync client does not decode base64 or quoted-printable emails. First patch adds base64/quoted-printable decoding to encoded multipart emails. Second patch, I'm not sure if it's needed, adds quoted-printable decoding for html and plain-text emails.

Steps To Reproduce

View base64 or quoted-printable emails in Android 4.4.2 ActiveSync client.

TagsNo tags attached.

Activities

vinyard

vinyard

2014-03-18 22:21

reporter  

0001-Decode-base64-and-quoted-printable-encoded-multipart.patch (1,912 bytes)   
From 19d4aca11f5215d2fba4f9aafc746c45e86f378d Mon Sep 17 00:00:00 2001
From: David Rivera <vinyard@vnetgaming.net>
Date: Tue, 18 Mar 2014 13:31:27 -0700
Subject: [PATCH] Decode base64 and quoted-printable encoded multipart emails

---
 ActiveSync/SOGoMailObject+ActiveSync.m | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/ActiveSync/SOGoMailObject+ActiveSync.m b/ActiveSync/SOGoMailObject+ActiveSync.m
index 9f0ce50..4285c58 100644
--- a/ActiveSync/SOGoMailObject+ActiveSync.m
+++ b/ActiveSync/SOGoMailObject+ActiveSync.m
@@ -42,6 +42,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 #import <NGCards/iCalTimeZone.h>
 
 #import <NGExtensions/NGBase64Coding.h>
+#import <NGExtensions/NGQuotedPrintableCoding.h>
 #import <NGExtensions/NSString+misc.h>
 #import <NGExtensions/NSString+Encoding.h>
 #import <NGImap4/NGImap4Envelope.h>
@@ -196,7 +197,7 @@ struct GlobalObjectId {
 //
 - (NSData *) _preferredBodyDataInMultipartUsingType: (int) theType
 {
-  NSString *key, *plainKey, *htmlKey, *type, *subtype;
+  NSString *key, *plainKey, *htmlKey, *type, *subtype, *encoding;
   NSDictionary *textParts, *part;
   NSEnumerator *e;
   NSData *d;
@@ -222,12 +223,19 @@ struct GlobalObjectId {
   if (theType == 2)
     {
       d = [[self fetchPlainTextParts] objectForKey: htmlKey];
+      encoding = [[self lookupInfoForBodyPart: htmlKey] objectForKey: @"encoding"];
     }
   else if (theType == 1)
     {
       d = [[self fetchPlainTextParts] objectForKey: plainKey];
+      encoding = [[self lookupInfoForBodyPart: plainKey] objectForKey: @"encoding"];
     }
 
+  if ([encoding caseInsensitiveCompare: @"base64"] == NSOrderedSame)
+            d = [d dataByDecodingBase64];
+  else if ([encoding caseInsensitiveCompare: @"quoted-printable"] == NSOrderedSame)
+            d = [d dataByDecodingQuotedPrintableTransferEncoding];
+
   return d;
 }
 
-- 
1.7.12.4

vinyard

vinyard

2014-03-18 22:22

reporter  

0002-Decode-quoted-printable-encoded-html-and-plain-text.patch (990 bytes)   
From 8bf6e24f9332d0e533a7f770afea554a2f8ce928 Mon Sep 17 00:00:00 2001
From: David Rivera <vinyard@vnetgaming.net>
Date: Tue, 18 Mar 2014 14:14:24 -0700
Subject: [PATCH] Decode quoted-printable encoded html and plain text emails

---
 ActiveSync/SOGoMailObject+ActiveSync.m | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ActiveSync/SOGoMailObject+ActiveSync.m b/ActiveSync/SOGoMailObject+ActiveSync.m
index 4285c58..51425eb 100644
--- a/ActiveSync/SOGoMailObject+ActiveSync.m
+++ b/ActiveSync/SOGoMailObject+ActiveSync.m
@@ -274,6 +274,8 @@ struct GlobalObjectId {
 
           if ([encoding caseInsensitiveCompare: @"base64"] == NSOrderedSame)
             d = [d dataByDecodingBase64];
+          else if ([encoding caseInsensitiveCompare: @"quoted-printable"] == NSOrderedSame)
+            d = [d dataByDecodingQuotedPrintableTransferEncoding];
 
           // Check if we must convert html->plain
           if (theType == 1 && [subtype isEqualToString: @"html"])
-- 
1.7.12.4

ludovic

ludovic

2014-03-18 22:24

administrator   ~0006720

Thanks - I've already produced a more extensive patch.

Can you test it? I've attached it to this ticket.

ludovic

ludovic

2014-03-18 22:27

administrator  

outlook-encoding.diff (7,392 bytes)   
diff --git a/ActiveSync/SOGoMailObject+ActiveSync.m b/ActiveSync/SOGoMailObject+ActiveSync.m
index 9f0ce50..933bee3 100644
--- a/ActiveSync/SOGoMailObject+ActiveSync.m
+++ b/ActiveSync/SOGoMailObject+ActiveSync.m
@@ -42,12 +42,21 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 #import <NGCards/iCalTimeZone.h>
 
 #import <NGExtensions/NGBase64Coding.h>
+#import <NGExtensions/NGQuotedPrintableCoding.h>
 #import <NGExtensions/NSString+misc.h>
 #import <NGExtensions/NSString+Encoding.h>
 #import <NGImap4/NGImap4Envelope.h>
 #import <NGImap4/NGImap4EnvelopeAddress.h>
 #import <NGObjWeb/WOContext+SoObjects.h>
 
+#import <NGMime/NGMimeBodyPart.h>
+#import <NGMime/NGMimeFileData.h>
+#import <NGMime/NGMimeMultipartBody.h>
+#import <NGMime/NGMimeType.h>
+#import <NGMail/NGMimeMessageParser.h>
+#import <NGMail/NGMimeMessage.h>
+#import <NGMail/NGMimeMessageGenerator.h>
+
 #include "iCalTimeZone+ActiveSync.h"
 #include "NSData+ActiveSync.h"
 #include "NSDate+ActiveSync.h"
@@ -196,7 +205,7 @@ struct GlobalObjectId {
 //
 - (NSData *) _preferredBodyDataInMultipartUsingType: (int) theType
 {
-  NSString *key, *plainKey, *htmlKey, *type, *subtype;
+  NSString *encoding, *key, *plainKey, *htmlKey, *type, *subtype;
   NSDictionary *textParts, *part;
   NSEnumerator *e;
   NSData *d;
@@ -219,15 +228,132 @@ struct GlobalObjectId {
         plainKey = key;
     }
 
+  key = nil;
+
   if (theType == 2)
+    key = htmlKey;
+  else if (theType == 1)
+    key = plainKey;
+
+  if (key)
     {
       d = [[self fetchPlainTextParts] objectForKey: htmlKey];
+
+      encoding = [[self lookupInfoForBodyPart: key] objectForKey: @"encoding"];
+      
+      if ([encoding caseInsensitiveCompare: @"base64"] == NSOrderedSame)
+        d = [d dataByDecodingBase64];
+      else if ([encoding caseInsensitiveCompare: @"quoted-printable"] == NSOrderedSame)
+        d = [d dataByDecodingQuotedPrintable];
     }
-  else if (theType == 1)
+
+  return d;
+}
+
+//
+//
+//
+- (void) _sanitizedMIMEPart: (id) thePart
+                  performed: (BOOL *) b
+{
+  if ([thePart isKindOfClass: [NGMimeMultipartBody class]])
+    {
+      NGMimeBodyPart *part;
+      NSArray *parts;
+      int i;
+
+      parts = [thePart parts];
+      
+      for (i = 0; i < [parts count]; i++)
+        {
+          part = [parts objectAtIndex: i];
+
+          [self _sanitizedMIMEPart: part
+                         performed: b];
+        }
+    }
+  else if ([thePart isKindOfClass: [NGMimeBodyPart class]])
     {
-      d = [[self fetchPlainTextParts] objectForKey: plainKey];
+      NGMimeFileData *fdata;
+      id body;
+
+      body = [thePart body];
+
+      if ([body isKindOfClass: [NGMimeMultipartBody class]])
+        {
+          [self _sanitizedMIMEPart: body
+                         performed: b];
+        }
+      else if ([body isKindOfClass: [NSData class]] &&
+               [[[thePart contentType] type] isEqualToString: @"text"])
+        {
+          // We make sure everything is encoded in UTF-8
+          NSString *charset, *s;
+          int encoding;
+
+          charset = [[thePart contentType] valueOfParameter: @"charset"];
+          encoding = [NGMimeType stringEncodingForCharset: charset];
+
+          s = [[NSString alloc] initWithData: body  encoding: encoding];
+          AUTORELEASE(s);
+
+          if (s)
+            {
+              body = [s dataUsingEncoding: NSUTF8StringEncoding];
+            }
+
+          NGMimeType *mimeType;
+
+          mimeType = [NGMimeType mimeType: [[thePart contentType] type]
+                                  subType: [[thePart contentType] subType]
+                               parameters: [NSDictionary dictionaryWithObject: @"utf-8"  forKey: @"charset"]];
+          [thePart setHeader: mimeType  forKey: @"content-type"];
+          
+          fdata = [[NGMimeFileData alloc] initWithBytes: [body bytes]
+                                                 length: [body length]];
+          
+          [thePart setBody: fdata];                  
+          RELEASE(fdata);
+          *b = YES;
+        }
     }
+}
+
+//
+//
+//
+- (NSData *) _sanitizedMIMEMessage
+{
+  NGMimeMessageParser *parser;
+  NGMimeMessage *message;
+  NSData *d;
+  
+  BOOL b;
+  
+  d = [self content];
+
+  parser = [[NGMimeMessageParser alloc] init];
+  AUTORELEASE(parser);
+  
+  message = [parser parsePartFromData: d];
+  b = NO;
+
+  if (message)
+    {
+      [self _sanitizedMIMEPart: [message body]
+                     performed: &b];
 
+      if (b)
+        {
+          NGMimeMessageGenerator *generator;
+          
+          generator = [[NGMimeMessageGenerator alloc] init];
+          AUTORELEASE(generator);
+          
+          d = [generator generateMimeFromPart: message];
+        }
+    }
+  
   return d;
 }
 
@@ -266,6 +392,8 @@ struct GlobalObjectId {
 
           if ([encoding caseInsensitiveCompare: @"base64"] == NSOrderedSame)
             d = [d dataByDecodingBase64];
+          else if ([encoding caseInsensitiveCompare: @"quoted-printable"] == NSOrderedSame)
+            d = [d dataByDecodingQuotedPrintable];
 
           // Check if we must convert html->plain
           if (theType == 1 && [subtype isEqualToString: @"html"])
@@ -286,7 +414,8 @@ struct GlobalObjectId {
     }
   else if (theType == 4)
     {
-      d = [self content];
+      d = [self _sanitizedMIMEMessage];
+      //d = [self content];
     }
 
   return d;
@@ -531,6 +660,7 @@ struct GlobalObjectId {
   //  [s appendFormat: @"<Reply-To xmlns=\"Email:\">%@</Reply-To>", [addressFormatter stringForArray: replyTo]];
   
   // InternetCPID - 65001 == UTF-8, we use this all the time for now.
+  //              - 20127 == US-ASCII
   [s appendFormat: @"<InternetCPID xmlns=\"Email:\">%@</InternetCPID>", @"65001"];
           
   // Body - namespace 17
@@ -544,11 +674,16 @@ struct GlobalObjectId {
       NSString *content;
       int len, truncated;
       
+      //d = [d dataByEncodingBase64];
       content = [[NSString alloc] initWithData: d  encoding: NSUTF8StringEncoding];
 
       // FIXME: This is a hack. We should normally avoid doing this as we might get
       // broken encodings. We should rather tell that the data was truncated and expect
       // a ItemOperations call to download the whole base64 encoding multipart.
+      //
+      // See http://social.msdn.microsoft.com/Forums/en-US/b9944e49-9bc9-4ab8-ba33-a9fc08557c5b/mime-raw-data-in-eas-sync-response?forum=os_exchangeprotocols
+      // for an "interesting" discussion around this.
+      //
       if (!content)
         content = [[NSString alloc] initWithData: d  encoding: NSISOLatin1StringEncoding];
       
@@ -561,10 +696,14 @@ struct GlobalObjectId {
       
       [s appendString: @"<Body xmlns=\"AirSyncBase:\">"];
       [s appendFormat: @"<Type>%d</Type>", preferredBodyType];
-      [s appendFormat: @"<EstimatedDataSize>%d</EstimatedDataSize>", len];
-      [s appendFormat: @"<Truncated>%d</Truncated>", 0];
+      [s appendFormat: @"<Truncated>%d</Truncated>", truncated];
+      [s appendFormat: @"<Preview></Preview>"];
+
       if (!truncated)
-        [s appendFormat: @"<Data>%@</Data>", content];
+        {
+          [s appendFormat: @"<Data>%@</Data>", content];
+          [s appendFormat: @"<EstimatedDataSize>%d</EstimatedDataSize>", len];
+        }
       [s appendString: @"</Body>"];
     }
 
outlook-encoding.diff (7,392 bytes)   
vinyard

vinyard

2014-03-18 22:40

reporter   ~0006721

Of course, I'll post back as soon as I get the chance.

vinyard

vinyard

2014-03-18 22:57

reporter   ~0006722

Found one error in your patch.

line 240 in SOGoMailObject+ActiveSync.m:
d = [[self fetchPlainTextParts] objectForKey: htmlKey];
should be
d = [[self fetchPlainTextParts] objectForKey: key];

vinyard

vinyard

2014-03-18 23:09

reporter   ~0006724

Also, I used dataByDecodingQuotedPrintableTransferEncoding instead of dataByDecodingQuotedPrintable as it was replacing underscore wiich was breaking linked images in emails adn some URLs. (URLs had undescores)

ludovic

ludovic

2014-03-18 23:10

administrator   ~0006725

Good catches.

vinyard

vinyard

2014-03-18 23:20

reporter   ~0006726

With the above changes to your patch, the emails are displayed properly on android.

ludovic

ludovic

2014-03-18 23:22

administrator   ~0006727

Can you disable sanitization at line 417 and retest?

Normally, it should only be used for Outlook clients.

vinyard

vinyard

2014-03-18 23:30

reporter   ~0006728

Sure, but I wouldn't think it would hit this code path as the preferred body type would be 2, line 417 would only be hit if the preferred body type would be 4, correct? I'll disable it and let you know.

ludovic

ludovic

2014-03-18 23:38

administrator   ~0006729

You would only hit it if your device asks for the complete MIME content.

vinyard

vinyard

2014-03-18 23:47

reporter   ~0006730

Oh I see, thanks. I've disabled sanitization, still displaying correctly on android.

ludovic

ludovic

2014-03-18 23:57

administrator   ~0006731

Perfect, thanks for your testing time, much appreciated.

vinyard

vinyard

2014-03-19 00:22

reporter   ~0006732

Glad I could help out.

ludovic

ludovic

2014-03-19 15:45

administrator   ~0006737

https://github.com/inverse-inc/sogo/commit/fc56493b782009e19fbc5d017bdbdb3b71dba6f6

Issue History

Date Modified Username Field Change
2014-03-18 22:21 vinyard New Issue
2014-03-18 22:21 vinyard File Added: 0001-Decode-base64-and-quoted-printable-encoded-multipart.patch
2014-03-18 22:22 vinyard File Added: 0002-Decode-quoted-printable-encoded-html-and-plain-text.patch
2014-03-18 22:24 ludovic Note Added: 0006720
2014-03-18 22:27 ludovic File Added: outlook-encoding.diff
2014-03-18 22:40 vinyard Note Added: 0006721
2014-03-18 22:57 vinyard Note Added: 0006722
2014-03-18 23:09 vinyard Note Added: 0006724
2014-03-18 23:10 ludovic Note Added: 0006725
2014-03-18 23:20 vinyard Note Added: 0006726
2014-03-18 23:22 ludovic Note Added: 0006727
2014-03-18 23:30 vinyard Note Added: 0006728
2014-03-18 23:38 ludovic Note Added: 0006729
2014-03-18 23:47 vinyard Note Added: 0006730
2014-03-18 23:57 ludovic Note Added: 0006731
2014-03-18 23:57 ludovic Target Version => 2.2.2
2014-03-19 00:22 vinyard Note Added: 0006732
2014-03-19 15:45 ludovic Note Added: 0006737
2014-03-19 15:45 ludovic Status new => closed
2014-03-19 15:45 ludovic Assigned To => ludovic
2014-03-19 15:45 ludovic Resolution open => fixed
2014-03-19 15:45 ludovic Fixed in Version => 2.2.2