View Issue Details

IDProjectCategoryView StatusLast Update
0001483SOGoApple iPhone OSpublic2011-11-17 16:22
ReporterMarcel Assigned Toludovic  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Version1.3.9 
Target Version1.3.10Fixed in Version1.3.10 
Summary0001483: CalDAV requests for URL-encoded UIDs with @ signs fail with 404 Not found
Description

When an iPhone (iOS5) receives CalDAV entries with @ signs in their UID-derived name as a result of PROPFIND (contenttype and getetag), it queries for them URL-encoded (in the REPORT request), i.e. @ is replaced by %40.

SOGo does not seem to URL-decode the entries first, which results in iOS5 silently not displaying these entries. As the popular German calendar generator site www.project24.info creates UIDs of the form <whatever>@PROJECT24.INFO, these will all be ignored, which hit several of our users after upgrading to iOS5.

======

I also noticed another potential problem with generating the URLs from the UIDs: Dots (.) are silently removed. The URL for UID test@project24.info is /SOGo/dav/user/Calendar/test@project24info.ics . I would thus expect that the URLs for UIDs test@project24info and test@project24.info would clash, which will result in one of them being hidden by the other one, again resulting in what seems to be SOGo-generated data loss. Please consider no longer stripping the dots.

Additional Information

North American calendars with this problem can be downloaded from http://www.project24.info/holidays.php . Do not import them in the SOGo web frontend, as this (IMHO wrongly) replaces the UIDs with new random ones; instead, use a client which preserves UIDs (I used iCal).

I have sent a tcpdump/wireshark trace to Ludovic, if you need further details. It is not included here as it contains some passwords which are hard to remove from such pcap traces.

TagsNo tags attached.

Activities

ludovic

ludovic

2011-11-09 13:38

administrator   ~0002982

Since the URL handling behavior hasn't changed in SOGo for a little while, I suspect that what has changed is GNUstep behavior. We'll investigate.

Marcel

Marcel

2011-11-10 10:47

reporter   ~0002984

It does not necessarily need to be due to a change in SOGo/GNUstep URL handling. My uneducated guess would rather be that iOS5 takes some liberties (namely encoding @ as %40 in XML-TEXT contained URLs) that nobody else has dared to take until now. I also cannot judge whether this is within the limits set by the CalDAV standard or not.

ludovic

ludovic

2011-11-16 16:18

administrator   ~0003020

Please try these patches ASAP:

#

old_revision [77fbf2ff8bd28e88111c38f0575e8bc609e7b80f]

#

patch "sope-appserver/NGObjWeb/WebDAV/SoWebDAVRenderer.m"

from [37cc4ed634049395b70d337724895cb72ba12049]

to [8f4a5a2f02b5135c5b391a7c5b18d0ac2fc7ced8]

#

--- sope-appserver/NGObjWeb/WebDAV/SoWebDAVRenderer.m 37cc4ed634049395b70d337724895cb72ba12049
+++ sope-appserver/NGObjWeb/WebDAV/SoWebDAVRenderer.m 8f4a5a2f02b5135c5b391a7c5b18d0ac2fc7ced8
@@ -763,7 +763,7 @@ static BOOL useRelativeURLs = YE
Note: we always need to apply XML escaping (even though higher-level
characters might be already encoded)!
*/

  • [r appendContentXMLString:[href stringValue]];
  • [r appendContentXMLString:[[href stringValue] stringByEscapingURL]];
    [r appendContentString:@"</D:href>"];
    if (formatOutput) [r appendContentCharacter:'\n'];
    }

and:

#

old_revision [92cceaf860bbf56d983063471093858a3871ec23]

#

patch "Documentation/SOGo Native Microsoft Outlook Configuration.odt"

from [82ab4d20c872dd1867b7deca0aa73cd1d65f200a]

to [d720db599503853ecbdac227aa29b8a6018faa53]

patch "SoObjects/SOGo/SOGoGCSFolder.m"

from [00268a7356b2255bd9d593b990a0c053fbf3f7ee]

to [d86dad2c0172303b993ae4cacb2d6418a43f18ec]

#

--- SoObjects/SOGo/SOGoGCSFolder.m 00268a7356b2255bd9d593b990a0c053fbf3f7ee
+++ SoObjects/SOGo/SOGoGCSFolder.m d86dad2c0172303b993ae4cacb2d6418a43f18ec
@@ -1985,8 +1985,7 @@ static NSArray *childRecordFields = nil;
unsigned int count, max;

[r appendFormat: @"<D:response><D:href>"];

  • [r appendString: baseURL];
  • [r appendString: [object objectForKey: @"c_name"]];
  • [r appendString: [[NSString stringWithFormat: @"%@%@", baseURL, [object objectForKey: @"c_name"]] stringByEscapingURL]];
    [r appendString: @"</D:href>"];

    // NSLog (@"(appendPropstats...): %@", [NSDate date]);
    @@ -2005,7 +2004,7 @@ static NSArray childRecordFields = nil;
    toBuffer: (NSMutableString
    ) r
    {
    [r appendString: @"<D:response><D:href>"];

  • [r appendString: href];
  • [r appendString: [href stringByEscapingURL]];
    [r appendString: @"</D:href><D:status>HTTP/1.1 404 Not Found</D:status></D:response>"];
    }

@@ -2032,7 +2031,7 @@ static NSArray *childRecordFields = nil;
for (count = 0; count < max; count++)
{
element = [refs objectAtIndex: count];

  • currentURL = [[element firstChild] nodeValue];
  • currentURL = [[[element firstChild] nodeValue] stringByUnescapingURL];
    [urls addObject: currentURL];
    }
Marcel

Marcel

2011-11-17 11:03

reporter   ~0003029

Last edited: 2011-11-17 11:14

This does not work, iCal on MacOSX 10.6.8 complains about the server not having a calendar location (when adding this as new), iCal on iOS just ignores that calendar (when reactivating an existing configuration). This seems to be due to the fact that also the slashes have been URL-encoded and therefore no path can be found.

The patch seems to address the outgoing URLs; however, I would expect the incoming URLs (as seen from the server) would need to be URL-decoded, at least in part, to get things working.

*** Request:

PROPFIND /SOGo/dav/pop91555/ HTTP/1.1
Host: xen2.disy.inf.uni-konstanz.de
User-Agent: DAVKit/4.0.3 (732.2); CalendarStore/4.0.4 (997.7); iCal/4.0.4 (1395.7); Mac OS X/10.6.8 (10K549)
Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Depth: 0
Content-Type: text/xml
Content-Length: 474
Connection: close

<?xml version="1.0" encoding="utf-8"?>
<x0:propfind xmlns:x1="urn:ietf:params:xml:ns:caldav" xmlns:x0="DAV:" xmlns:x2="http://calendarserver.org/ns/&quot;>
<x0:prop>
<x0:principal-collection-set/>
<x1:calendar-home-set/>
<x1:calendar-user-address-set/>
<x1:schedule-inbox-URL/>
<x1:schedule-outbox-URL/>
<x2:dropbox-home-URL/>
<x2:xmpp-uri/>
<x2:notification-URL/>
<x0:displayname/>
<x0:principal-URL/>
<x0:supported-report-set/>
</x0:prop>
</x0:propfind>

*** Reply (XML formatted for readability; it does not seem to like the %2FSOGo%2Fdav%2Fpop91555%2F, expecting /SOGo/dav/pop91555/ instead):

HTTP/1.1 207 Multi-Status
Date: Thu, 17 Nov 2011 10:54:37 GMT
Server: SOPE 4.9.37/WebDAV
pragma: no-cache
cache-control: no-cache
ms-author-via: DAV
content-length: 1777
content-type: text/xml; charset="utf-8"
x-dav-error: 200 No error
dav: 1, 2, access-control, addressbook, calendar-access, calendar-schedule, calendar-auto-schedule, calendar-proxy
Connection: close

<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:a="urn:ietf:params:xml:ns:caldav" xmlns:b="http://calendarserver.org/ns/&quot; xmlns:D="DAV:">
<D:response>
<D:href>%2FSOGo%2Fdav%2Fpop91555%2F</D:href>
<D:propstat>
<D:status>HTTP/1.1 200 OK</D:status>
<D:prop>
<D:principal-collection-set xmlns:D="DAV:">
<D:href>/SOGo/dav/</D:href>
</D:principal-collection-set>
<a:calendar-home-set>
<D:href xmlns:D="DAV:">/SOGo/dav/pop91555/Calendar/</D:href>
</a:calendar-home-set>
<a:calendar-user-address-set>
<D:href xmlns:D="DAV:">mailto:Marcel.Waldvogel@uni-konstanz.de</D:href>
<D:href xmlns:D="DAV:">mailto:pop91555@uni-konstanz.de</D:href>
</a:calendar-user-address-set>
<a:schedule-inbox-URL>
<D:href xmlns:D="DAV:">/SOGo/dav/pop91555/Calendar/inbox/</D:href>
</a:schedule-inbox-URL>
<a:schedule-outbox-URL>
<D:href xmlns:D="DAV:">/SOGo/dav/pop91555/Calendar/personal/</D:href>
</a:schedule-outbox-URL>
<D:displayname>Marcel Waldvogel</D:displayname>
<D:principal-URL>
<D:href xmlns:D="DAV:">/SOGo/dav/pop91555/</D:href>
</D:principal-URL>
<D:supported-report-set xmlns:n1="urn:inverse:params:xml:ns:inverse-dav" xmlns:n2="urn:ietf:params:xml:ns:carddav" xmlns:D="DAV:">
<D:supported-report>
<D:report>
<n1:acl-query/>
</D:report>
</D:supported-report>
<D:supported-report>
<D:report>
<n1:collection-query/>
</D:report>
</D:supported-report>
<D:supported-report>
<D:report>
<n1:user-query/>
</D:report>
</D:supported-report>
<D:supported-report>
<D:report>
<D:expand-property/>
</D:report>
</D:supported-report>
<D:supported-report>
<D:report>
<n2:addressbook-query/>
</D:report>
</D:supported-report>
</D:supported-report-set>
</D:prop>
</D:propstat>
<D:propstat>
<D:status>HTTP/1.1 404 Resource Not Found</D:status>
<D:prop>
<b:dropbox-home-URL/>
<b:xmpp-uri/>
<b:notification-URL/>
</D:prop>
</D:propstat>
</D:response>
</D:multistatus>

ludovic

ludovic

2011-11-17 11:59

administrator   ~0003030

In my patch above, instead of doing:

[r appendFormat: @"<D:response><D:href>"];

  • [r appendString: baseURL];
  • [r appendString: [object objectForKey: @"c_name"]];
  • [r appendString: [[NSString stringWithFormat: @"%@%@", baseURL, [object objectForKey: @"c_name"]] stringByEscapingURL]];
    [r appendString: @"</D:href>"];

use:

[r appendFormat: @"<D:response><D:href>"];
[r appendString: baseURL];

  • [r appendString: [object objectForKey: @"c_name"]];
  • [r appendString: [[object objectForKey: @"c_name"] stringByEscapingURL]];
    [r appendString: @"</D:href>"];
Marcel

Marcel

2011-11-17 13:59

reporter   ~0003031

It is still encoding / to %2F, which iCal (likely not limited to them) does not like

ludovic

ludovic

2011-11-17 15:36

administrator   ~0003034

Ok, try those two patches:

#

old_revision [e38c570559610854b85c2c517053417779a76964]

#

patch "SoObjects/SOGo/SOGoGCSFolder.m"

from [00268a7356b2255bd9d593b990a0c053fbf3f7ee]

to [062164edd685f7699da1bfce2e10aba75177fa2d]

#

--- SoObjects/SOGo/SOGoGCSFolder.m 00268a7356b2255bd9d593b990a0c053fbf3f7ee
+++ SoObjects/SOGo/SOGoGCSFolder.m 062164edd685f7699da1bfce2e10aba75177fa2d
@@ -1986,7 +1986,7 @@ static NSArray *childRecordFields = nil;

[r appendFormat: @"<D:response><D:href>"];
[r appendString: baseURL];

  • [r appendString: [object objectForKey: @"c_name"]];
  • [r appendString: [[object objectForKey: @"c_name"] stringByEscapingURL]];
    [r appendString: @"</D:href>"];

    // NSLog (@"(appendPropstats...): %@", [NSDate date]);
    @@ -2032,7 +2032,7 @@ static NSArray *childRecordFields = nil;
    for (count = 0; count < max; count++)
    {
    element = [refs objectAtIndex: count];

  • currentURL = [[element firstChild] nodeValue];
  • currentURL = [[[element firstChild] nodeValue] stringByUnescapingURL];
    [urls addObject: currentURL];
    }

and:

#

old_revision [d48f86f9609dac535dc37b591151381ac34b8227]

#

patch "sope-appserver/NGObjWeb/WebDAV/SoWebDAVRenderer.m"

from [37cc4ed634049395b70d337724895cb72ba12049]

to [b7ce043ab2e0a03d701adaf38afe075f34dd822d]

#

--- sope-appserver/NGObjWeb/WebDAV/SoWebDAVRenderer.m 37cc4ed634049395b70d337724895cb72ba12049
+++ sope-appserver/NGObjWeb/WebDAV/SoWebDAVRenderer.m b7ce043ab2e0a03d701adaf38afe075f34dd822d
@@ -763,7 +763,13 @@ static BOOL useRelativeURLs = YE
Note: we always need to apply XML escaping (even though higher-level
characters might be already encoded)!
*/

  • [r appendContentXMLString:[href stringValue]];
  • [r appendContentXMLString: [NSString stringWithFormat: @"%@/%@",
  • [[href stringValue] stringByDeletingLastPathComponent],
  • [[[href stringValue] lastPathComponent] stringByEscapingURL]]];
  • if ([[href stringValue] hasSuffix: @"/"])
  • [r appendContentXMLString: @"/"];
  • [r appendContentString:@"</D:href>"];
    if (formatOutput) [r appendContentCharacter:'\n'];
    }

Marcel

Marcel

2011-11-17 15:49

reporter   ~0003035

Great! After clearing the cache on the iPhone (deactivating/reactivating the account in Preferences), all the @-containing UIDs appear. Thanks!

ludovic

ludovic

2011-11-17 16:22

administrator   ~0003036

Fix pushed in SOPE and SOGo.

Issue History

Date Modified Username Field Change
2011-11-02 15:04 Marcel New Issue
2011-11-09 13:34 ludovic Target Version => 1.3.10
2011-11-09 13:38 ludovic Note Added: 0002982
2011-11-10 10:47 Marcel Note Added: 0002984
2011-11-16 16:18 ludovic Note Added: 0003020
2011-11-17 11:03 Marcel Note Added: 0003029
2011-11-17 11:14 Marcel Note Edited: 0003029
2011-11-17 11:59 ludovic Note Added: 0003030
2011-11-17 13:59 Marcel Note Added: 0003031
2011-11-17 15:36 ludovic Note Added: 0003034
2011-11-17 15:49 Marcel Note Added: 0003035
2011-11-17 16:22 ludovic Note Added: 0003036
2011-11-17 16:22 ludovic Status new => resolved
2011-11-17 16:22 ludovic Fixed in Version => 1.3.10
2011-11-17 16:22 ludovic Resolution open => fixed
2011-11-17 16:22 ludovic Assigned To => ludovic
2011-11-17 16:22 ludovic Status resolved => closed