Relationship Graph
View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0001110 | SOGo Connector | with external server | public | 2011-01-31 16:04 | 2012-11-21 21:17 |
Reporter | fedel | Assigned To | |||
Priority | normal | Severity | feature | Reproducibility | always |
Status | closed | Resolution | fixed | ||
Product Version | 3.104 | ||||
Summary | 0001110: Various patches for synchronizing Thunderbird address books with iPhone, iPad etc. and DAViCal | ||||
Description | Thought I'd share the results of my Sogo Thunderbird connector debugging exercises. I am running the latest Sogo connector (3.104 nightly 1/30/2011) on Thunderbird 3.1.7 with a DAViCal server version 0.9.9.4 running on Debian. I am trying to synchronize my Thunderbird address books with all my iToys. With the original Sogo connector from http://inverse.ca/downloads/extensions/nightly/ this did NOT work at all. Calendar syncing worked fine, but address book sync was broken all over the place. After hours of debugging I can now synchronize TB3 with my iToys. The attached series of patches may be applied against the current Sogo connector nightly (http://inverse.ca/downloads/extensions/nightly/sogo-connector-3.104.xpi at the time of this submission). I'm assuming you know how to patch a Mozilla add-on. All patches are individually explained in the "Additional Information" field. | ||||
Additional Information | (1) sogo-connector-3.104-nightly-20110130-bugfix-sync-response.patch The Sogo connector is unable to receive any updates from compliant DAV servers (except Sogo server :-)) because the Sogo connector expects deprecated DAV response tokens. For more details, see: http://www.sogo.nu/bugs/bug_view_advanced_page.php?bug_id=857 This patch lets the Sogo connector accept "response" tags in addition to "sync-response", thus removes the need for setting "use_old_sync_response_tag = true" in DAViCal. (2) sogo-connector-3.104-nightly-20110130-ios-quirk-vcard-grouping.patch Apple uses an optional grouping feature defined in RFC2426, chapter 3.1.2. For example, iOS sends content lines prefixed with "item1." as in
Unfortunately, the current Sogo connector does not understand this syntax. It only tries matching parameter names against the entire first field until ; and ignores any VCARD records with unknown parameter names. Now, in real life this means that any contacts touched on an Apple toy will cause TB contacts without any addresses. Next time such a contact is modified in TB, this contact's address information will be gone forever. Sigh! This patch lets the Sogo connector ignore the group prefix and luckily import addresses with or without group prefix. (3) sogo-connector-3.104-nightly-20110130-ios-quirk-multiline-street.patch Apple seems to incorrectly encode multi-line address values using line-feeds (instead of using COMMAs as defined in RFC2426). This patch makes the Sogo connector accept such misformed VCARDs by translating line feeds within address fields into COMMAs. Please note that this patch eventually concatenates multi-line addresses into comma separated single-line address fields which is not perfect but still better than loosing precious address data. It would have been better to cleanly fix Apple's VCARD writer but that seems out of reach. (4) sogo-connector-3.104-nightly-20110130-bugfix-vcard-escape.patch The Sogo connector does not seem to escape values correctly when encoding VCARDs (see RFC2426). For example, COMMAs are escaped twice, some line breaks are not escaped at all (e.g. in notes), commas in values allowing for comma separated lists (e.g. in ADR, NICKNAME...) were still escaped and so on. This patch tries correcting the escaping logic in the Sogo connector. Instead of individually applying separate patches, you may also apply all patches at once by using sogo-connector-3.104-nightly-20110130-all.patches. | ||||
Tags | No tags attached. | ||||
related to | 0001098 | resolved | GroupDavSynchronizer: obsolete DAV:sync-response after DAV:sync-collection request? |
2011-01-31 16:04
|
|
2011-01-31 16:05
|
sogo-connector-3.104-nightly-20110130-bugfix-sync-response.patch (1,442 bytes)
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js --- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js Sun Jan 30 07:44:44 2011 +++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\sync.addressbook.groupdav.js Mon Jan 31 13:11:45 2011 @@ -814,6 +814,15 @@ this.newWebdavSyncToken = jsonResponse["multistatus"][0]["sync-token"][0]; let responses = jsonResponse["multistatus"][0]["sync-response"]; + /* WebDAV Sync Draft 3+ propose regular "DAV:response" instead of + "DAV:sync-response" in response to "DAV:sync-collection" */ + if (typeof responses == "undefined") { + responses = jsonResponse["multistatus"][0]["response"]; + for each (let response in responses) { + if (typeof response["status"] == "undefined") + response["status"] = ["HTTP/1.1 200 OK"]; + } + } for each (let response in responses) { let href = response["href"][0]; let keyArray = href.split("/"); |
2011-01-31 16:05
|
sogo-connector-3.104-nightly-20110130-bugfix-vcard-escape.patch (7,047 bytes)
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js --- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js Sun Jan 30 07:44:44 2011 +++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js Mon Jan 31 15:23:20 2011 @@ -38,14 +38,19 @@ jsInclude(["chrome://inverse-library/content/uuid.js", "chrome://inverse-library/content/quoted-printable.js"]); +/* Individually escape COMMA separated components of value list */ +function escapedForCardList(theString) { + let theList = theString.split(","); + for (let k in theList) + theList[k] = escapedForCard(theList[k]); + return theList.join(","); +} + function escapedForCard(theString) { theString = theString.replace(/\\/g, "\\\\"); theString = theString.replace(/,/g, "\\,"); theString = theString.replace(/;/g, "\\;"); - theString = theString.replace(/,/g, "\\,"); - - // theString.replace(/\n/g, "\\n,"); - // theString.replace(/\r/g, "\\r,"); + theString = theString.replace(/\n|\r\n/g, "\\n"); return theString; } @@ -72,17 +77,6 @@ return multiValue.split("\u001A"); } -function unescapedFromCard(theString) { - theString = theString.replace(/\\/g, "\\"); - theString = theString.replace(/\,/g, ","); - theString = theString.replace(/\;/g, ";"); - theString = theString.replace(/\,/g, ","); - // theString.replace(/\\n/g, "\n,"); - // theString.replace(/\\r/g, "\r,"); - - return theString; -} - /* this method parses a versit directory: - normalizing the charset and encodings; - returning the lines as hashes filled with the tag, parameters and values @@ -408,33 +405,7 @@ props["PreferMailFormat"] = value; }, categories: function(props, parameters, values) { - let commaValues = values[0]; - let newValues = []; - if (commaValues.length > 0) { - let escaped = false; - let currentValue = ""; - for (let i = 0; i < commaValues.length; i++) { - if (escaped) { - currentValue += commaValues[i]; - escaped = false; - } - else { - if (commaValues[i] == "\\") { - escaped = true; - } - else if (commaValues[i] == ",") { - newValues.push(currentValue); - currentValue = ""; - } - else { - currentValue += commaValues[i]; - } - } - } - newValues.push(currentValue); - } - props["Categories"] = newValues.join("\u001A"); - // props["Categories"] = arrayToMultiValue(values); + props["Categories"] = values[0].split(",").join("\u001A"); }, note: function(props, parameters, values) { props["Notes"] = values.join(";"); @@ -496,6 +467,11 @@ } }; + /* Remove address grouping, if any, for known tags */ + let tagparts = tag.split("."); + if ((tagparts.length == 2) && (typeof _insertCardMethods[tagparts[1]] != "undefined")) + tag = tagparts[1]; + if (typeof _insertCardMethods[tag] != "undefined") _insertCardMethods[tag](properties, parameters, values); else @@ -614,8 +590,8 @@ let lastName = card.getProperty("LastName", ""); let firstName = card.getProperty("FirstName", ""); if (lastName.length || firstName.length) - vCard += foldedLine("N:" + escapedForCard(lastName) - + ";" + escapedForCard(firstName)) + "\r\n"; + vCard += foldedLine("N:" + escapedForCardList(lastName) + + ";" + escapedForCardList(firstName)) + "\r\n"; let displayName = card.getProperty("DisplayName", ""); if (displayName.length) @@ -629,11 +605,11 @@ let nickName = card.getProperty("NickName", ""); if (nickName.length) - vCard += foldedLine("NICKNAME:" + escapedForCard(nickName)) + "\r\n"; + vCard += foldedLine("NICKNAME:" + escapedForCardList(nickName)) + "\r\n"; let categories = card.getProperty("Categories", ""); if (categories.length) - vCard += foldedLine("CATEGORIES:" + categories.split("\u001A").join(",")) + "\r\n"; + vCard += foldedLine("CATEGORIES:" + escapedForCardList(categories.split("\u001A").join(","))) + "\r\n"; let workAddress = card.getProperty("WorkAddress", ""); let workAddress2 = card.getProperty("WorkAddress2", ""); @@ -643,8 +619,9 @@ let workCountry = card.getProperty("WorkCountry", ""); if ((workAddress + workAddress2 + workCity + workState + workZipCode + workCountry).length) - vCard += foldedLine("ADR;TYPE=work:;" + escapedForCard(workAddress2) - + ";" + escapedForCard(workAddress) + vCard += foldedLine("ADR;TYPE=work:" + + ";" + escapedForCardList(workAddress2) + + ";" + escapedForCardList(workAddress) + ";" + escapedForCard(workCity) + ";" + escapedForCard(workState) + ";" + escapedForCard(workZipCode) @@ -658,8 +635,9 @@ let homeCountry = card.getProperty("HomeCountry", ""); if ((homeAddress + homeAddress2 + homeCity + homeState + homeZipCode + homeCountry).length) - vCard += foldedLine("ADR;TYPE=home:;" + escapedForCard(homeAddress2) - + ";" + escapedForCard(homeAddress) + vCard += foldedLine("ADR;TYPE=home:" + + ";" + escapedForCardList(homeAddress2) + + ";" + escapedForCardList(homeAddress) + ";" + escapedForCard(homeCity) + ";" + escapedForCard(homeState) + ";" + escapedForCard(homeZipCode) @@ -722,7 +700,7 @@ let jobTitle = card.getProperty("JobTitle", ""); if (jobTitle.length) - vCard += foldedLine("TITLE:" + jobTitle) + "\r\n"; + vCard += foldedLine("TITLE:" + escapedForCard(jobTitle)) + "\r\n"; let birthYear = card.getProperty("BirthYear", 0); let birthMonth = card.getProperty("BirthMonth", 0); @@ -742,7 +720,7 @@ let notes = card.getProperty("Notes", ""); if (notes.length) { vCard += foldedLine("NOTE:" - + escapedForCard(notes.replace(/\n/g, "\r\n"))) + "\r\n"; + + escapedForCard(notes)) + "\r\n"; } let aimScreenName = card.getProperty("_AimScreenName", ""); |
2011-01-31 16:05
|
sogo-connector-3.104-nightly-20110130-ios-quirk-multiline-street.patch (943 bytes)
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js --- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js Sun Jan 30 07:44:44 2011 +++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js Mon Jan 31 11:52:10 2011 @@ -344,6 +338,9 @@ }, adr: function(props, parameters, values) { let types = this._upperTypes(parameters["type"]); + /* Concat multi-line(feed) address field with commas (quirk for iOS) */ + values[1] = values[1].split("\n").join(", "); + values[2] = values[2].split("\n").join(", "); if (types.indexOf("WORK") > -1) { props.extend({ "WorkAddress2": values[1], "WorkAddress": values[2], |
2011-01-31 16:06
|
sogo-connector-3.104-nightly-20110130-ios-quirk-vcard-grouping.patch (858 bytes)
diff -Nur sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js --- sogo-connector-3.104-nightly-20110130\chrome\content\sogo-connector\general\vcards.utils.js Sun Jan 30 07:44:44 2011 +++ sogo-connector-3.104-nightly-20110130-patched\chrome\content\sogo-connector\general\vcards.utils.js Mon Jan 31 10:40:28 2011 @@ -496,6 +467,11 @@ } }; + /* Remove address grouping, if any, for known tags */ + let tagparts = tag.split("."); + if ((tagparts.length == 2) && (typeof _insertCardMethods[tagparts[1]] != "undefined")) + tag = tagparts[1]; + if (typeof _insertCardMethods[tag] != "undefined") _insertCardMethods[tag](properties, parameters, values); else |
I am not familiar with patching, but the patches are that one i need to use my iphone address book. |
|
Hi Tigono, Don't worry. I am currently having a look at this bugreport. Most of the code should be integrated in the next nightlies... |
|
Issues 2 and 4 above are fixed. Please try the nightlies. The issue with webdav sync will be fixed later. I need yet to double-check issue 3. Thanks for the patches! |
|
Further on webdav-sync: since it requires changes to the SOGo server, it's postponed as I want to do it cleanly. Thanks nonetheless. |
|
Issue 3 has been patched as well. So this bugreport is mostly fixed. |
|
I tested (2) and what I saw it is fixed, this helps a lot :-) But there is another new behavior in concern to Thunderbird (only using thunderbird, sogo connector and in this case apple address book server). => Changing some properties of the contact does not update the contact anymore! |
|
After making some additional tests I see that changing contacts is not possible anymore with Thunderbird3.19, SoGo Connector Nightly Build and Apple Addressbook Server. The Server gives the following error message: 2011-03-11 05:45:31+0100 [-] [carddav-8802] [-] [twistedcaldav.method.put_addressbook_common#error] Invalid vcard data: More than one VERSION property in component <Component: "<VCARD| [<VERSION{}3.0>, <VERSION{}3.0>, <PRODID{}-//Inverse inc.//SOGo Connector 1.0//EN>, <PRODID{}-//Inverse inc.//SOGo Connector 1.0//EN>, <UID{}C4B05D06-42E0-0001-A6CB-1900166319E7>, <ADR{u'TYPE': [u'work']}Ich weiss nicht 10\nStadt, 33055>, <ADR{u'TYPE': [u'home']}Das ist eine Strasse 44\n, >, <FN{}Dieter Wudel>, <N{} Dieter Wudel >, <X-MOZILLA-HTML{}FALSE>]>"> Please let me know if I should open a new bug? |
|
Hi Tigono, No it's fine. I'll have a look right away. Thanks for reporting this! |
|
Hi Tigono, That bug is now fixed. Please try again with the latest nightlies: |
|
Hi, the first error message is gone but the second remains -> [2011-03-12 07:30:22+0100 [-] [carddav-8801] [PooledMemCacheProtocol,client] [twistedcaldav.method.put_addressbook_common#error] Cannot overwrite vcard resource C4B0A621-5660-0001-B45B-66216B351A4C.vcf with different UID C4B0A621-56A0-0001-95D3-1E6F30406570 AND: It works like a charme in the pre4 version (but not with my iphone:-) I hope this has no influence: the thunderbird version with pre4 is on a windows pc and the test producing the error is on a mac (only for information) REMARK: |
|
From reading the comments, this bug seems fixed. |
|
Date Modified | Username | Field | Change |
---|---|---|---|
2011-01-31 16:04 | fedel | New Issue | |
2011-01-31 16:04 | fedel | File Added: sogo-connector-3.104-nightly-20110130-all.patches | |
2011-01-31 16:05 | fedel | File Added: sogo-connector-3.104-nightly-20110130-bugfix-sync-response.patch | |
2011-01-31 16:05 | fedel | File Added: sogo-connector-3.104-nightly-20110130-bugfix-vcard-escape.patch | |
2011-01-31 16:05 | fedel | File Added: sogo-connector-3.104-nightly-20110130-ios-quirk-multiline-street.patch | |
2011-01-31 16:06 | fedel | File Added: sogo-connector-3.104-nightly-20110130-ios-quirk-vcard-grouping.patch | |
2011-03-08 19:29 | tigono | Note Added: 0002223 | |
2011-03-08 20:16 |
|
Note Added: 0002227 | |
2011-03-09 20:18 |
|
Note Added: 0002239 | |
2011-03-09 20:19 |
|
Relationship added | related to 0001098 |
2011-03-09 20:20 |
|
Note Added: 0002240 | |
2011-03-09 20:51 |
|
Note Added: 0002241 | |
2011-03-10 07:41 | tigono | Note Added: 0002243 | |
2011-03-11 04:50 | tigono | Note Added: 0002248 | |
2011-03-11 19:12 |
|
Note Added: 0002251 | |
2011-03-11 20:29 |
|
Note Added: 0002256 | |
2011-03-12 06:45 | tigono | Note Added: 0002257 | |
2011-03-12 06:47 | tigono | Note Edited: 0002257 | |
2011-04-11 12:35 | tigono | Note Edited: 0002257 | |
2012-11-21 21:17 | ludovic | Note Added: 0004964 | |
2012-11-21 21:17 | ludovic | Status | new => closed |
2012-11-21 21:17 | ludovic | Resolution | open => fixed |