View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0005416 | SOGo | Backend Calendar | public | 2021-10-26 07:12 | 2023-01-31 13:44 |
Reporter | landry | Assigned To | sebastien | ||
Priority | normal | Severity | major | Reproducibility | always |
Status | closed | Resolution | fixed | ||
Product Version | 5.2.0 | ||||
Fixed in Version | 5.8.0 | ||||
Summary | 0005416: sogo mysteriously adds XML-encoded UTF-8 BOM to calendar-data & getetag REPORT replies | ||||
Description | using sogo with thunderbird clients to access calendars, previously running 5.0.1 on OpenBSD 6.9. frontend nginx 1.18, backend postgresql 12.6, database with UTF8 encoding and C locale. upgraded the complete stack to OpenBSD 7.0, with SOGo 5.2.0, nginx 1.20, postgresql 12.8 - suddenly the events saved by thunderbird cant be displayed/parsed back by thunderbird, which loudly complaints (filed https://bugzilla.mozilla.org/show_bug.cgi?id=1737067 with details about thunderbird) Thunderbird complains that it cant find the BEGIN:VCALENDAR item in the returned ICS file:
Dug a bit more with tcpdump/wireshark to look at what was sent back by sogo after PUT'ing an event, and the content mysteriously contains XML-encoded UTF-8 BOM character (eg ) for the calendar-data and getetag XML tags:
afaict In the psql database, the event c_content field doesnt seem to contain the UTF-8 char...
although i'm not 100% sure as printing only the first char of the field prints nothing (unsure if substr starts at pos 0 or 1..) This is ... mystifying. The same thunderbird instance can display events saved in davical, so it's not a thunderbird issue. Keeping OpenBSD 7.0, i've tried reverting to SOGo 5.0.1, no improvement. To nginx 1.18, no improvement. I'm looking for help figuring out where in the sogo code the REPORT replies are generated, i've perused the sogo changelog between 5.0.1 and 5.2.0 and didnt find anything vaguely related to this. | ||||
Tags | CalDAV | ||||
of course the strange XML-encoded BOM is stripped by mantis HTML parseer, but i'm attaching the resulting XML showing that the opening tags calendar-data getetag start by & followed by #65279; mget.ics (1,159 bytes)
<?xml version="1.0" encoding="utf-8"?> <D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:response><D:href>/SOGo/dav/breuil/Calendar/personal/fc0baba2-1a35-4f51-a4b8-990e04f68b47.ics</D:href><D:propstat><D:prop><C:calendar-data>BEGIN:VCALENDAR PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN VERSION:2.0 BEGIN:VTIMEZONE TZID:Europe/Paris BEGIN:DAYLIGHT TZOFFSETFROM:+0100 TZOFFSETTO:+0200 TZNAME:CEST DTSTART:19700329T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU END:DAYLIGHT BEGIN:STANDARD TZOFFSETFROM:+0200 TZOFFSETTO:+0100 TZNAME:CET DTSTART:19701025T030000 RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT CREATED:20160908T082555Z LAST-MODIFIED:20160908T082611Z DTSTAMP:20160908T082611Z UID:fc0baba2-1a35-4f51-a4b8-990e04f68b47 SUMMARY:Appel PIGMA georchestra DTSTART;TZID=Europe/Paris:20160909T100000 DTEND;TZID=Europe/Paris:20160909T110000 TRANSP:OPAQUE CLASS:PUBLIC END:VEVENT END:VCALENDAR</C:calendar-data><D:getetag>"gcs00000000"</D:getetag></D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus> |
|
to be 100% clear, with a space added to workaround the parser 'eating' the xml-encoded char, here's the edited content i get on the wire: ... |
|
after hours of debugging, found out the culprit: gnustep-base had got upgraded from 1.27.0 to 1.28.0, and if i revert this gnustep-base upgrade (and build sogo/sope 5.2.0 on top) i have a working setup, no more crazy & #65279; hidden char in the caldav transaction. now to peruse the gnustep-base changelog to figure out what broke it.. |
|
filed https://github.com/gnustep/libs-base/issues/212 at gnustep-base repo. |
|
after reverting https://github.com/gnustep/libs-base/commit/bd5f2909e6edc8012a0a6e44ea1402dfbe1353a4 on top of gnustep 1.28.0, i have a working sogo 5.2.0. So that's a sogo regression from this commit, dunno if sogo/sope should adapt to this behaviour. |
|
I've been bitten by this, too, with the same update history. Dear team sogo, is the change in gnustep/libs-base wrong? sogo.err.emclient.txt (5,326 bytes)
11:50:00 merma@phnx [CalDAV / CardDAV] MailClient.Accounts.ConnectionException: The format of value '"gcs00000001"' is invalid. ---> System.FormatException: The format of value '"gcs00000001"' is invalid. at System.Net.Http.Headers.HttpHeaderParser.ParseValue(String value, Object storeValue, Int32& index) at System.Net.Http.Headers.HttpHeaders.ParseAndAddValue(HeaderDescriptor descriptor, HeaderStoreItemInfo info, String value) at System.Net.Http.Headers.HttpHeaders.Add(HeaderDescriptor descriptor, String value) at System.Net.Http.Headers.HttpHeaders.Add(String name, String value) at MailClient.Protocols.CalDav.ProtocolCommands.PutCommand.Prepare(HttpRequestMessage request) at MailClient.Protocols.CalDav.ProtocolCommands.Connector.RunCommand(CalDavAccount account, ICommand command, CancellationToken cancellationToken) --- End of inner exception stack trace --- at MailClient.Protocols.CalDav.ProtocolCommands.Connector.RunCommand(CalDavAccount account, ICommand command, CancellationToken cancellationToken) at MailClient.Protocols.CalDav.Synchronizer.UploadCommand`1.DoUpload(Uri selectPath, TStorageItem item, Boolean overwrite) at MailClient.Protocols.CalDav.Synchronizer.UploadCommand`1.Execute(WorkerStatus status) at MailClient.Commands.Command.Process(WorkerStatus status) EnqueuedStackTrace = at MailClient.Storage.Application.Item`2.EndUpdate() at MailClient.Storage.Application.Schedule.ScheduleItem`1.EndUpdate() at MailClient.UI.Forms.formEvent.SaveDataToEventWithDescription(String innerHtmlText, String innerDocumentHtml, Action`1 onSuccess, Boolean saveToFolder, EventItem& evnt) at MailClient.UI.Forms.formEvent.<>c__DisplayClass62_1.<SaveDataToEvent>b__1(String innerDocumentHtml) at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebBrowserEx.<>c__DisplayClass209_0.<GetCleanedDocumentHtml>b__0(CefProcessMessage message) at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebBrowserEx.<>c__DisplayClass231_0.<EvaluateJavaScript>b__0(CefProcessMessage message, CefFrame f) at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebBrowserEx.OnMessageReceived(CefProcessMessage message) at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebClientEx.OnProcessMessageReceived(CefBrowser browser, CefProcessId sourceProcess, CefProcessMessage message) at Xilium.CefGlue.CefClient.on_process_message_received(cef_client_t* self, cef_browser_t* browser, CefProcessId source_process, cef_process_message_t* message) at Interop.User32.DispatchMessageW(MSG& msg) at System.Windows.Forms.Application.ComponentManager.Interop.Mso.IMsoComponentManager.FPushMessageLoop(UIntPtr dwComponentID, msoloop uReason, Void* pvLoopData) at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(msoloop reason, ApplicationContext context) at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(msoloop reason, ApplicationContext context) at System.Windows.Forms.Application.Run(ApplicationContext context) at MailClient.Program.<>c.<Main>b__169_1() at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Delegate.DynamicInvokeImpl(Object[] args) at System.Delegate.DynamicInvoke(Object[] args) at System.Windows.Forms.Control.InvokeMarshaledCallbackDo(ThreadMethodEntry tme) at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Windows.Forms.Control.InvokeMarshaledCallback(ThreadMethodEntry tme) at System.Windows.Forms.Control.InvokeMarshaledCallbacks() at System.Windows.Forms.Control.WndProc(Message& m) at System.Windows.Forms.ScrollableControl.WndProc(Message& m) at System.Windows.Forms.ContainerControl.WndProc(Message& m) at System.Windows.Forms.Form.WndProc(Message& m) at MailClient.Common.UI.Forms.BaseForm.WndProc(Message& m) at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m) at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m) at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, WM msg, IntPtr wparam, IntPtr lparam) at Xilium.CefGlue.Interop.libcef.run_message_loop() at MailClient.Program.Main(String[] args) GUIStatus_Exception_Reported = True sogo.cadaver-dav.ics.txt (996 bytes)
$ cat 53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0.ics Displaying `/SOGo/dav/merma/Calendar/personal/53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0.ics' : BEGIN:VCALENDAR PRODID:DAVx5/4.0-ose ical4j/3.1.0 (org.lineageos.etar) VERSION:2.0 BEGIN:VTIMEZONE TZID:Europe/Vienna LAST-MODIFIED:20201010T011803Z BEGIN:DAYLIGHT TZNAME:CEST TZOFFSETFROM:+0100 TZOFFSETTO:+0200 DTSTART:19810329T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU END:DAYLIGHT BEGIN:STANDARD TZNAME:CET TZOFFSETFROM:+0200 TZOFFSETTO:+0100 DTSTART:19961027T030000 RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT DTSTAMP:20211028T091255Z UID:53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0 SUMMARY:Test22 LOCATION:Test22 DESCRIPTION:Test22 DTSTART;TZID=Europe/Vienna:20211030T070000 DTEND;TZID=Europe/Vienna:20211030T080000 STATUS:CONFIRMED BEGIN:VALARM TRIGGER:-PT1H ACTION:DISPLAY DESCRIPTION:Test2 END:VALARM CLASS:PUBLIC END:VEVENT END:VCALENDARdav:/SOGo/dav/merma/Calendar/personal/> sogo.err.caldavsynchronizer.txt (1,817 bytes)
ERROR 2021-10-28 12:06:35,997 [ 24] onizer.Implementation.CalDavRepository`1 . TryDeserializeCalendar - Could not deserialize ICalData of '/SOGo/dav/merma/Calendar/personal/53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0.ics' . DEBUG 2021-10-28 12:06:35,999 [ 24] onizer.Implementation.CalDavRepository`1 . TryDeserializeCalendar - ICalData: ?BEGIN:VCALENDAR PRODID:DAVx5/4.0-ose ical4j/3.1.0 (org.lineageos.etar) VERSION:2.0 BEGIN:VTIMEZONE TZID:Europe/Vienna LAST-MODIFIED:20201010T011803Z BEGIN:DAYLIGHT TZNAME:CEST TZOFFSETFROM:+0100 TZOFFSETTO:+0200 DTSTART:19810329T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU END:DAYLIGHT BEGIN:STANDARD TZNAME:CET TZOFFSETFROM:+0200 TZOFFSETTO:+0100 DTSTART:19961027T030000 RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT DTSTAMP:20211028T091255Z UID:53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0 SUMMARY:Test22 LOCATION:Test22 DESCRIPTION:Test22 DTSTART;TZID=Europe/Vienna:20211030T070000 DTEND;TZID=Europe/Vienna:20211030T080000 STATUS:CONFIRMED BEGIN:VALARM TRIGGER:-PT1H ACTION:DISPLAY DESCRIPTION:Test2 END:VALARM CLASS:PUBLIC END:VEVENT END:VCALENDAR System.ArgumentOutOfRangeException: Der Index lag au�erhalb des Bereichs. Er darf nicht negativ und kleiner als die Sammlung sein. Parametername: index bei System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource) bei CalDavSynchronizer.Implementation.CalDavRepository`1.DeserializeCalendar(Str ing iCalData, IStringSerializer calendarSerializer) bei CalDavSynchronizer.Implementation.CalDavRepository`1.TryDeserializeCalendar( String iCalData, IICalendar& calendar, WebResourceName uriOfCalendarForLogging, IStringSerializer calendarSerializer, ILoadEntityLogger logger) |
|
I've upgraded a Ubuntu mail server runing Sogo from 20.04 to 22.04 and I can confirm that Thunderbird could not display any newly fetchted calendar events. Ubuntu 22.04 is providing gnustep-base package 1.28.0. |
|
I have debugged this a bit and found the cause for this. here is a fix: https://github.com/Alinto/sogo/pull/324 |
|
Date Modified | Username | Field | Change |
---|---|---|---|
2021-10-26 07:12 | landry | New Issue | |
2021-10-26 07:12 | landry | Tag Attached: CalDAV | |
2021-10-26 07:26 | landry | Note Added: 0015581 | |
2021-10-26 07:26 | landry | File Added: mget.ics | |
2021-10-26 08:15 | landry | Note Added: 0015582 | |
2021-10-26 13:40 | landry | Note Added: 0015583 | |
2021-10-26 14:06 | landry | Note Added: 0015584 | |
2021-10-26 14:50 | landry | Note Added: 0015585 | |
2021-10-28 13:21 | mcmer | Note Added: 0015591 | |
2021-10-28 13:21 | mcmer | File Added: sogo.err.emclient.txt | |
2021-10-28 13:21 | mcmer | File Added: sogo.cadaver-dav.ics.txt | |
2021-10-28 13:21 | mcmer | File Added: sogo.err.caldavsynchronizer.txt | |
2022-08-01 07:48 | lsalle@aventin.com | Note Added: 0016147 | |
2022-08-01 11:40 | francis | Note Edited: 0016147 | |
2022-08-01 11:42 | francis | Description Updated | |
2022-08-15 14:35 | the_nic | Note Added: 0016167 | |
2023-01-31 13:43 | sebastien | Assigned To | => sebastien |
2023-01-31 13:43 | sebastien | Status | new => resolved |
2023-01-31 13:43 | sebastien | Resolution | open => fixed |
2023-01-31 13:43 | sebastien | Fixed in Version | => 5.8.0 |
2023-01-31 13:44 | sebastien | Status | resolved => closed |