View Issue Details

IDProjectCategoryView StatusLast Update
0005463SOGoBackend Calendarpublic2023-01-30 20:26
Reporterworryboy Assigned To 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformDesktopOSWindowsOS Version10
Product Version5.4.0 
Summary0005463: Carddav Server Response is not UTF8 complient
Description

i have several issues with a strict VCARD implementations using SOGo as CardDav Server.

The from SOGo return VCard must be UTF-8 but was Latin-1. https://datatracker.ietf.org/doc/html/rfc6350#section-3.1

Any suggestion to solve this issue.

Steps To Reproduce

Reproduce with an curl to the CardDav Server :

curl --user "carddavtestuser@SOGo.Domain:SecretPWD" -s -X PROPFIND -H "Content-Type: application/xml"  -sD /dev/stderr https://SOGo.Domain/SOGo/dav/carddavtestuser@SOGo.Domain/Contacts/personal/

Return looks like UTF-8 but is Latin-1

Additional Information

none

TagsCardDAV, utf-8

Activities

worryboy

worryboy

2022-01-19 19:44

reporter   ~0015786

Hi Francis ,
sorry for my inconvenience . Mantis convert Latin 1 text. Please delete both comments i recently made and leave this one.

I attached a screen describing this issue.

Frank

example.issue.jpg (317,855 bytes)
raw.text.as.example.txt (1,573 bytes)   
Example : a german 'ü' is encoded as 252 but must be c3bc (50108).

<D:displayname>Meißner Üösülämi</D:displayname>

Is

<D:displayname>Mei&#223;ner &#220;&#246;s&#252;l&#228;mi</D:displayname>

Return

HTTP/1.1 207 Multi-Status

Date: Mon, 10 Jan 2022 16:30:27 GMT
Server: SOPE 4.9.37/WebDAV
Referrer-Policy: same-origin
Content-Type: text/xml; charset="utf-8"
X-Dav-Error: 200 No error
Ms-Author-Via: DAV
Pragma: no-cache
Content-Length: 3548
Cache-Control: no-cache

<?xml version="1.0" encoding="utf-8"?>

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:ap="http://apache.org/dav/props/">
   <D:response>
      <D:href>/SOGo/dav/carddavtestuser@SOGo.Domain/Contacts/personal/183358-6133BB80-3-7158B900.vcf</D:href>
      <D:propstat>
         <D:status>HTTP/1.1 200 OK</D:status>
         <D:prop>
            <D:creationdate>Sat, 04 Sep 2021 20:32:42 +0100</D:creationdate>
            <D:getcontentlength>168</D:getcontentlength>
            <D:getlastmodified>Sat, 04 Sep 2021 21:34:07 +0100</D:getlastmodified>
            <D:getetag>&quot;gcs00000001&quot;</D:getetag>
            <D:resourcetype />
            <D:getcontenttype>text/x-vcard</D:getcontenttype>
            <D:displayname>Mei&#223;ner &#220;&#246;s&#252;l&#228;mi</D:displayname>
            <D:href>/SOGo/dav/carddavtestuser@SOGo.Domain/Contacts/personal/183358-6133BB80-3-7158B900.vcf</D:href>
            <ap:executable>0</ap:executable>
         </D:prop>
      </D:propstat>
   </D:response>
</D:multistatus>


raw.text.as.example.txt (1,573 bytes)   
francis

francis

2022-01-19 20:49

administrator   ~0015791

Have you set AddDefaultCharset UTF-8 in your &lt;Proxy> directive?

worryboy

worryboy

2022-01-20 07:59

reporter   ~0015799

sory for the format issues again.

conf attached

SOGo.conf.txt (2,364 bytes)   
Alias /SOGo.woa/WebServerResources/ \
      /usr/lib/GNUstep/SOGo/WebServerResources/
Alias /SOGo/WebServerResources/ \
      /usr/lib/GNUstep/SOGo/WebServerResources/

<Directory /usr/lib/GNUstep/SOGo/>
    AllowOverride None

    <IfVersion < 2.4>
        Order deny,allow
        Allow from all
    </IfVersion>
    <IfVersion >= 2.4>
        Require all granted
    </IfVersion>

    # Explicitly allow caching of static content to avoid browser specific behavior.
    # A resource's URL MUST change in order to have the client load the new version.
    <IfModule expires_module>
      ExpiresActive On
      ExpiresDefault "access plus 1 year"
    </IfModule>
</Directory>

ProxyRequests Off
SetEnv proxy-nokeepalive 1
ProxyPreserveHost On

# Enable to use Microsoft ActiveSync support
# Note that you MUST have many sogod workers to use ActiveSync.
# See the SOGo Installation and Configuration guide for more details.
#
ProxyPass /Microsoft-Server-ActiveSync \
 http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync \
 retry=60 connectiontimeout=5 timeout=360

ProxyPass /SOGo http://127.0.0.1:20000/SOGo retry=0

<Proxy http://127.0.0.1:20000/SOGo>

  ## adjust the following to your configuration
  ## and do not forget to enable the headers module
  <IfModule headers_module>
    RequestHeader set "x-webobjects-server-port" "443"
    RequestHeader set "x-webobjects-server-name" "%{HTTP_HOST}e" env=HTTP_HOST
    RequestHeader set "x-webobjects-server-url" "https://%{HTTP_HOST}e" env=HTTP_HOST

    ## When using proxy-side autentication, you need to uncomment and
    ## adjust the following line:
    RequestHeader unset "x-webobjects-remote-user"
    #RequestHeader set "x-webobjects-remote-user" "%{REMOTE_USER}e" env=REMOTE_USER

    RequestHeader set "x-webobjects-server-protocol" "HTTP/1.0"
  </IfModule>

  AddDefaultCharset UTF-8

  <IfVersion < 2.4>
      Order deny,allow
      Allow from all
  </IfVersion>
  <IfVersion >= 2.4>
      Require all granted
  </IfVersion>
</Proxy>

# For Apple autoconfiguration
<IfModule rewrite_module>
  RewriteEngine On
  RewriteRule ^/.well-known/caldav/?$ /SOGo/dav [R=301]
  RewriteRule ^/.well-known/carddav/?$ /SOGo/dav [R=301]
</IfModule>

Redirect /.well-known/caldav  /SOGo/dav/
Redirect /.well-known/carddav /SOGo/dav/
SOGo.conf.txt (2,364 bytes)   
worryboy

worryboy

2022-01-22 15:01

reporter   ~0015810

Any other suggestion ? (since AddDefaultCharset UTF-8 in <Proxy> directive is configured)

DB ?

francis

francis

2022-01-24 20:03

administrator   ~0015813

Can you verify if the VCARD as exported with sogo-tool backup is properly encoded?

worryboy

worryboy

2022-01-25 15:51

reporter   ~0015815

Last edited: 2022-01-25 15:58

it looks a little bit strange

 cat carddavtestuser@SOGo.Domain | grep Mei
N:Mei\U00DFner;\U00DC\U00F6s\U00FCl\U00E4mi
FN:Mei\U00DFner \U00DC\U00F6s\U00FCl\U00E4mi

a "ü" is encoded as "\U00FC" whis seems also Latin1 https://www.fileformat.info/info/unicode/char/00fc/index.htm

it looks that is wrong stored .. setup issue from me ?

worryboy

worryboy

2022-01-25 19:05

reporter   ~0015816

Last edited: 2022-01-25 20:44

one update more .. table layout looks good

SELECT default_character_set_name FROM information_schema.SCHEMATA WHERE schema_name = &quot;sogo&quot;; 

DEFAULT_CHARACTER_SET_NAME
utf8

SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,
       information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
  AND T.table_schema = &quot;sogo&quot;
  AND T.table_name = &quot;sogocarddavt0014ede51d5&quot;;

CHARACTER_SET_NAME
utf8

SELECT character_set_name FROM information_schema.`COLUMNS` 
WHERE table_schema = &quot;sogo&quot;
  AND table_name = &quot;sogocarddavt0014ede51d5&quot;
  AND column_name = &quot;c_content&quot;;

CHARACTER_SET_NAME
utf8

SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,
       information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
  AND T.table_schema = &quot;sogo&quot;
  AND T.table_name = &quot;sogocarddavt0014ede51d5_quick&quot;;

CHARACTER_SET_NAME
utf8

SELECT character_set_name FROM information_schema.`COLUMNS` 
WHERE table_schema = &quot;sogo&quot;
  AND table_name = &quot;sogocarddavt0014ede51d5_quick&quot;
  AND column_name = &quot;c_givenname&quot;;

CHARACTER_SET_NAME
utf8
francis

francis

2022-01-26 20:15

administrator   ~0015822

From my understanding, it is perfectly valid to send numeric entity references in the DAV response. The client should handle those encoded entities the same way it would handle & or &lt;.

worryboy

worryboy

2022-01-27 09:18

reporter   ~0015825

Last edited: 2022-01-31 14:58

sumarized

Encoding Note char ü
UTF-8 "Bytes" <B>Needed alphanumeric response</B> from SOGo CardDav response. The most widely used encoding for Unicode characters. Used in some operating systems (GNU/Linux, Unix) and sometimes in various internet services (e-mail, web). Characters are encoded in different lengths from 1 to 4 bytes. C3BC
UTF-16 or Unicode "HEX" <B>alphanumeric stored on SOGo DB</B> Characters are encoded in either 2 or 4 bytes. See also isthisthingon-unicode and fileformat-unicode 00FC
UTF-16 or Unicode "DEC" <B>numeric response SOGo CarDav</B> Characters are encoded in either 2 or 4 bytes. See also isthisthingon-unicode and fileformat-unicode 252

rfc3629
https://datatracker.ietf.org/doc/html/rfc6350#section-3.1 expect the first row the UTF 8 byte sequence or ???
indeed the position on the UTF code page is as well position 252 and SOGo translate it from UTF16

if i understand you correctly, the SOGo response defined on header and body as UTF8 must interpret & # 252 as UTF8 Unicode Dec to ü ?

francis

francis

2022-01-27 13:21

administrator   ~0015827

SOPE uses decimal HTML entities to represent extended characters (ö => &0000246;) in XML-based response.

worryboy

worryboy

2022-01-27 18:56

reporter   ~0015828

understood , get back to you after a chat with developers from AVM

worryboy

worryboy

2022-01-31 07:43

reporter   ~0015836

AVM ticket reference https://avm.zendesk.com/agent/tickets/4482503 is still waiting for a reply.

worryboy

worryboy

2023-01-30 20:26

reporter   ~0016626

avm ticket closed . AVM make possible changes on AVM OS 7.50 . please close the ticket

Issue History

Date Modified Username Field Change
2022-01-14 09:17 worryboy New Issue
2022-01-14 09:17 worryboy Tag Attached: CardDAV
2022-01-14 09:17 worryboy Tag Attached: utf-8
2022-01-19 18:58 francis Steps to Reproduce Updated
2022-01-19 19:44 worryboy Note Added: 0015786
2022-01-19 19:44 worryboy File Added: example.issue.jpg
2022-01-19 19:44 worryboy File Added: raw.text.as.example.txt
2022-01-19 20:29 francis Steps to Reproduce Updated
2022-01-19 20:49 francis Note Added: 0015791
2022-01-20 07:59 worryboy Note Added: 0015799
2022-01-20 07:59 worryboy File Added: SOGo.conf.txt
2022-01-22 15:01 worryboy Note Added: 0015810
2022-01-24 20:03 francis Note Added: 0015813
2022-01-25 15:51 worryboy Note Added: 0015815
2022-01-25 15:58 francis Note Edited: 0015815
2022-01-25 19:05 worryboy Note Added: 0015816
2022-01-25 20:44 francis Note Edited: 0015816
2022-01-26 20:12 francis Steps to Reproduce Updated
2022-01-26 20:15 francis Note Added: 0015822
2022-01-27 09:18 worryboy Note Added: 0015825
2022-01-27 13:21 francis Note Added: 0015827
2022-01-27 18:56 worryboy Note Added: 0015828
2022-01-31 07:43 worryboy Note Added: 0015836
2022-01-31 14:58 francis Note Edited: 0015825
2023-01-30 20:26 worryboy Note Added: 0016626