View Issue Details

IDProjectCategoryView StatusLast Update
0004308SOGoBackend Mailpublic2017-10-11 13:36
ReporterAndrey Cherepanov Assigned Tofrancis  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformServerOSALT LinuxOS Version8
Product Version3.2.10 
Fixed in Version4.0.0 
Summary0004308: Unable to create mail folder "???"
Description

I configure Dovecot to work with UTF-8 in mail_location.
I set
SOGoSieveFolderEncoding = "UTF-8";
in /etc/sogo/sogo.conf

But I have problems with create and view mail folder with Cyrillic letters with third letter '?' in name. Any other Cyrillic letter as third letter works without problem.

Try to create mail folder: An error occured while creating the mailbox "???".
Created in Thunderbird folder "???" show in Web UI as "???"

Steps To Reproduce

In Mail view try to create folder with name "???".

Dialog is appeared: An error occured while creating the mailbox "???".

TagsNo tags attached.

Relationships

related to 0002318 closed Problem with cyrillic imap folders 
has duplicate 0004299 resolvedfrancis "POST /SOGo/so/example@example.eu/Mail/0/folderINBOX/createFolder HTTP/1.1" 500 74/25 0.271 - - 0 

Activities

Andrey Cherepanov

Andrey Cherepanov

2017-10-06 15:52

reporter   ~0012356

echo "0L/QsNC/Cg==" | base64 -d

Ivan Zakharyaschev

Ivan Zakharyaschev

2017-10-06 17:50

reporter   ~0012357

Third letters which cause this problem:

U+043F CYRILLIC SMALL LETTER PE
(UTF-8: 0xD0 0xBF)

U+047F CYRILLIC SMALL LETTER OT
(UTF-8: 0xD1 0xBF)

U+04BF CYRILLIC SMALL LETTER ABKHASIAN CHE WITH DESCENDER
(UTF-8: 0xD2 0xBF)

If such a folder (three Cyrillic letters, the third one a "special" one) is created in the filesystem directly, it is displayed by SOGo with the third letter replaced with:

U+04FF CYRILLIC SMALL LETTER HA WITH STROKE
(UTF-8: 0xD3 0xBF)

Ivan Zakharyaschev

Ivan Zakharyaschev

2017-10-06 17:52

reporter   ~0012358

This could also be observed when a Russian folder "Sent" ("????????????") was created in the filesystem directly (after an import). Then its 3rd letter was exactly a bad one, so it was displayed incorrectly in SOGo.

Ivan Zakharyaschev

Ivan Zakharyaschev

2017-10-06 17:59

reporter   ~0012359

Either 0xbf or the last 2 bits set (3 == 11B) in UCS seem to be the trigger.

Ivan Zakharyaschev

Ivan Zakharyaschev

2017-10-06 18:35

reporter   ~0012360

Every third Cyrillic letter is affected: I've just created (in the filesystem) a folder with a very long name consisting solely of Cyrillic small PE, and this is how it is shown in SOGo web-interface:

*__*__**__...

  • denotes the bad sign here, _ -- the good letter.

I wonder whether other interfaces have this problem. (What other interaces does SOGo server except for the web-interface?.. Well, probably, only ActiveSync. IMAP is done directly through dovecot, and dovecot treats them fine, was confirmed by an experiment)

Ivan Zakharyaschev

Ivan Zakharyaschev

2017-10-10 11:29

reporter  

sope-correct-mUTF7-last-digit.patch (3,781 bytes)   
From 45db3976d3f4790337e3e8bfb149f053e88174f9 Mon Sep 17 00:00:00 2001
From: Ivan Zakharyaschev <imz@altlinux.org>
Date: Mon, 9 Oct 2017 18:15:35 +0300
Subject: [PATCH] _{encodeTo,decodeOf}ModifiedUTF7(): 64th digit of the base is
 "," (RFC3501)

According to https://tools.ietf.org/html/rfc3501#section-5.1.3 ,
in comparison to Base64 encoding, "," should be used instead of "/"
as the greatest digit of the base.

Of course, this meant that the encodings used by an IMAP server (e.g.,
dovecot) and SOGo did not match when they were communicating with each
other.

For example, when SOGo attempts to create a folder whose encoding
includes the base64 "/" digit, it is not accepted as valid mUTF7 by
the IMAP server:

C[0x5649de0d5d30]: 5 LIST "" ""
S[0x5649de0f3640]: * LIST (\Noselect) "/" ""
S[0x5649de0f3640]: 5 OK List completed (0.001 + 0.040 + 0.039 secs).
C[0x5649de0d5d30]: 6 create "INBOX/&BD8EMAQ/BDA-"
S[0x5649de0f3640]: 6 NO Mailbox name is not valid mUTF-7 (0.001 + 0.000 secs).

In the other direction, when such a folder is listed by the IMAP
server, it appears with a ",", for example: "&BD8EMAQ/BDA-". And this
is not correctly decoded by SOGo due to the fact that "," is not known
to SOGo as a digit in this base and is mapped to -1 by the internal
table, which results in a wrong Unicode code.

An example of a Unicode string which would trigger this bug is a
string with U+043F CYRILLIC SMALL LETTER PE as the third letter:

$ printf '\x04\x3f\x04\x30\x04\x3f' | recode ucs-2..
$ printf '\x04\x3f\x04\x30\x04\x3f' | base64
BD8EMAQ/

Three two-byte Unicode (Cyrillic) characters correspond to eight 6-bit
base64 digits, the last one being "/":

U+043F                U+0430                 U+043F
0000 01|00 0011 1111  00|00 0100 0011 00|00  0000 0100 00|11 1111
      B|      D|       8|      E|      M|       A|      Q|      /|

This bug has been reported in:

https://sogo.nu/bugs/view.php?id=4308
https://bugzilla.altlinux.org/show_bug.cgi?id=32426
https://bugzilla.altlinux.org/show_bug.cgi?id=33722

and a part of the problem described in
https://sogo.nu/bugs/view.php?id=2318 must be caused by incorrect
mUTF7 encoding, too (and hence must have not been fixed completely
before):

And I see next messages when I try to open Cyrillic folder:
C[0x7fb2e23a77a0]: 5 select "&BBQEPgRBBEIEQwT/BEs-"
S[0x7fb2e21fb690]: 5 NO Invalid mailbox name: &BBQEPgRBBEIEQwT/BEs-

As can you see sogo sends wrong folder name:
received "&BBQEPgRBBEIEQwQ,BEs-"
sent "&BBQEPgRBBEIEQwT/BEs-"
---
 sope-mime/NGImap4/NSString+Imap4.m | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sope-mime/NGImap4/NSString+Imap4.m b/sope-mime/NGImap4/NSString+Imap4.m
index 76f5486..b0788cc 100644
--- a/sope-mime/NGImap4/NSString+Imap4.m
+++ b/sope-mime/NGImap4/NSString+Imap4.m
@@ -279,7 +279,7 @@ static void _encodeToModifiedUTF7(unsigned char *_buf, int encLen,
 /* check metamail output for correctness */
 
 static unsigned char basis_64[] =
-   "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+   "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+,";
 
 static void writeChunk(int c1, int c2, int c3, int pads, unsigned char **result_,
                        unsigned int *cntRes_) {
@@ -318,7 +318,7 @@ static void writeChunk(int c1, int c2, int c3, int pads, unsigned char **result_
 static char index_64[128] = {
     -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1,
     -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1,
-    -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,62, -1,-1,-1,63,
+    -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,62, 63,-1,-1,-1,
     52,53,54,55, 56,57,58,59, 60,61,-1,-1, -1,-1,-1,-1,
     -1, 0, 1, 2,  3, 4, 5, 6,  7, 8, 9,10, 11,12,13,14,
     15,16,17,18, 19,20,21,22, 23,24,25,-1, -1,-1,-1,-1,
-- 
2.7.4

Ivan Zakharyaschev

Ivan Zakharyaschev

2017-10-10 11:33

reporter   ~0012364

The patch has been submitted as a pull request at github.

The same problem must also be a part of https://sogo.nu/bugs/view.php?id=2318 .

Perhaps, it was left unnoticed because in practice it appears only in the rare case when there is a non-Latin two-byte Unicode character which has all last 6 bits "1" at the third position in a folder name.

francis

francis

2017-10-10 12:35

administrator   ~0012365

See https://github.com/inverse-inc/sope/commit/45db3976d3f4790337e3e8bfb149f053e88174f9

Issue History

Date Modified Username Field Change
2017-10-06 15:50 Andrey Cherepanov New Issue
2017-10-06 15:52 Andrey Cherepanov Note Added: 0012356
2017-10-06 17:50 Ivan Zakharyaschev Note Added: 0012357
2017-10-06 17:52 Ivan Zakharyaschev Note Added: 0012358
2017-10-06 17:59 Ivan Zakharyaschev Note Added: 0012359
2017-10-06 18:35 Ivan Zakharyaschev Note Added: 0012360
2017-10-10 11:29 Ivan Zakharyaschev File Added: sope-correct-mUTF7-last-digit.patch
2017-10-10 11:33 Ivan Zakharyaschev Note Added: 0012364
2017-10-10 12:35 francis Note Added: 0012365
2017-10-10 12:35 francis Status new => resolved
2017-10-10 12:35 francis Fixed in Version => 4.0.0
2017-10-10 12:35 francis Resolution open => fixed
2017-10-10 12:35 francis Assigned To => francis
2017-10-10 12:47 francis Relationship added related to 0002318
2017-10-11 13:36 francis Relationship added has duplicate 0004299