View Issue Details

IDProjectCategoryView StatusLast Update
0006014SOGoBackend Address Bookpublic2024-08-29 11:19
Reporterelnone Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Platform[Server] LinuxOSUbuntuOS Version16.04 LTS
Product Version5.9.1 
Summary0006014: CardDAV client and for each contact DML SELECT to DB
Description

Hello,

The issue case is related to Global Address Book and CardDAV client.
The scenario has an effect where database or LDAP address book includes starting with 10.000 users/contacts. Postgres database is used to hold contacts. In test env table gal_view.

So, here is a configuration in sogo.conf

SOGoUserSources = (
    {
        id = "gal";
        displayName = "Corporate Address Book";
        type = sql;
        viewURL = postgresql://sogo:password@127.0.0.1:5432/post/gal;
        canAuthenticate = NO;
        isAddressBook = YES;
        MailFieldNames = (
            "mail",
            "aliases"
        );
        SearchFieldNames = (
            "c_cn",
            "mail",
            "aliases",
            "title",
            "ou",
            "o",
            "st",
            "l"
        );
    },

The client Thunderbird is configured with CardDAV URL: https://post.domain.com/SOGo/dav/cody.zuniga@domain.com/Contacts/gal/
On synchronization of address book, the mail client sends HTTP request to SOGo:

  1. 18:34:57 -> "PROPFIND /SOGo/dav/craig.zuniga@domain.com/Contacts/gal/ HTTP/1.1\r\n"
  2. 18:42:28 <- "HTTP/1.1 207 Multi-Status\r\n" (see 2.txt)
    • in between this step 1 and 2 it takes 8 minutes, sogod fork process makes 30.000 (in our contacts within) SELECT requests to the database:
      ...
      SELECT FROM gal WHERE c_uid = 'tabitha.vasquez@domain.com'
      SELECT
      FROM gal WHERE c_uid = 'tracy.rivas@domain.com'
      SELECT FROM gal WHERE c_uid = 'frances.vasquez@domain.com'
      SELECT
      FROM gal WHERE c_uid = 'christina.vasquez@domain.com'
      SELECT * FROM gal WHERE c_uid = 'andrea.vasquez@domain.com'
      ...
  3. 18:42:30 -> "REPORT /SOGo/dav/craig.zuniga@domain.com/Contacts/gal/ HTTP/1.1\r\n"
  4. 18:51.27 <- "HTTP/1.1 207 Multi-Status\r\n" (see 4.txt)
  • and again between 3 and 4 steps it takes about 9 minutes to select delta up to 30.000 times of SELECT.

So as result if SOGo service is behind some load balancer, like HAProxy, there must be set timeout of 30 and more minutes for frontend and backend to remain the TCP session.
Alongside there is WOWatchDog timeout setting of 60 minutes which kills the sogod fork process, while it makes SELECTs...

Also there was a test with 250.000 contacts. After one hour the WatchDog killed the process...
Aug 16 12:55:01 sogod [26300]: [WARN] <0x0x5b8c2c70d500[WOWatchDogChild]> pid 26511 has been hanging in the same request for 59 minutes
Aug 16 12:56:01 sogod [26300]: [WARN] <0x0x5b8c2c70d500[WOWatchDogChild]> safety belt -- sending KILL signal to pid 26511

By default Thunderbird has 30 minutes contacts synchronization interval. So, if there are 10.000 users and more, the SOGo, like to say, DDoS'es the database with thousands of SELECT DML.

Steps To Reproduce

To reproduce, we need a configured database with data. Attached you can find a file with fake users to populate a database table "gal".
Configure Thunderbird client to use address book https://post.domain.com/SOGo/dav/cody.zuniga@domain.com/Contacts/gal/ and make sync.
See from logs of postgres /var/lib/postgresql/11/main/pg_log/postgresql-*.log, and log of sogo.log
Also it is seen via "tcpdump -iany port 5432 or port 20000" on the SOGo server

TagsCardDAV, sql

Activities

elnone

elnone

2024-08-29 11:19

reporter  

gal.7z.001 (5,242,880 bytes)
gal.7z.002 (3,375,489 bytes)
Archive.zip (1,867,733 bytes)

Issue History

Date Modified Username Field Change
2024-08-29 11:19 elnone New Issue
2024-08-29 11:19 elnone Tag Attached: CardDAV
2024-08-29 11:19 elnone Tag Attached: sql
2024-08-29 11:19 elnone File Added: gal.7z.001
2024-08-29 11:19 elnone File Added: gal.7z.002
2024-08-29 11:19 elnone File Added: Archive.zip
2024-08-29 11:19 elnone File Added: Screenshot_20240829_141755.bmp