If you have a default reminder set, let's say 5 minutes before, then for all-day events you will get an alert at 23:55 the previous night. This is probably rarely, if ever, a user's expectation. I would usually turn off the reminder for an all-day event, but it's possible to forget to do this, and the failure mode potentially involves ringing an alarm late at night if you have your calendar synced to your phone.

For comparison, if you create an all-day event in Google calendar it automatically changes the reminder setting. I believe it applies the default reminder setting to 17:00 the previous day, so if your default reminder is 5 minutes before then you'll get the reminder at 16:55 one day before.