Proper Ceph setup on PVE

Having Ceph storage in homelab gives a lot of benefits, however setting this up properly is not that obvious. Minimal requirements on Ceph webpage (https://docs.ceph.com/en/reef/start/hardware-recommendations/) are suggesting enterprise NVMe, but this is not mandatory. Well – IT IS. I tried approach with consumer grade NVMe – but with that Ceph reports daily slow operations in bluestore (my setup assumes main storage on HDD and DB/WAL on NVMe). Transition to enterprise-grade NVMes resolved all issues with my setup.