microcloud status will show MicroCeph not found when having a lots os OSDs

[yoshikado]

I have deployed a MicroCloud cluster on 14 nodes, and each having 10 OSD disks.
I have all initialized and deployed the cluster without any issues, but when I do microcloud status, I get WARNING MicroCeph is not found on xxx.
Following is the full output, but the nodes that says not found is always random when I re-run the command.

$ sudo microcloud status

 Status: WARNING

 ┃ ! MicroCeph is not found on micro-a6, micro-b5, micro-a7, micro-b6, micro-b7, micro-b1, micro-b4, micro-a3, micro-a2, micro-b3, micro-a1

┌─────────────┬────────────────┬──────┬─────────────────┬────────────────────────┬────────┐
│    Name     │    Address     │ OSDs │ MicroCeph Units │     MicroOVN Units     │ Status │
├─────────────┼────────────────┼──────┼─────────────────┼────────────────────────┼────────┤
│ micro-a1    │ 172.20.1.11    │  -   │        -        │ central,chassis,switch │ ONLINE │
│ micro-a2    │ 172.20.1.12    │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-a3    │ 172.20.1.13    │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-a4    │ 172.20.1.14    │  10  │        -        │     chassis,switch     │ ONLINE │
│ micro-a5    │ 172.20.1.15    │  10  │        -        │     chassis,switch     │ ONLINE │
│ micro-a6    │ 172.20.1.16    │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-a7    │ 172.20.1.17    │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-b1    │ 172.20.1.111   │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-b2    │ 172.20.1.112   │  10  │   mds,mgr,mon   │ central,chassis,switch │ ONLINE │
│ micro-b3    │ 172.20.1.113   │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-b4    │ 172.20.1.114   │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-b5    │ 172.20.1.115   │  -   │        -        │     chassis,switch     │ ONLINE │
│ micro-b6    │ 172.20.1.116   │  -   │        -        │ central,chassis,switch │ ONLINE │
│ micro-b7    │ 172.20.1.117   │  -   │        -        │     chassis,switch     │ ONLINE │
└─────────────┴────────────────┴──────┴─────────────────┴────────────────────────┴────────┘

I have checked the status for each services, lxd, microceph, microovn, but those looked all ok. And the only error I could find was from microcloud snap.

ubuntu@micro-a2:~$ sudo snap logs microcloud
2026-04-03T07:47:30Z systemd[1]: Started snap.microcloud.daemon.service - Service for snap application microcloud.daemon.
2026-04-03T07:47:30Z microcloud.daemon[12780]: time="2026-04-03T07:47:30Z" level=warning msg="Microcluster database is uninitialized"
2026-04-03T09:31:21Z microcloud.daemon[12780]: time="2026-04-03T09:31:21Z" level=error msg="Failed to get service status" name=micro-a2 type=MicroCeph

One thing to note is that, when I started the deployment with just 1 OSD in each server, the microcloud status output was ok. I have only tested once, but after having 5 OSDs in each server, then this WARNING message came out from micrlcoud status.

[roosterfish]

Thanks for reporting, sounds like a bug to me.

Do you see the Failed to get service status error on all members? And is it a reoccurring error?

[yoshikado]

I do see this in all of the members, but only when I run microcloud status.

[roosterfish]

Ok. As long as microceph cluster list reports ONLINE for all members of the cluster (you can also check microceph status), it's likely a bug in the status code itself.

[yoshikado]

Here is the output from microceph cluster list and also confirmed that microcloud service list will generate the output fine as well.

$ sudo microceph cluster list
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
|       NAME        |       ADDRESS       |   ROLE   |                           FINGERPRINT                            | STATUS |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-a1          | 172.20.1.11:7443    | voter    | 99663aa85dab012a2be73779edc7b8c54e252a090f283eaa0030eedbbaa11651 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-a2          | 172.20.1.12:7443    | spare    | a8c33aeea0e3f4f26c0b266d63777fa3566ea991628038dfd1b4da2d1db9e619 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-a3          | 172.20.1.13:7443    | stand-by | 7c7f3ade09b8e534e7d3cfe71215a2b922ec74995367661ba01370100e53d196 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-a4          | 172.20.1.14:7443    | spare    | fc1bf5b646a9d2f615161eb59dc16a5401646712ebb702aa54ed0a81b0788028 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-a5          | 172.20.1.15:7443    | stand-by | cafaa28641a2455f5042837dd5be7048a81d9c12a42ee43cc6f8c27dcd52e6b2 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-a6          | 172.20.1.16:7443    | spare    | 4425fb902ae0e7425e8f5f118d83864bccf1f6e821dc1d240455d7ee1da0a5a6 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-a7          | 172.20.1.17:7443    | spare    | 20c83ca83d14bda905b480ccff2e77ceec455bac9c29421e163f41ea0509321e | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-b1          | 172.20.1.111:7443   | spare    | ec8bfc186234ea03dd4102d7ada58dd0e7318ff6f5e872c1c16b64932319d1c3 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-b2          | 172.20.1.112:7443   | spare    | 65db2d84e97b21d2beb1eb308aaca2c79b52bf47ab9cde35ba55f3fda497a78d | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-b3          | 172.20.1.113:7443   | spare    | 0c876091776c5baed70c7e4ddd39e08506c7bcb98b9dc066ab2147bbc5c63b17 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-b4          | 172.20.1.114:7443   | voter    | 54d242281d129ba7299842461ce69443ea9f536829b1d9526d60e7dd3545ef0c | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-b5          | 172.20.1.115:7443   | stand-by | 969cec96ee95d25454f4b7991bfd88d780a193ebd053be255dd4e3c49c1a0450 | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-b6          | 172.20.1.116:7443   | spare    | afe1a4ea350129de74da710fcff4bbbc847333d184ce3b3efbca106c205cd12b | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+
| micro-b7          | 172.20.1.117:7443   | spare    | 6562e32ff79b8d88baf9dc95a830670dc92bba20f17d1f1333dbb7970bdd14ae | ONLINE |
+-------------------+---------------------+----------+------------------------------------------------------------------+--------+

$ sudo microcloud service list
MicroOVN:
┌───────────────────┬─────────────────────┬──────────┬────────┐
│       NAME        │       ADDRESS       │   ROLE   │ STATUS │
├───────────────────┼─────────────────────┼──────────┼────────┤
│    micro-a1       │ 172 .20.1.11:6443   │  voter   │ ONLINE │
│    micro-a2       │ 172 .20.1.12:6443   │  spare   │ ONLINE │
│    micro-a3       │ 172 .20.1.13:6443   │  spare   │ ONLINE │
│    micro-a4       │ 172 .20.1.14:6443   │  spare   │ ONLINE │
│    micro-a5       │ 172 .20.1.15:6443   │ stand-by │ ONLINE │
│    micro-a6       │ 172 .20.1.16:6443   │  spare   │ ONLINE │
│    micro-a7       │ 172 .20.1.17:6443   │  spare   │ ONLINE │
│    micro-b1       │ 172 .20.1.111:6443  │ stand-by │ ONLINE │
│    micro-b2       │ 172 .20.1.112:6443  │  voter   │ ONLINE │
│    micro-b3       │ 172 .20.1.113:6443  │ stand-by │ ONLINE │
│    micro-b4       │ 172 .20.1.114:6443  │  spare   │ ONLINE │
│    micro-b5       │ 172 .20.1.115:6443  │  spare   │ ONLINE │
│    micro-b6       │ 172 .20.1.116:6443  │  voter   │ ONLINE │
│    micro-b7       │ 172 .20.1.117:6443  │  spare   │ ONLINE │
└───────────────────┴─────────────────────┴──────────┴────────┘
LXD:
┌───────────────────┬─────────────────────────────┬──────────────────┬────────┐
│       NAME        │           ADDRESS           │       ROLE       │ STATUS │
├───────────────────┼─────────────────────────────┼──────────────────┼────────┤
│    micro-a1       │ https://172.20.1.11:8443    │   ovn-chassis    │ Online │
│                   │                             │ database-leader  │        │
│                   │                             │     database     │        │
│    micro-a2       │ https://172.20.1.12:8443    │   ovn-chassis    │ Online │
│    micro-a3       │ https://172.20.1.13:8443    │   ovn-chassis    │ Online │
│    micro-a4       │ https://172.20.1.14:8443    │   ovn-chassis    │ Online │
│    micro-a5       │ https://172.20.1.15:8443    │   ovn-chassis    │ Online │
│    micro-a6       │ https://172.20.1.16:8443    │   ovn-chassis    │ Online │
│    micro-a7       │ https://172.20.1.17:8443    │   ovn-chassis    │ Online │
│    micro-b1       │ https://172.20.1.111:8443   │   ovn-chassis    │ Online │
│                   │                             │     database     │        │
│    micro-b2       │ https://172.20.1.112:8443   │   ovn-chassis    │ Online │
│    micro-b3       │ https://172.20.1.113:8443   │   ovn-chassis    │ Online │
│    micro-b4       │ https://172.20.1.114:8443   │   ovn-chassis    │ Online │
│                   │                             │ database-standby │        │
│    micro-b5       │ https://172.20.1.115:8443   │   ovn-chassis    │ Online │
│                   │                             │     database     │        │
│    micro-b6       │ https://172.20.1.116:8443   │   ovn-chassis    │ Online │
│    micro-b7       │ https://172.20.1.117:8443   │   ovn-chassis    │ Online │
│                   │                             │ database-standby │        │
└───────────────────┴─────────────────────────────┴──────────────────┴────────┘
MicroCeph:
┌───────────────────┬─────────────────────┬──────────┬────────┐
│       NAME        │       ADDRESS       │   ROLE   │ STATUS │
├───────────────────┼─────────────────────┼──────────┼────────┤
│    micro-a1       │ 172 .20.1.11:7443   │  voter   │ ONLINE │
│    micro-a2       │ 172 .20.1.12:7443   │  spare   │ ONLINE │
│    micro-a3       │ 172 .20.1.13:7443   │ stand-by │ ONLINE │
│    micro-a4       │ 172 .20.1.14:7443   │  spare   │ ONLINE │
│    micro-a5       │ 172 .20.1.15:7443   │ stand-by │ ONLINE │
│    micro-a6       │ 172 .20.1.16:7443   │  spare   │ ONLINE │
│    micro-a7       │ 172 .20.1.17:7443   │  spare   │ ONLINE │
│    micro-b1       │ 172 .20.1.111:7443  │ stand-by │ ONLINE │
│    micro-b2       │ 172 .20.1.112:7443  │  spare   │ ONLINE │
│    micro-b3       │ 172 .20.1.113:7443  │  spare   │ ONLINE │
│    micro-b4       │ 172 .20.1.114:7443  │  voter   │ ONLINE │
│    micro-b5       │ 172 .20.1.115:7443  │  voter   │ ONLINE │
│    micro-b6       │ 172 .20.1.116:7443  │  spare   │ ONLINE │
│    micro-b7       │ 172 .20.1.117:7443  │  spare   │ ONLINE │
└───────────────────┴─────────────────────┴──────────┴────────┘
MicroCloud:
┌───────────────────┬─────────────────────┬──────────┬────────┐
│       NAME        │       ADDRESS       │   ROLE   │ STATUS │
├───────────────────┼─────────────────────┼──────────┼────────┤
│    micro-a1       │ 172 .20.1.11:9443   │  voter   │ ONLINE │
│    micro-a2       │ 172 .20.1.12:9443   │  voter   │ ONLINE │
│    micro-a3       │ 172 .20.1.13:9443   │ stand-by │ ONLINE │
│    micro-a4       │ 172 .20.1.14:9443   │  spare   │ ONLINE │
│    micro-a5       │ 172 .20.1.15:9443   │  spare   │ ONLINE │
│    micro-a6       │ 172 .20.1.16:9443   │  spare   │ ONLINE │
│    micro-a7       │ 172 .20.1.17:9443   │  spare   │ ONLINE │
│    micro-b1       │ 172 .20.1.111:9443  │ stand-by │ ONLINE │
│    micro-b2       │ 172 .20.1.112:9443  │  spare   │ ONLINE │
│    micro-b3       │ 172 .20.1.113:9443  │  spare   │ ONLINE │
│    micro-b4       │ 172 .20.1.114:9443  │ stand-by │ ONLINE │
│    micro-b5       │ 172 .20.1.115:9443  │  spare   │ ONLINE │
│    micro-b6       │ 172 .20.1.116:9443  │  spare   │ ONLINE │
│    micro-b7       │ 172 .20.1.117:9443  │  spare   │ ONLINE │
└───────────────────┴─────────────────────┴──────────┴────────┘

[yoshikado]

And here is the output from microceph status

$ sudo microceph status
MicroCeph deployment summary:
- micro-a1 (172.20.1.11)
  Services: mds, mgr, mon, osd
  Disks: 10
- micro-a2 (172.20.1.12)
  Services: osd
  Disks: 10
- micro-a3 (172.20.1.13)
  Services: osd
  Disks: 10
- micro-a4 (172.20.1.14)
  Services: osd
  Disks: 10
- micro-a5 (172.20.1.15)
  Services: osd
  Disks: 10
- micro-a6 (172.20.1.16)
  Services: osd
  Disks: 10
- micro-a7 (172.20.1.17)
  Services: osd
  Disks: 10
- micro-b1 (172.20.1.111)
  Services: mds, mgr, mon, osd
  Disks: 10
- micro-b2 (172.20.1.112)
  Services: osd
  Disks: 10
- micro-b3 (172.20.1.113)
  Services: osd
  Disks: 10
- micro-b4 (172.20.1.114)
  Services: osd
  Disks: 10
- micro-b5 (172.20.1.115)
  Services: osd
  Disks: 10
- micro-b6 (172.20.1.116)
  Services: osd
  Disks: 10
- micro-b7 (172.20.1.117)
  Services: mds, mgr, mon, osd
  Disks: 10

[roosterfish]

And here is the output from microceph status

Great to see this. So the MicroCeph status seems to be ok.
Likely microcloud status falsely formats the status output here.

[roosterfish]

@yoshikado I had some time to get back to this. Is microcloud status returning immediately, or did it take a while to get the response? Unfortunately the Failed to get service status log message doesn't include the observed error but I wonder if a context deadline is reached. But we could rule that out if microcloud status returns immediately.

[roosterfish]

I have added #1497. Would you be able to test this in your environment Using 3/edge once its merged? Or is this a prod setup?