可以看到42个数据节点中有11个数据节点处于 down 状态；
三、故障分析及解决：3.1、数据检查：分别去down掉的节点中去查看数据文件（此处我们只取一个节点进行展示对比）：
[gpadmin@data02 gpseg16]$ pwd/greenplum/gpdata/primary/gpseg16[gpadmin@data02 gpseg16]$ lsbasepg_hba.confpg_serialpg_utilitymodedtmredofts_probe_file.bakpg_ident.confpg_snapshotsPG_VERSIONglobalpg_logpg_statpg_xloginternal.auto.confpg_logicalpg_stat_tmppostgresql.auto.confpg_clogpg_multixactpg_subtranspostgresql.confpg_distributedlogpg_notifypg_tblspcpostmaster.optspg_dynshmempg_replslotpg_twophase可以发现都缺少了 postmaster.pid 文件 。
为了看的更清楚，我们找一个状态正常的节点查看对比：
[gpadmin@data01 gpseg1]$ pwd/greenplum/gpdata/primary/gpseg1[gpadmin@data01 gpseg1]$ lsbasepg_hba.confpg_serialpg_utilitymodedtmredofts_probe_file.bakpg_ident.confpg_snapshotsPG_VERSIONglobalpg_logpg_statpg_xloginternal.auto.confpg_logicalpg_stat_tmppostgresql.auto.confpg_clogpg_multixactpg_subtranspostgresql.confpg_distributedlogpg_notifypg_tblspcpostmaster.optspg_dynshmempg_replslotpg_twophasepostmaster.pid[gpadmin@data01 gpseg1]$ cat postmaster.pid20517/greenplum/gpdata/primary/gpseg1165202270555001/tmp* 55001001393219说明：很多人说此时重启集群可以轻易的解决这个问题，但实际上重启集群并不能保证一定会解决问题，并且重启集群会导致业务中断 。
3.2、在线生成一个配置文件：[gpadmin@master ~]$ gprecoverseg -o ./recover20221127:22:10:22:020909 gprecoverseg:master:gpadmin-[INFO]:-Starting gprecoverseg with args: -o ./recover20221127:22:10:22:020909 gprecoverseg:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'20221127:22:10:22:020909 gprecoverseg:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'20221127:22:10:22:020909 gprecoverseg:master:gpadmin-[INFO]:-Obtaining Segment details from master...20221127:22:10:22:020909 gprecoverseg:master:gpadmin-[INFO]:-Configuration file output to ./recover successfully.
[gpadmin@master ~]$ lsgpAdminLogsrecover[gpadmin@master ~]$ more recoverdata04|55000|/greenplum/gpdata/primary/gpseg9data04|55001|/greenplum/gpdata/primary/gpseg10data04|55002|/greenplum/gpdata/primary/gpseg11data05|55000|/greenplum/gpdata/primary/gpseg12data05|55001|/greenplum/gpdata/primary/gpseg13data05|55002|/greenplum/gpdata/primary/gpseg14data01|55003|/greenplum/gpdata/primary/gpseg15data02|55003|/greenplum/gpdata/primary/gpseg16data03|55003|/greenplum/gpdata/primary/gpseg17data04|55003|/greenplum/gpdata/primary/gpseg18data05|55003|/greenplum/gpdata/primary/gpseg193.3、通过生成的配置文件进行修复集群：[gpadmin@master ~]$ gprecoverseg -i ./recover -F 执行过程省略，但有个选项需要确认：Continue with segment recovery procedure Yy|Nn (default=N):> y3.4、检查数据：[gpadmin@master ~]$ psql -c "select * from gp_segment_configuration order by content asc,dbid;" dbid | content | role | preferred_role | mode | status | port|hostname|address|datadir------+---------+------+----------------+------+--------+-------+--------------+--------------+-----------------------------------44 |-1 | p| p| s| u|5432 |master|master| /greenplum/gpdata/master/gpseg-145 |-1 | m| m| s| u|5432 |standby|standby| /greenplum/gpdata/master/gpseg-12 |0 | p| p| s| u| 55000 |data01|data01| /greenplum/gpdata/primary/gpseg011 |0 | m| m| s| u| 56000 |data02|data02| /greenplum/gpdata/mirror/gpseg03 |1 | p| p| s| u| 55001 |data01|data01| /greenplum/gpdata/primary/gpseg112 |1 | m| m| s| u| 56001 |data02|data02| /greenplum/gpdata/mirror/gpseg14 |2 | p| p| s| u| 55002 |data01|data01| /greenplum/gpdata/primary/gpseg213 |2 | m| m| s| u| 56002 |data02|data02| /greenplum/gpdata/mirror/gpseg25 |3 | p| p| s| u| 55000 |data02|data02| /greenplum/gpdata/primary/gpseg314 |3 | m| m| s| u| 56000 |data03|data03| /greenplum/gpdata/mirror/gpseg36 |4 | p| p| s| u| 55001 |data02|data02| /greenplum/gpdata/primary/gpseg415 |4 | m| m| s| u| 56001 |data03|data03| /greenplum/gpdata/mirror/gpseg47 |5 | p| p| s| u| 55002 |data02|data02| /greenplum/gpdata/primary/gpseg516 |5 | m| m| s| u| 56002 |data03|data03| /greenplum/gpdata/mirror/gpseg58 |6 | p| p| s| u| 55000 |data03|data03| /greenplum/gpdata/primary/gpseg617 |6 | m| m| s| u| 56000 |data01|data01| /greenplum/gpdata/mirror/gpseg69 |7 | p| p| s| u| 55001 |data03|data03| /greenplum/gpdata/primary/gpseg718 |7 | m| m| s| u| 56001 |data01|data01| /greenplum/gpdata/mirror/gpseg710 |8 | p| p| s| u| 55002 |data03|data03| /greenplum/gpdata/primary/gpseg819 |8 | m| m| s| u| 56002 |data01|data01| /greenplum/gpdata/mirror/gpseg821 |9 | m| p| s| u| 55000 |data04|data04| /greenplum/gpdata/primary/gpseg930 |9 | p| m| s| u| 56000 |data05|data05| /greenplum/gpdata/mirror/gpseg922 |10 | m| p| s| u| 55001 |data04|data04| /greenplum/gpdata/primary/gpseg1031 |10 | p| m| s| u| 56001 |data05|data05| /greenplum/gpdata/mirror/gpseg1023 |11 | m| p| s| u| 55002 |data04|data04| /greenplum/gpdata/primary/gpseg1132 |11 | p| m| s| u| 56002 |data05|data05| /greenplum/gpdata/mirror/gpseg1124 |12 | m| p| s| u| 55000 |data05|data05| /greenplum/gpdata/primary/gpseg1227 |12 | p| m| s| u| 56000 |data04|data04| /greenplum/gpdata/mirror/gpseg1225 |13 | m| p| s| u| 55001 |data05|data05| /greenplum/gpdata/primary/gpseg1328 |13 | p| m| s| u| 56001 |data04|data04| /greenplum/gpdata/mirror/gpseg1326 |14 | m| p| s| u| 55002 |data05|data05| /greenplum/gpdata/primary/gpseg1429 |14 | p| m| s| u| 56002 |data04|data04| /greenplum/gpdata/mirror/gpseg1433 |15 | m| p| s| u| 55003 |data01|data01| /greenplum/gpdata/primary/gpseg1539 |15 | p| m| s| u| 56003 |data02|data02| /greenplum/gpdata/mirror/gpseg1534 |16 | m| p| s| u| 55003 |data02|data02| /greenplum/gpdata/primary/gpseg1640 |16 | p| m| s| u| 56003 |data03|data03| /greenplum/gpdata/mirror/gpseg1635 |17 | m| p| s| u| 55003 |data03|data03| /greenplum/gpdata/primary/gpseg1741 |17 | p| m| s| u| 56003 |data04|data04| /greenplum/gpdata/mirror/gpseg1736 |18 | m| p| s| u| 55003 |data04|data04| /greenplum/gpdata/primary/gpseg1842 |18 | p| m| s| u| 56003 |data05|data05| /greenplum/gpdata/mirror/gpseg1837 |19 | m| p| s| u| 55003 |data05|data05| /greenplum/gpdata/primary/gpseg1938 |19 | p| m| s| u| 56003 |data01|data01| /greenplum/gpdata/mirror/gpseg19(42 rows)