Upgrade to 14.2, can't boot

is the server capable of booting efi ? or is too old for that ?
if it is just enable efi and boot from the stick. see if you can see the pool at the loader prompt .
 
also you can try to boot from a 15-CURRENT image. see if you can read the pool at the loader prompt. if it works you may try to install bootcode from -current.
 
Check your Bios->HBA settings, it's limiting your disk to 2TB in Legacy mode.

edit:
If you have a full backup and you are going to risk, you can try to remove the SWAP partitions create a new ESP partition on each disk, then create again the SWAP partition and switch from BIOS to EFI.

It will look like this:
1 EFI (300MB)
2 freebsd-swap (3.9G)
3 freebsd-zfs (3.6T) NO CHANGE
Click to expand...
C

ZFS Thread 'ZFS Boot Issue'

I have a FreeBSD production system with 12 3TB SATA drives in a a JBoD triple-parity ZFS configuration spanning around 30TB. This was working well for months, through numerous reboots, however after a power outage Friday it's been unable to boot, giving 3 or more (it varies) `zio_read error: 97` entries, followed by or with interspersed `ZFS: i/o error - all block copies unavailable` in what appears to be every fourth entry (e.g. 3 error 97 entries then 1 all block copies unavailable entry, repeating 0 or more times.) The subsequent message is usually `ZFS: can not read MOS config`...
  • CoryG
  • zfs zfs boot zpool zpool faulted
  • Replies: 51
  • Forum: Storage
 
a.png


I changed system boot from BIOS to UEFI, and then I boot the machine via LiveCD.
And then lsdev could show the zpool correctly,and I can also use below command to use livecd kernel to boot system

load /boot/kernel/kernel
load /boot/kernel/zfs.ko
load /boot/kernel/nullfs.ko
load /boot/kernel/mrsas.ko
set currdev="zfs:zroot/ROOT/default:"
boot


So I'm thinking the 2T bug still exists...
 
VladiBG said:
Check your Bios->HBA settings, it's limiting your disk to 2TB in Legacy mode.

edit:
If you have a full backup and you are going to risk, you can try to remove the SWAP partitions create a new ESP partition on each disk, then create again the SWAP partition and switch from BIOS to EFI.

It will look like this:

C

ZFS Thread 'ZFS Boot Issue'

I have a FreeBSD production system with 12 3TB SATA drives in a a JBoD triple-parity ZFS configuration spanning around 30TB. This was working well for months, through numerous reboots, however after a power outage Friday it's been unable to boot, giving 3 or more (it varies) `zio_read error: 97` entries, followed by or with interspersed `ZFS: i/o error - all block copies unavailable` in what appears to be every fourth entry (e.g. 3 error 97 entries then 1 all block copies unavailable entry, repeating 0 or more times.) The subsequent message is usually `ZFS: can not read MOS config`...
  • CoryG
  • zfs zfs boot zpool zpool faulted
  • Replies: 51
  • Forum: Storage
Click to expand...
Dell server doesn't have setting like legacy mode.

I'm thinking to create ESP partition on the first disk. Do I need to do this on every disk?
 
It's the 2TB problem again. The good thing is that switching to UEFI boot removes the limitation on your machine which is not the case on all affected machines.

Please acknowledge that Andriy pointed you in the right direction less than 40 minutes after your initial post.

For other people facing the same problem: You can try booting from the ZFS boot environment that was created before the upgrade. In many cases that would work and save you from unscheduled downtime.
 
I once heard about some problematic mother boards that have problematic UEFI firmware exhibiting exactly the same limitations even on UEFI boot if CSM (BIOS compatibility service) is enabled. In these cases, to avoid the limitation, disabling CSM (maybe listed as BIOS boot) and setting UEFI-only boot.

But as your computer sanely finds your pool on booting from installation media, it (hopefully) would not the case. But even UEFI boot fails, confirm this would worth trying.
 
meteor8488 said:
I'm using a Dell server with PERC 330 card in HBA mode, and 12 SSD with raidz2
Click to expand...
What's the exact type of that 330 card?

Because it looks like the first one of these: Dell PowerEdge RAID Controller 9 User’s Guide H330, H730, and H830:
Enabling boot support
NOTE: See your system documentation to ensure that the proper boot order is selected in the system BIOS.

In a multiple controller environment, you can enable BIOS on multiple controllers. However, if you want to boot from a specific controller, enable the BIOS on that controller and disable it on the other controllers. The system can then boot from the BIOS-enabled controller.

NOTE: BIOS displays 4 KB and 512-byte drives, whereas you can only boot using 512-byte drives. All 4 KB drives must boot only from UEFI mode.

Perform the following steps to enable the controller BIOS:

Press <Ctrl> <N> to access Ctrl Mgmt menu screen.
Press <Tab> to move the cursor to Enable Controller BIOS in the Settings box.
Press the spacebar to select Enable Controller BIOS.
An X is displayed beside Enable Controller BIOS.
Press <Tab> to move the cursor to the Apply button, and then press <Enter> to apply the selection.
The controller BIOS is enabled.

To disable the controller BIOS, use the spacebar to de-select the Enable Controller BIOS control, and then select Apply and press <Enter>.
Click to expand...
 
meteor8488 said:
yes. But with more than 20T data, restore is painful…
Click to expand...
Ok, Then you can try this
Boot from LiveCD under UEFI, you need to set your BIOS to UEFI only and disable the legacy BIOS so you can set the boot var using efibootmgr

Check if the disk that you want to convert from freebsd-boot to EFI is correct one. In the example bellow is da0

gpart show


Delete freebsd-boot, freebsd-swap on da0

gpart delete -i 1 da0
gpart delete -i 2 da0


create ESP

gpart add -a 4k -l efiboot0 -t efi -s 260M "da0"


Format ESP

newfs_msdos "/dev/gpt/efiboot0"


Create new SWAP to the rest of the freespace between ESP and freebsd-zfs (Note: You must NOT have any free space at the end of the disk otherwise you will need to specify the start offset for the partition)
gpart add -a 1m -l swap0 -t freebsd-swap "da0"

mount the ESP partition

mkdir /tmp/boot/
mount -t msdosfs /dev/da0p1 /tmp/boot


Copy the EFI and create the boot record in the UEFI

mkdir -p /tmp/boot/efi/EFI/BOOT
cp /boot/loader.efi /tmp/boot/efi/EFI/BOOT/BOOTX64.efi
efibootmgr --create --activate --label "FreeBSD" --loader "/tmp/boot/efi/EFI/BOOTX64.efi"



umount /tmp/boot
shutdown -r now


Note:
By default the efi partitons is also mounted under /boot/efi so the update of the future efi could be easy but this is valid only for the first disk, when you are using for example raid1 you need to update the efi on the secondary disk by hand. So check your /etc/fstab
 
VladiBG said:
Ok, Then you can try this
Boot from LiveCD under UEFI, you need to set your BIOS to UEFI only and disable the legacy BIOS so you can set the boot var using efibootmgr

Check if the disk that you want to convert from freebsd-boot to EFI is correct one. In the example bellow is da0

gpart show


Delete freebsd-boot, freebsd-swap on da0

gpart delete -i 1 da0
gpart delete -i 2 da0


create ESP

gpart add -a 4k -l efiboot0 -t efi -s 260M "da0"


Format ESP

newfs_msdos "/dev/gpt/efiboot0"


Create new SWAP to the rest of the freespace between ESP and freebsd-zfs (Note: You must NOT have any free space at the end of the disk otherwise you will need to specify the start offset for the partition)
gpart add -a 1m -l swap0 -t freebsd-swap "da0"

mount the ESP partition

mkdir /tmp/boot/
mount -t msdosfs /dev/da0p1 /tmp/boot


Copy the EFI and create the boot record in the UEFI

mkdir -p /tmp/boot/efi/EFI/BOOT
cp /boot/loader.efi /tmp/boot/efi/EFI/BOOT/BOOTX64.efi
efibootmgr --create --activate --label "FreeBSD" --loader "/tmp/boot/efi/EFI/BOOTX64.efi"



umount /tmp/boot
shutdown -r now


Note:
By default the efi partitons is also mounted under /boot/efi so the update of the future efi could be easy but this is valid only for the first disk, when you are using for example raid1 you need to update the efi on the secondary disk by hand. So check your /etc/fstab
Click to expand...
Thanks for the instruction. I'll have a try.
 
Just for general information, error 97 is FreeBSD's mapping of ZFS's ECKSUM to native EINTEGRITY.
In older FreeBSD version ECKSUM used to have a special value of 122.

If you get 97 while trying to boot from a pool but you import and access the pool just fine (and can read all files under /boot), then the most likely explanation is that the firmware is reading something else than it's supposed to.

Also, on a live system it's possible to use zdb command to examine block pointers (e.g., of a file or of some pool metadata) and see their disk offsets.
 
Andriy said:
Just for general information, error 97 is FreeBSD's mapping of ZFS's ECKSUM to native EINTEGRITY.
In older FreeBSD version ECKSUM used to have a special value of 122.

If you get 97 while trying to boot from a pool but you import and access the pool just fine (and can read all files under /boot), then the most likely explanation is that the firmware is reading something else than it's supposed to.

Also, on a live system it's possible to use zdb command to examine block pointers (e.g., of a file or of some pool metadata) and see their disk offsets.
Click to expand...
Thanks for your reply. The information your provided are interesting and new to me.

here is the zdb output. How to check if offset is correct or not?

zdb
zroot:
version: 5000
name: 'zroot'
state: 0
txg: 37347356
pool_guid: 9360550561432276301
errata: 0
hostname: ''
com.delphix:has_per_vdev_zaps
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 9360550561432276301
create_txg: 4
com.klarasystems:vdev_zap_root: 6314
children[0]:
type: 'raidz'
id: 0
guid: 5743027174022440926
nparity: 2
metaslab_array: 78
metaslab_shift: 37
ashift: 12
asize: 47957840756736
is_log: 0
create_txg: 4
com.delphix:vdev_zap_top: 65
children[0]:
type: 'disk'
id: 0
guid: 12015296855768086397
path: '/dev/da0p3'
whole_disk: 1
DTL: 2314
create_txg: 4
com.delphix:vdev_zap_leaf: 2820
children[1]:
type: 'disk'
id: 1
guid: 2385734231858165544
path: '/dev/da1p3'
whole_disk: 1
DTL: 204
create_txg: 4
com.delphix:vdev_zap_leaf: 2244
children[2]:
type: 'disk'
id: 2
guid: 3317263698871277805
path: '/dev/da10p3'
whole_disk: 1
DTL: 47744
create_txg: 4
com.delphix:vdev_zap_leaf: 46120
 
There are no block pointers in that output.
You might to start with something like zdb -u -vvvvvv rpool.
You may want to proceed to things zdb -ddddd -bbbbbb rpool (note: huge output).
zdb is a swiss army knife, it has lots of options, not all of them fully / clearly documented.

Some techniques can be glanced from resources like
 
In my view, instead of trying not converting to UEFI booting and using zdb analysis (and the effort and research you will have to put in it) to arrive at a viable and reliable long term solution of supporting non-UEFI ZFS booting with larger than 2 TB disks (to be duplicated over the multiple disks to be able to boot from) is unlikely.

I note that the PERC H330 is not a 'clean bare bones' HBA controller that can operate solely in HBA in pass-through mode. Also, the explicite note about UEFI and 4K sector drives in the Dell PowerEdge RAID Controller 9 User’s Guide p. 23 :
-- 4 KB sector disk drives
PERC H330, H730, H730P, H730P MX, H830, FD33xS, and FD33xD cards support 4 KB sector disk drives, which enable you to
efficiently use the storage space.
Before installing Windows on 4 KB sector drives, refer Windows operating system installation errors .
NOTE:
● Mixing 512–byte native and 512–byte emulated drives in a virtual disk is allowed, but mixing 512–byte and 4 KB native
drives in a virtual disk is not allowed.
● 4 KB sector disk drives boot only in UEFI mode.
Click to expand...

Relating to Dell controllers and pass-through mode, the Dell PERC HBA330 (pdf-HBS330) would likely be the better option for ZFS use.
 
You must log in or register to reply here.
Back
Top
OSZAR »