Solaris patch install failure

During a recent install of Solaris patches, there was a problem with disk space and the system had to be rebooted. The system would no longer boot, complaining that the boot file (which defaults to /kernel/unix) was not executable:

Rebooting with command: boot disk -s
Boot device: /sbus/SUNW,fas@e,8800000/sd@0,0 File and args: -s
Boot load failed.
The file just loaded does not appear to be executable.
Boot device: disk3:a File and args:
Boot load failed.
The file just loaded does not appear to be executable.
{0} ok

I even attempted to upgrade the OBP, since the system was quite old, but it would not run that file either.

My first thought was that the bootblk was bad, but restoring that from the OS CD didn't help:

# dd if=/usr/platfrom/`arck -k`/lib/ufs/fs/bootblk of=/dev/rdsk/c0t0d0s0 conv=sync bs=1b oseek=1

The file system passed fsck, mounted fine, showed no problems. I then did a pkgchk on SUNWcsr, no problem. On a working system, I determined that the "real" boot unix (/kernel/unix) lives in /platform/`arck -k`/kernel/unix, and with pkgchk -v -p /platform/`arch -k`/kernel/unix I determined that the appropriate packages are SUNWcar and SUNWcarx. Performing pkgchk SUNWcar and pkgchk SUNWcarx immediately showed that there was a serious problem, most of the files in /platform were gone.

The bottom line: the /platform files had been removed from the root drive by the failed patchadd, and nothing will boot from a hard drive with out them, even a supposedly "stand-alone" program like the OBP patch.

As a quick fix, I restored the files from a working system, an alternate idea would be to use pkgadd to reinstall from the Solaris CDROM and then reapply patches to the system.

In order to do this, put the Solaris CDROM 1 of 2 into the CD drive, halt the system and use "boot cdrom -s" to bring the system up. If the "-s" flag is not supplied, the system will try to reinstall Solaris, you'll have to use STOP-A to halt the system and try again.

Once mounted, run fsck on the old ROOT (/), /usr and /var (if appropriate) and mount them under /a:

bash-2.05# fsck /dev/rdsk/c0t0d0s0
+++ OUTPUT SNIPPED +++
bash-2.05# fsck /dev/rdsk/c0t0d0s3
+++ OUTPUT SNIPPED +++
bash-2.05# mount /dev/dsk/c0t0d0s0 /a
bash-2.05# mount /dev/dsk/c0t0d0s3 /a/usr

A good pkgchk run from the CDROM Solaris should look something like this:

bash-2.05# pkgchk -R /a SUNWcar SUNWcarx
ERROR: /a/platform/sun4u/boot.conf
modtime <04/06/02> expected <07/18/03> actual
bash-2.05#


Errors from "modtime" are typically not a problem, also other pkgchk might show files that are changed during the normal operation of the system, For example, a pkgchk on SUNWcsr would show many, many modified files, you have to check each one to be sure it is appropriate:

bash-2.05# pkgchk SUNWcsr
ERROR: /etc/.login
group name expected actual
modtime <04/06/02> expected <10/28/04> actual
file size <524> expected <534> actual
file cksum <39191> expected <39926> actual
ERROR: /etc/cron.d/at.deny
modtime <04/06/02> expected <10/28/04> actual
file size <45> expected <237> actual
file cksum <4171> expected <21761> actual

+++ OUTPUT SNIPPED +++

ERROR: /var/spool/cron/crontabs/adm
modtime <04/06/02> expected <07/18/03> actual
ERROR: /var/spool/cron/crontabs/root
group name expected actual
modtime <04/06/02> expected <09/24/04> actual
file size <421> expected <255> actual
file cksum <33544> expected <18497> actual
ERROR: /var/spool/locks
permissions <0755> expected <0777> actual