The FAT system was created way back when PCs only had one (very low-density) floppy-disk drive. When hard-disks were introduced to the PC world (around DOS 2.0), FAT was enhanced to allow directories and sub-directories. This capability let you group files into separate collections. And that's about as far as FAT progressed, even though VFAT (the file system available to Win95 users) promises one last gasp, albeit a small one.
OS/2 Warp provides a performance-enhanced version of FAT that allows you to maintain your old installation seamlessly and still use OS/2's WorkPlace Shell interface. With today's larger hard-disks and Object-Oriented interfaces, however, FAT is really starting to show its age. Unfortunately, there's little that can be done to overcome its initial design limitations (though some are still trying).
OS/2's Easy Install method automatically selects the FAT system because that's what most computers still use. Since Version 1.2, however, OS/2 has shipped with an alternate, redesigned file system called HPFS. The designers of HPFS saw several fundamental limitations in the way FAT operated that limited its use for networking and large hard-disks. To overcome these limitations, they set out to create a completely new system rather than try to twist FAT into shape for the '90s.
Rather than being built into the operating system (as is FAT to DOS), HPFS is installable just like a device-driver, which means that it can exist alongside other Installable File Systems (CD-ROM, optical, and a few others). Technically, HPFS is an abstract layer of functions that listen to OS/2 for file-management requests, passing them to and from the hard-disk(s) onboard (and sometimes network cards) via their device drivers. When new file systems become available (like the Toronto Virtual and ZipStream File Systems) they can be addedand removed almost like a device driver. Only the HPFS driver (to-date) re-formats a disk.
The easiest way to explain something as esoteric as a file system is to
look at What You Pay versus What You Get. To explain What You Get we'll
look at what you've been using all these years (FAT) versus HPFS.
Since HPFS is essentially a file system "driver" like you would have for a CD-ROM or printer, it requires a certain amount of RAM. The RAM cost is about 150k. If you have only 4MB of RAM, forget about HPFS. However, with the recent RAM glut (and its price falling rapidly) brought on by over-eager demand projections, a RAM upgrade is the cheapest, most effective upgrade you can make these days: 32-bit OSs (and even pretenders) love RAM.
Another "cost" is that native ("real") DOS cannot see HPFS partitions. This implies it cannot be used for a Dual Boot installation unless you can organize your files so that the DOS side doesn't need files from the OS/2 side. There are shareware utilities ( Amos and HPFS/DOS come to mind) that let DOS read HPFS partitions (and write to them in the registered versions).
There's also a temporary slight cost in Personal Insecurity with a "new" (and very important) way of configuring your machine. When I first got OS/2 in 1992, I installed it as a Dual-Boot system with FAT because that's what I understood from my DOS days. I had a handy set of tools to cope with any problems. After a while I converted to a Boot Manager setup with a few HPFS partitions. Once I saw HPFS was easy to install and worked great I became hooked on its benefits. Now I do HPFS-only installations whenever possible.
If you're a new OS/2 user, you may be a little too intimidated to convert to HPFS. But if you've just bought a brand-new machine and/or a large hard-disk, keep on reading to see what you would be missing. If you have multiple hard-disk partitions you may want to experiment with formatting one or more to HPFS. If you're just getting OS/2 Warp on a brand-new machine then you might want to take the leap right from the start.
The point is that you could. Most people don't get carried away with this. But as you get used to being away from DOS's 8.3 world you begin to find inventive uses for this space, especially when combined with OS/2's Folders. You can name a letter "Happy Birthday Note to Grandma" or a database "1996 Christmas Cards." Any HPFS-aware program (just about any for OS/2) will quite happily use it. Since DOS programs come from the 8.3 dark ages, they will be blind to these names, though.
Here's a view of the same directory from the command line and Icon View. There are several files that easily break the 8.3 format, with mixed-case and "odd" extensions. I tend to avoid spaces in file names I intend to use from the command line though, because they need to be quoted ("") to be treated as a single file and that's more work than I care to do. It may be nice to have files called "Happy Birthday Note to Grandma", but from the command line (which is where I work a lot) if I forget the '"'s then a
del Happy Birthday Note to Grandma
becomes del Happy and del Birthday and del Note
and... well you can see the point. Similarly, Rexx is not too smart about handling filenames with spaces, as it uses a space to separate arguments from the command-line, even if they are enclosed in '"'s. Applications that a long-name-friendly, however, should have no trouble managing these names (though if you get carried away file-picker boxes can be unmanageable.
For different reasons, HPFS has an advantage when it comes to path lengths (when you refer to a file), letting you specify up to 259 bytes versus FAT's 64. With long file and directory names the space is needed.
FAT, however, is still fundamentally limited in what it can store about a file (no matter who twists it this way and that). The OS/2 FAT uses some of what little room is left to point to a database of EAs. This database is in a file in the root directory of every partition called EA DATA. SF. Unfortunately no matter how well this is done, there still is a link that needs to be maintained for every file on the partition, and plenty of room for Murphy's Laws to creep in and break links or, worse, corrupt EA DATA. SF (always do a proper shutdown). HPFS stores a file's EA information as part of a structure called its FNODE. This limits any possible corruption to one file rather than a "file of other files' data."
What do EAs look like? You can see from yourself via a folder's Details View, the File tab on a file or folder's Settings notebook, or using one of a number of EA browsers. Since the sample I chose is a Rexx command file (actually a WPCommandFile, a subclass of the WPProgramFile class) it contains EAs used in the the actual execution of the command. [It's actually a collection of "Stupid Icon Tricks" that can be done with Rexx on the OS/2 Desktop.]
This is no problem for a hard-disk with a couple hundred files. But when you collect a couple thousand files, the time to find any one of them increases dramatically. It's quite common for a hard disk to have several thousand files. My (OS/2 Warp Connect) C:\OS2 directory tree has 2889 files and my installation of Lotus SmartSuite has another 738. Typically, each application you have brings another 100+ files onto your hard-disk. You can see they add up to quite a lengthy list.
HPFS uses a different arrangement, designed to minimize the number of times a disk needs to be searched (and thus get data to you faster). Rather than a flat list of directories and files, HPFS uses a Binary Tree for directories, files and sectors. This kind of layout means that every time something is looked up, the number of things left to search is cut in half. The Tree looks roughly like that of a folder's Tree View.
This "divide and conquer" method extends all the way through HPFS, from directories you see to the way bytes are written to disk. It is an extremely powerful way to "think smarter" about organizing lots of things. Where FAT would have to look through a huge list of files and directories to find a file, HPFS only needs to look at a lot fewer directories. This reduction can be an amazing speed boost. Furthermore, HPFS directories are stored right in the middle of all your files, rather than scattered among them.
Another performance booster is the way HPFS pre-stores (or "caches") a little extra of everything it reads. It assumes that if you wanted some, you'll want more fairly soon. Very often, it's correct, and it reads this info directly from memory (which is asignificantly faster than from disk). Files are also cached differently depending on what they are and how frequently they have been requested in the past.
Finally, HPFS has a more intelligent way of managing space than FAT. FAT stores the address of free space areas as special entries in its single list. HPFS stores a "map" of free space that can be searched very quickly. FAT grabs exactly as much space as a file needs when it is created or needs to grow. HPFS grabs a little extra, even more if the program requesting the space knows how to ask, and "over-grows" a file temporarily until it is closed.
HPFS was designed to manage nice bite-sized pieces quickly. HPFS hands out disk space in sectors of 512 bytes. Your AUTOEXEC.BAT would then waste only a half-sector (1/4k) rather than almost 8k. Wherever possible, these sectors are allocated so that there is room for more to be added nearby later. FAT allocates more space at the first place it is available. This leads files to become scattered in pieces across the disk. HPFS packs smaller pieces together tighter and more intelligently so that fragmentation is minimal. After three-plus years of round-the-clock operation my desktop computer reports 2% fragmentation for over 20,000 files (1.5G).
HPFS also offers a Hot Fix feature that is usually only found on Network Operating Systems (like Novell). This feature automatically and invisibly moves data from damaged (or suspect) areas of disk to known good areas. A section of disk (called the SpareBlock) is reserved for the (hopefully very) infrequent occasions when a disk sector is found to be bad and the data a program is waiting to store needs to be put somewhere. The bad sector is marked as such, your file never notices anything happening and you only see a warning notice. Later, when you next start OS/2, the bad sector is marked permanently by CHKDSK and the file that was affected is moved away from the bad sector.
IFS=C:\OS2\HPFS.IFS /CACHE:cachesize /CRECL:4 /AUTOCHECK:disklist
Cachesize is the number of kilobytes of RAM to reserve for disk caching. More is not always better, because this RAM can only be used for caching at the expense of all other potential RAM users. The default cachesize is selected based on how much RAM you have, and is usually the best choice. There is an upper-limit of 2M for this cache.
/CRECL specifies the largest disk read that will be cached. It's usually not necessary to cache everything that's read from disk because there's an inverse relationship between the size of reads and the likelihood they'll need to be re-read anytime soon. Data from smaller reads is generally used more than that from larger reads. Boosting this number can improve performance if you have enough cache. Otherwise, the larger reads "swamp" the cache too often.
When you boot, HPFS does a "sanity check" of all drives before it lets OS/2 really get going. If it sees that a drive was not shut down properly, CHKDSK is run on all drives listed in disklist.
If CHKDSK isn't enough, GammaTech Utilities include a repair utility that goes the extra distance. GammaTech also has a file defragmenter.
PowerQuest makes a product called PartitionMagic that converts FAT partitions to HPFS without using the Backup-Format-Restore process. This makes it even easier for you to lose FAT and gain all the benefits of HPFS.
There are actually two "flavours" of HPFS: the one that comes with OS/2 and HPFS386 which comes with IBM LAN Server Advanced. HPFS386 is a "heavy duty" version of HPFS designed for very-demanding environments like file and application servers. It has three major features that make it well worth its extra (dollar) cost and not really suitable for The Average Joe.
386HPFS operates as an SMB (Server Message Block) processor running at Ring 0, as well as being a file system. Why is this good? NetBIOS does its work with SMBs: they are the most common "frame"s exchanged between two machines over the wire. Since almost all Server traffic is network-to-disk (assisted with RAM in the form of a cache), letting your network card talk directly to the file system without consulting the OS means a lot of speed, especially given the tremendous traffic generated for NetBIOS User-Server requests. You would be amazed at what gets tossed back and forth for the simplest (from a user's view) transactions.
Of course this is all moot if you have no network connections, and of marginal use until you get about 20-30 heavy users connected to a Server.
All Access Control Profiles (permissions) are stored as FNode entries, like Extended Attributes and Directory Limits. Controlling security and space limitations from within the File System means these functions become a part of every file and directory rather than a "third party" (with to- and from-links that can be compromised), much more seamless and impenetrable.
This feature makes local security possible, whereby a user can be forced to login at the Server keyboard to be granted access to files. This is actually more hassle than it is worth except to the security-sensitive, because all of the Server's local activities (things like scheduled operations) must be wrapped in some extra stuff so it can run even when an Administrator isn't logged onto the Server locally.
By contrast, LAN Server Entry and Warp Connect Peer use a separate file called NET.ACC to store permissions, and there's a trip made to that file for every Read, Write, Create, Delete, Permit and Execute request before the request is serviced.
For Regular Folks, the only real advantage of 386HPFS is that its cache has no 2M top-end, and even that's of little benefit to all but keeners. This benefit is alleged to be going away in Merlin, though even then there might only be a higher top-end. 386HPFS will use 20-60% of system memory for caching, which also means you need enough to begin with (over 16M).
And that just about wraps up all the reasons why HPFS is easily worth its (relatively) light cost. It was designed to manage a computing life that is starting to include us all: a multi-tasking environment using many files on large disks (and sometimes over networks). Suffice to say that HPFS is worth the trouble if you use your machine more than a little. It's well-proven to be faster and more reliable and more fun than FAT. Converts rarely go back. If you didn't understand much from the 386HPFS discussion then you don't need it.
As with any technology there are places it belongs and places it does not. HPFS does not belong on machines with less than 8M RAM, disk partitions less than 100M and machines that require Dual Boot (because of a DOS-only application). HPFS does belong on everything else and is a "free upgrade" included with OS/2 Warp.
The future, however, may force your hand. The Java Developers Kit included in the next OS/2 release (Merlin) requires HPFS for long filename handling. Since Java comes from the land of Unix, and Unix people have had the luxury for so long, long filenames are assumed to be available. Get used to it. Even smug Macintosh users are starting to realize that their once-advanced 32-character names are too small.
I ran for three years without a tape backup (but you shouldn't do this :) ) using a bit of Rexx and a file archiver. Along the way I had accumulated two 512M disks containing four partitions: [System, Applications], [Data, and Scratch]. One day my disk controller failed, making a little bit of a mess along the way, but no matter - that was for later. For various reasons (the cost of another dead-end ISA SCSI controller among them) I decided to upgrade my system, and move my OS/2 data to a new larger drive in the process. I wanted to move the system intact - I HATE re-installing and consider it the very last solution to OS/2 problems, especially for long-running systems like mine. Plus, I'm lazy, and figure that OS/2 is modular enough that I can do small, surgical, changes to avoid re-doing three years of customizing.
So, in went both old disks to the new machine because I didn't (as yet) have a full backup of my System and Applications partitions. From a floppy boot, I used a file-archiver to compress (after some careful janitorial work) Data and Scratch, to make room for System and Applications. In this case I chose to use XCOPY to backup System and Applications, because I could better edit them if need be. The "trick" to this is the time-honoured RTFM method: I looked up all the switches I'd need for XCOPY to literally duplicate my partitions into directories on another disk. Here's tha magic incantation:
XCOPY C:\*.* F:\SysRebuild\Drive_C /T /H /R /S /E /V
giving me all sysTem, Hidden and Read-only files in all Subdirectories (even if they are Empty), Verified along the way. Took about two hours to do System and Applications. Then, the compressed Data partition was copied over to the first drive and the drive was pulled and put aside as a backup.
The new drive went in, and again from a floppy-boot (that Create Utility Disks utility in OS/2 System-System Setup is a must-do for your own sanity: no installation program to load and abort makes booting a lot faster) the drive was partitioned (using Boot Manager to mark the first partition as Startable) and formatted. Time to restore my system to the new disk, so we modify the incantation to read:
XCOPY F:\SysRebuild\Drive_C\*.* C:\ /T /H /R /S /E /V
and go away for two hours. On return, I diddled with my video and disk-controller setup (I got new cards) and rebooted to my old system on new hardware.
The relevance of this to HPFS is that there is only one extra step needed to use this method to convert from FAT. OS/2 uses C:\OS2LDR to distinguish between FAT and HPFS partitions at boot-time. This file is used to let the boot process read the disk without knowing exactly (because CONFIG.SYS hasn't been loaded yet) whether it is FAT or HPFS. This file comes in HPFS and FAT flavours, and it's simply a matter of putting the appropriate file on a formatted partition. By the way, this file is marked as System, Hidden and Read-only.
If my old setup had been FAT-only I would need to remove this file from my backup before restoring it to the new disk. If I'd forgotten this, there are two ways the file can be changed: 1. literally getting a copy from another bootable HPFS partition with the exact same OS/2 version, or 2. using SYSINSTX C: after booting to a command-line from the OS/2 boot disks.
End of Document
