Filesystem metadata support

From Mac OS X Server FAQ

Jump to: navigation, search

Contents

Filesystem metadata types

A list of filesystem metadata types supported for Mac OS X and Mac OS X Server. These are listed and described by the accessing API or technology used to work with them as a programmer might consider them. It is important to note that a number of these APIs and access methods implement the same metadata but accessed in different ways by different methods so the metadata that is needed may incidentally be captured by an accessor API that a tool supports while not supporting another accessor API. Conversely metadata may be captured more than once by a tool supporting more than one accessor API, as well, potentially causing problems.

File type

While usually assumed to be supported there are notable differences between different file types that may conceal what type they actually are. Some of major the file types include:

  • named pipe (fifo)
  • character special
  • directory
  • block special
  • regular
  • symbolic link
  • hard link
  • socket
  • whiteout

Classic stat struct

The "classic" UNIX stat structure implements the majority of the most visible per-file filesystem metadata such as:

  • Owner (st_uid)
  • Group (st_gid)
  • Size (st_size)
  • Permissions ("protection mode") (st_mode)
  • Timestamps (including nanosecond resolution)
    • Time of last access (st_atime, st_atimespec.tv_sec, st_atimespec.tv_nsec)
    • Time of last modification (st_mtime, st_mtimespec.tv_sec, st_mtimespec.tv_nsec)
    • Time of last status change (st_ctime, st_ctimespec.tv_sec, st_ctimespec.tv_nsec)
  • Flags (#chflags(2)) (st_flags)
  • Other special metadata related to device nodes (st_dev, st_rdev), block size & allocation (st_blocks, st_blksize), inode (st_ino), and file generation number (st_gen).

The stat structure is defined in various POSIX standards as well as described in the Single Unix Specification.

This is data typically modified with tools such as chmod, chown, and many, many others.

chflags(2)

The chflags(2) metadata is a BSD-dervied set of file flags that define for example some operational and access control properties of files. Some of the flags are:

  • Do not dump the file. (Owner or super-user)
  • The file may not be changed. (Owner or super-user)
  • The file may only be appended to. (Owner or super-user)
  • The file may not be changed. (Super-user)
  • The file may only be appended to. (Super-user)

Most familiar may be a file's "locked" status (as seen from the "Get Info" dialog). This is implemented as the user-immutable flag (uchg).

The data can be manipulated using the chflags tool.

The chflags(2) data is stored in the stat struct per-file (st_flags) and so if a tool can claim full support for the stat struct it may incidently support this metadata. It is an include-file defined bitfield so it may not be portable cross-platform if stored from this variable (eg, different OS vendors may define the bitfield values differently and thus the bitfield in binary form may not be portable).

Apple/HFS file attributes

Apple Mac OS/HFS file attributes are proprietary to Mac OS X/Darwin. The getattrlist(2) API has a detailed description of various Apple file attribute details. An incomplete list of some file attribute metadata:

  • Type of file system on which file resides
  • Per-mount per-volume unique file identifier
  • Text encoding hint for file name
  • Backed-up time
  • Finder info
    • Type (files)
    • Creator (files)
    • Finder flags (files & folders)
      • Classic label (color)
      • Alias file (a), Has bundle (b), Custom icon (c), Located on the desktop (d), Extension is hidden (e), Inited (i), Locked (l), Shared (m), File has no INIT resource (n), System file (s), "Stationery Pad" file (t), Invisible (v), Busy (z)
    • Folder location (files & folders)
    • Window bounds (folders)
    • Extended finder flags (files & folders)
    • Put away folder ID (files & folders)
    • Scroll position (files)
  • Creation time
  • Logical & physical size of all forks and single forks
  • Optimal read/write block size
  • Allocation clump size
  • Initial fork extents

Apple has some reference documentation that describes some of these, as well.

There may be overlap of some attributes and other metadata types (POSIX stat, for example).

Some of this data can be manipulated manually by the SetFile and GetFileInfo Developer tools (part of Xcode).

Resource forks

Resource forks are the classic Mac OS way of storing some metadata. They are well-structured and extensible.

The Rez and DeRez Developer tools (part of Xcode) can be used to manipulate resource forks. Resource forks may also be directly accessed from any tool using the "..namedfork/rsrc" path attribute. For example if you have a text file "/myfile.txt" you may read its resource fork data by opening the file "/myfile.txt/..namedfork/rsrc".

Extended Attributes

A new feature of Mac OS X 10.4 Tiger was extended attributes. This enabled arbitrary metadata through a standard API to be stored on files. It uses an API very similar to the BSDs as well as Linux.

Apple has implemented a number of special-case extended attributes to store other data using the EA APIs. Specifically resource forks and Finder info can be saved and restored using these methods.

A third party tool, xattr (link may not work due to anti-deep-linking techniques), may be used to view and manipulate extended attributes. Also, the Finder's GetInfo dial box allows setting some "Spotlight" text which is implemented as extended attributes.

ACLs

Another feature of Mac OS X 10.4 Tiger was ACL support. ACLs are actually implemented on a per-file basis as "system"-level extended attributes, which may allow, from an API perspective, the preservation of them by just implementing the EA API. Apple has provided a somewhat-standard implementation of ACLs which they use in some of their CLI application that is based off of the POSIX.1e draft standard. Unfortunately Apple's implementation of the draft standard API is not compliant.

The chmod(1) tool with the -a, +a, and =a flags may be used to manipulate ACLs. The "ls" tool with the -e flag may be used to view ACLs.

Volume metadata

Volume metadata is classifed as various volume-global information that may pertain to a set of files that reside on a volume. For example file system labels, free space, blocks used, file system type, and other data may be captured under this moniker. statfs(2) may be a common way to get some of this data. As well, the getattrlist(2) can return some volume-specific metadata.

The diskutil(8) utility with the info command may view some volume/filesystem information.

Metadata files

Some files contain and describe metadata about other files and are integrated/proprietary to Mac OS X.

DS_Store

The .DS_Store files in directories contain some information on folder visualization (sort/listing order, etc.) as well as Get Info/Finder comments for files in some cases. These files are mostly related to the Finder itself. See also an article discussing .DS_Store files and folder views and an article on the origins of .DS_Store.

Desktop Database

The Desktop Manager maintains a desktop database that is used for, among other things, "maintaining user comments associated with files and managing the icons used by applications." The Apple More Macintosh Toolbox chapter on Desktop Manager has more documentation.

Dot-underscore

Files prefixed with a period and underscore before the name of an accompanying file have some metadata such as resource forks, ACLs, EAs, Finder Info and others stored in them when the container medium (filesystem, zip/tar file, etc.) cannot natively support such metadata. The format and implementation for Mac OS X 10.4.2 is located in a Darwin libc source file. The file is in AppleDouble format which is documented in RFC 1740. Some of metadata captured with this method are:

  • ACLs
  • Extended attributes
  • Finder Info
  • Resource forks

Notable is that a large portion of the Apple file attributes such as creation time are missing.

Filesystem metadata tool support

This is a reference table of the various per-file and per-volume filesystem metadata types available in Mac OS X cross referenced with various built-in and third party tools support of capture and usage of them. This table assumes an HFS+ filesystem. It is also assumed that all of these tools support the "data" fork, that is the actual contents of files. Note: a status of Yes does not necessarily mean that all of the particular metadata is supported. Please check the actual support through each tool's support channels.

Tool/metadata support matrix

Tool/Metadata stat Rsrc forks File attribs chflags(2) EAs ACLs* Vol info
Built-in tools
tar Yes >=10.4 >=10.4 No >=10.4 >=10.4 N/A
ditto Yes Yes Partial Yes N/A
pax N/A
cpio N/A
ASR (device mode) Yes Yes Yes Yes Yes Yes Yes
ASR (file mode) Yes N/A
rsync Yes >=10.4** >=10.4** No >=10.4** >=10.4** N/A
Third-party tools
hfstar Yes Yes Yes No No No N/A
xar Yes Yes Yes Yes Yes No, but should
hfspax N/A
Bacula file daemon Yes Yes
BRU agent Yes
LBackup >=10.4 >=10.4 >=10.4 >=10.4 >=10.4 >=10.4 N/A
Retrospect & Retrospect client Yes Yes Yes Maybe?
Backup Exec agent Yes
psync Yes Yes Yes
RsyncX Yes Yes Yes N/A

Native metadata tools

Starting with Mac OS X 10.4 Tiger a number of built-in command line tools have had the ability to preserve metadata that the traditional unix tools could not. For example the tar utility can archive resource forks and other metadata.

Broken POSIX ACLs

Mac OS X 10.4 appears to have a somewhat "broken" implementation of the draft-standrard POSIX.1e ACL API. Because of this some tools that should be able to preserve this metadata, such as Xar, cannot, at least through standard APIs.

rsync Problems

There have been serious reported problems with Apple's implementation of rsync's extended metadata support. As an alternative there is RsyncX.

See also

Retrieved from "http://macos-x-server.com/wiki/index.php?title=Filesystem_metadata_support"
Personal tools
Toolbox