Next: , Previous: , Up: Top   [Contents][Index]


4 Invoking fswatch

This chapter is about how fswatch is invoked. There are many options and two styles for writing them.

4.1 Synopsis of fswatch

fswatch is invoked using the following syntax:

fswatch (options)* (paths)+

fswatch interprets file names as being relative to the working directory and canonicalizes them using realpath.

If a directory is used as an argument, the directory object is watched and, optionally and depending on the monitor being used, the directory is scanned recursively and all its children are watched as well.

Depending on the monitor being used, recursively scanning huge directory hierarchies or big set of files may be resource consuming, CPU intensive or even impossible. The characteristics of the available monitors in a system should be assessed in order to choose the best monitor according to the specific needs.

Besides successful exits4, indicated with the exit code 0, fswatch may exit with an error. fswatch will try to print a diagnostic description on stderr when an unexpected error occurs.

The documented5 exit codes of fswatch are the following:

0

FSW_EXIT_OK: No error occurred.

1

FSW_EXIT_UNK_OPT: An unknown option was input.

2

FSW_EXIT_USAGE: Help message was requested.

3

FSW_EXIT_LATENCY: Invalid latency.

4

FSW_EXIT_STREAM: A stream related problem occurred.

5

FSW_EXIT_ERROR: An unkown error occurred.

6

FSW_EXIT_ENFILE: A file could not be opened.

7

FSW_EXIT_OPT: Unused.

8

FSW_EXIT_MONITOR_NAME: The specified monitor does not exist.

9

FSW_EXIT_FORMAT: The specified monitor is invalid.

4.2 The Two Option Styles

fswatch implements two option styles which are common in Unix and Unix-like operating systems and GNU software: short and long options. The biggest difference between short and long options are argument placing (for options taking one).

Whether long options are available in a system depend on the availability of the getopt_long function at build time. For this reason, users should familiarise themselves with short options and use them when possible and do not rely on long options to be available on any fswatch installation.

4.2.1 Long Options

In systems where the getopt_long function is available, each short option has a corresponding long option with a mnemonic name starting with two dashes (e.g.: --version). Long options are meant to be easy to remember and to provide hints about what a command is going to perform. The following command:

$ fswatch --event-flags --numeric --recursive ~

is clearer than:

$ fswatch -xnr ~

If a long option takes an argument, it can be specified in two ways, depending on whether the argument is optional or mandatory:

4.2.2 Short Options

Most options have a short form consisting of a dash followed by a single character, such as -l (which is equivalent to --latency). When available, a short form is interchangeable with the long one.

If a short option takes an argument, it can be specified in two ways:

Short options can be stuck together provided all the options but the last one take no argument, in which case it can be specified as described above. The command:

$ fswatch -xnrl 5 ~

is equivalent to:

$ fswatch -x -n -r -l 5 ~

where ‘5’ is the argument of -l.

4.3 fswatch Options

In the following table you can find the list, in alphabetical order, of fswatch’s options.

--access
-a

Monitor file access. This functionality is supported by selected monitors only.

--allow-overflow

Sets the allow overflow flag of the monitor. When this flag is set, monitor buffer overflows are reported as change events of type fsw_event_flag::Overflow.

--batch-marker

Print a marker at the end of every batch.

--directories
-d

Request the monitor to watch directories only during a recursive scan. This feature helps reducing the number of open file descriptors if a generic change event for a directory is acceptable instead of events on directory children.

--event

Filter events by type using the specified event name or numeric mask (see Filtering by Event Type). Specified event names are included in the output. Multiple event types can be specified using this option multiple times.

--event-flags
-x

Print the event flags.

--event-flag-separator

Use the specified string as event flag separator.

--exclude
-e

Exclude paths matching regex.

--extended
-E

Use extended regular expressions.

--filter-from

Load filters from the specified file.

--fire-idle-events

Fire idle events.

--follow-links
-L

Symbolic links are followed instead of being watched as file system objects.

--format

Use the specified record format.

--format-time
-f

Print the event time using the specified format.

--help
-h

Show the help message.

--include
-i

Include paths matching regex.

--insensitive
-I

Use case insensitive regular expressions.

--latency
-l

Set the latency using the specified value.

--list-monitors
-M

List the available monitors.

--monitor
-m

Use the specified monitor.

--monitor-property

Pass the specified property to the monitor (see Monitor Tunables).

--numeric
-n

Print a numeric event mask.

--one-per-batch
-o

Print a single message with the number of change events in the current batch.

--one-event
-1

Exit fswatch after the first set of events is received.

--print0
-0

Use the ASCIINUL’ (‘\0’) as record separator.

--recursive
-r

Recurse subdirectories.

--timestamp
-t

Print the event timestamp.

--utc-time
-u

Print the event time as UTC time.

--verbose
-v

Print verbose output.

--version

Print the version of fswatch and exit.

4.4 Whitespace and Record Format

As seen in Observing File System Changes, file names may contain characters such as ‘\n’ which are commonly used as line separators. Many commonly used Unix commands and shell builtins use characters in the $IFS environment variable6 as separators to split words. By default, $IFS contains the characters ‘ ’ (SPC), ‘\t’, ‘\n’ and ‘\0’ (‘NUL’).

Therefore, if a file contains such a separator character (and all but ‘NUL’ are legal), then a parsing ambiguity may arise when using certain record formats such as:

path( flag)+

In this case, for example, if n > 1 flags are present in the record, and hence more than one ‘ ’ (SPC) is present, then it is not known whether any subset containing a number x of consecutive flags (x < n) is part of the path or not.

The same reasoning applies when splitting lines instead of words: since ‘\n’ may be a legal file name character, then it is now known whether ‘\n’ indicates a record’s end or simply is part of a file name.

For this reason, in order to avoid parsing ambiguity, this options instructs fswatch to use ASCIINUL’ as record separator.

Warning: The use of the --print0 solves the line splitting ambiguity but not the word splitting ambiguity when using textual event flags. A solution to this problem is provided by custom record formats (see Custom Record Formats).

Another way to get an unambiguous record format is using numeric event flags (see Numeric Event Flags).

4.5 Custom Record Formats

To solve the problem of line splitting ambiguities and to provide users the possibilities of tailoring the record format to their needs, fswatch allows users to specify the event record format using the --format option.

This options requires a printf-like7 format string ordinary text containing zero or more directives. Characters not belonging to a format directive are copied unchanged to the output, while directives are interpreted and replaced with the result of their evaluation.

4.5.1 Format Directives

Directives start with ‘%’ which is always treated as a special character: either it marks the beginning of a directive or it is interpreted as an escape character8.

The available directives are:

%%

Inserts the ‘%’ character.

%0

Inserts an ASCIINUL’ (‘0’) character.

%n

Inserts a newline character.

%f

Inserts the list of event flags, separated by default by the space character (‘ ’) or by the separator specified with the --event-flag-separator option (see Event Flag Separator).

%p

Inserts the path.

%t

Inserts the timestamp, formatted with strftime using the format optionally specified with the --format-time option.

4.5.2 Record Termination

Each record is terminated by either a newline character (‘\n’) or an ASCIINUL’ character when -0 is specified. The record termination character has the following characteristics:

4.5.3 Event Flag Separator

When the list of event flags is printed, textual items are separated by default by spaces (‘ ’). The user can specify an alternate event flag separator using the --event-flag-separator and passing the desired separator string as argument.

For instance, if the user wants event flags to be separated by a comma, the following command can be used:

$ fswatch --event-flag-separator=, -x (options)* (paths)+

4.5.4 Builtin Formats

The format used by fswatch when a custom format is not specified is determined as follows9:

4.6 Batch Marker

Since fswatch typically outputs an endless event stream, processing parties parsing its output may be interested in ‘batch event processing’: that is, processing batches of events instead of endlessly processing events one by one.

To support this use case, fswatch provides the --batch-marker option; when specified, fswatch will output a customizable ‘batch marker record’ processing parties can use as batch delimiters. Batch demarcation is made naturally using the monitor’s processing loop and its latency setting: every time the monitor loops (typically when latency is elapsed), then a batch marker is printed as final record, as shown in the next example:

$ fswatch --batch-marker -r ~
/home/fswatch/.zsh_history.LOCK
NoOp
/home/fswatch/.zsh_history.new
/home/fswatch/.zsh_history
/home/fswatch/.zsh_history.LOCK
NoOp

By default, the batch marker takes the form of a single-line record:

NoOp(\n | \0)

terminated with either ‘\n’ or ‘NUL’ (‘\0’) depending on other fswatch settings. However, the user can customize it by providing the desired marker string as optional argument to --batch-marker:

% ./fswatch --batch-marker="*** BATCH END ***" -r ~
/home/fswatch/.zsh_history.LOCK
*** BATCH END ***

4.7 Idle events

An idle event is a special event type that can optionally be emitted by fswatch if no change events were collected in a period of time whose average length is equal to the monitor latency (see Latency). Idle events come in handy when an observer wants to perform an operation every time a change is detected or after a specified amount of time even if no changes were detected.

Idle events were introduced in version 1.9.0 and are available only when fswatch is built on a platform that supports C++11 threads (std::thread) and can be enabled using the --fire-idle-events option.

An idle event has got the following characteristics:

4.8 Filtering by Path

Filters are regular expression which are evaluated against the monitored object path to determine whether a path must be accepted or rejected. Sometimes, the exclusion of a path may result in the exclusion of an object from the list of monitored objects, while other times a path must be evaluated only when an event is detected and in this case the corresponding object cannot be removed from the monitored object list in advance10.

Event though event filtering is commonly performed when processing fswatch output, the possibility of filtering paths ‘at the source’ provides not only a greater amount of flexbility, but also:

Filters are implemented using the C++11 <regex> library. This feature is now required to successfully build libfswatch and fswatch.

4.8.1 Specifying Filters

Path filters can be specified in two ways:

4.8.1.1 Filter File Format

The filter file is made up of records separated by a new line charactere, or formally a ASCIILF’ (line feed) character. The structure of the record is the following:

type pattern

where ‘type’ indicated the filter type and ‘pattern’ is the filter regular expression. ‘type’ may contain the following characters:

The following filter file instructs fswatch to ignore all files except those ending in ‘.cpp’, ignoring case.

- .*
+i \.cpp$

4.8.2 Types of Filters and Order of Execution

Two types of filters are available:

As their name indicates, they are used to include and exclude paths from the monitored object list and from resulting events. fswatch processes filters this way:

Said another way:

4.8.3 Filter Modifiers

Filters are regular expression executed using the regcomp function which is able to interpret case-sensitive and case-insensitive basic and extended regular expressions as described in Base Definitions volume of IEEE Std 1003.1-2001, Chapter 9, Regular Expressions.

The (--insensitive, -I) option instructs fswatch to use case insensitive regular expressions. The following example adds an exclusion filter so that fswatch ignores any file system object whose name ends with .log, no matter the case.

$ fswatch -Ie ".*\.log$" ~

The (--extended, -E) option instructs fswatch to use extended regular expressions, such as:

$ fswatch -Ee "xl[st]+" ~

Treating the characteristics and the difference between different kinds of regular expressions is out of scope in this manual.

4.9 Filtering by Event Type

Events can be filtered by event type passing fswatch a list of event type names or masks to accept using the --event option:

$ fswatch -x --event Created --event Removed ~
$ fswatch -x --event 10 ~

In this example, the mask of the Created event is 2 and the mask of the Removed event is 8, so the mask of both events is 10.

If no event type filters are specified, fswatch will accept events of any type; on the other hand, as soon as a filter is specified, only events with a matching type will be accepted.

4.10 Latency

The latency l, expressed in seconds, is the amount of time that passes between the moment fswatch outputs a set of detected changes and the next. What happens during the time in-between is a monitor-specific implementation detail.

Some APIs, such as macOS’s FSEvents, implement the concept of latency themselves and fswatch appears idle in between. Only when the specified amount of time passes, change events are received, processed and written to standard output. Others, such as Linux’s inotify, do not12; in this case, the inotify monitor waits for events a maximum of l seconds; after that, the monitor logic loops again, performs house-keeping activities13 and starts waiting again.

The important thing to keep in mind is that latency and a monitor’s behaviour are implementation-dependent: check the documentation of the monitor you are using to get further information about how latency is handled.

4.11 Symbolic Links

Symbolic links are commonly used file system objects and, as it is customary for file system utilities, fswatch can either follow them and monitor the linked object14 or monitor the link itself.

4.12 Event Flags

Event flags identify the kind of change a file system object has undergone. Many of them directly map to common file system operations (such as creation, deletion, update, etc.), others are less common (such as attribute modification), and others are monitor and platform specific.

Currently, fswatch maps monitor-specific event flags to ‘global’ event flags acting as a sort of ‘greatest common denominator’ of all the available monitor flags. The list of all the available global event flags, defined in c/cevent.h, is the following:

NoOp

Idle event, optionally issued when no changes were detected.

PlatformSpecific

This event maps a platform-specific event that has no corresponding flag.

Created

The object has been created.

Updated

The object has been updated. The kind of update is monitor-dependent.

Removed

The object has been removed.

Renamed

The object has been renamed.

OwnerModified

The object’s owner has changed.

AttributeModified

An object’s attribute has changed.

MovedFrom

The object has moved from this location to a new location of the same file system.

MovedTo

The object has moved from another location in the same file system into this location.

IsFile

The object is a regular file.

IsDir

The object is a directory.

IsSymLink

The object is a symbolic link.

Link

The object link count has changed.

Overflow

The monitor has overflowed.

4.12.1 Peculiarities and Pitfalls

As you can see, the list of event flags contains element whose meaning is overlapping, at least partially. Link, for instance, may be equivalent to Create or Removed, depending on the whether the new link count is 1 or 0. MovedFrom and MovedTo may be equivalent to Create and Removed if the monitor is unable to discern a move operation has taken place (which is not always possible, as in the case of the poll monitor).

fswatch is unable to univocally map the specific flags of all the monitors consistently. Forcefully, the mapping depends on the capabilities of the monitor which, in turn, depend on the capabilities of the API being used.

For this reason, when processing change events, either the behaviour of the underlying monitor is known and taken into account, or all the flags which could possibly be attached at the operation being looked for must be taken into account.

Warning: As already explained (see Whitespace and Record Format), the record format when using event flags in textual form is ambiguous. For this reason, using numeric event flags (see Numeric Event Flags) or a custom record format (see Custom Record Formats) is recommended when fswatch output must be processed.

4.12.2 Numeric Event Flags

When using the (--numeric, -n) fswatch will output event flags in numeric format. A change event record may have multiple event flags and the numeric value is calculated as the bitwise or of the numeric values of all the flags. Since the value of an event flag is guaranteed to be unique and to be a number n in the form n = 2^k for a certain integer k, then the numeric value of a set of event flags is univocally determined.

To check whether a given event flag is present when processing fswatch output, iti s sufficient to check whether its bit is set to 1 in the event value. Let’s suppose we want to check whether the event flag whose value is e is present in a record whose flag numerical value is n. If the result r of

where \wedge is the bitwise and operator, is r > 0, then the flag e is present in n.

The numeric value of all the event flags is the following:

4.13 Choosing a Monitor

fswatch is a front-end to multiple monitors, each taking advantage of different monitoring APIs that may be available in a system. When building fswatch, configure scans the system to check which APIs are available and builds support for all of them.

A ‘special’ monitor, the poll monitor, manually scans the file system looking for differences. This is a fallback monitor for situations where other, more efficient APIs are not available. The poll monitor is available on any system providing the stat function.

Although fswatch chooses the ‘best’ monitor between the available ones, a user may wish to use another. A specific monitor can be chosen using the (--monitor, -m) option. The list of available monitors can be obtained using the (--list-monitors, -M) option or at the end of the help message:

$ fswatch --list-monitors
  fsevents_monitor
  kqueue_monitor
  poll_monitor
$ fswatch --help
[...]
Available monitors in this platform:

  fsevents_monitor
  kqueue_monitor
  poll_monitor
[...]

A monitor can then be chosen by passing the mandatory ‘name’ argument to the -m option:

$ fswatch -m kqueue_monitor ~

In this case, the ‘kqueue_monitor’ is manually chosen.

4.14 Recursive Scanning

fswatch’s behaviour when scanning a directory may vary on a monitor by monitor basis. The semantics of the (--recursive, -r) option is: recursively scan subdirectories. However, implementations may silently add ‘if the monitor does not do so already’. Since each monitor uses a different API, its behaviour depends on that of the backing API, and it is monitor-specific.

In general, users should always use the -r option according to its semantics, no matter what the monitor does. The only case when -r is ‘not’ honoured is when a monitor adds information by recursively monitoring children even when -r is not specified. Please notice that when this happens, there may be no performance overhead since the backing API is specifically designed to behave like this.

The authors think this is not a problem. If you think this behaviour can be improved, please fill a bug report (see Reporting Bugs and Suggestions).

4.14.1 Recursively Watching Directories

Some monitors such as the kqueue monitor require a file descriptor to be open for each watched file system object. This imposes a limitation on the maximum number of files that can be watched by fswatch. Before version 1.7, a user could only overcome this problem by increasing the maximum number of open file handles on its system.

fswatch 1.7.0 introduced a new option, -d/--directories; when this option is used with a monitor that supports it, only directory objects will be watched during recursive scans. When a change occurs on a file, instead of reporting which file has changed and how, fswatch will report a change event on the parent directory: this way, the number of required open file handles decreases at the expense of change event information granularity.

4.15 Monitor Tunables

Some monitors may accept monitor-specific parameters to tune their behaviour. To this purpose, fswatch offers a mechanism to pass key-value pair which are literally passed to the underlying monitor. A key-value pair (k, v) can be passed to a monitor using the --monitor-property option:

$ fswatch --monitor-property k=v ~

Multiple key-value pairs can be passed by using the --monitor-property option multiple times.


Footnotes

(4)

Depending on the monitor and options being used, fswatch may not exit unless stopped with a signal such as TERM or QUIT.

(5)

Exit codes are documented in c/error.h of libfswatch.

(6)

IFS (Internal Field Separators).

(7)

Although the available directive are much less than what printf offers.

(8)

Which is the same as considering escaped characters the result of a directive.

(9)

In the following example, the record termination character is not shown.

(10)

This behaviour is monitor-specific.

(11)

Whether an object whose path is matched by an exclusion filter is monitored or not is a monitor-specific implementation detail.

(12)

inotify publishes changes on a file identified by a descriptor which is read by fswatch.

(13)

Such as re-scanning objects which did not exist in the previous iteration.

(14)

When following links, the resolution is recursive: that is, if a link points to another symbolic link, this link is followed as well, and so on, until an object of a different kind is found.

(15)

But manually filtering out events based on paths, but fswatch does not do so by design.


Next: , Previous: , Up: Top   [Contents][Index]