The Ceres Database

Ceres is a time-series database format intended to replace Whisper as the default storage format for Graphite. In contrast with Whisper, Ceres is not a fixed-size database and is designed to better support sparse data of arbitrary fixed-size resolutions. This allows Graphite to distribute individual time-series across multiple servers or mounts.

Ceres is not actively developped at the moment. For alternatives to whisper look at alternative storage backends.

Storage Overview

Ceres databases are comprised of a single tree contained within a single path on disk that stores all metrics in nesting directories as nodes.

A Ceres node represents a single time-series metric, and is composed of at least two data files. A slice to store all data points, and an arbitrary key-value metadata file. The minimum required metadata a node needs is a 'timeStep'. This setting is the finest resolution that can be used for writing. A Ceres node however can contain and read data with other, less-precise values in its underlying slice data.

Other metadata keys that may be set for compatibility with Graphite are 'retentions' , 'xFilesFacter', and 'aggregationMethod'.

A Ceres slice contains the actual data points in a file. The only other information a slice holds is the timestamp of the oldest data point, and the resolution. Both of which are encoded as part of its filename in the format timestamp@resolution.

Data points in Ceres are stored on-disk as a contiguous list of big-endian double-precision floats. The timestamp of a datapoint is not stored with the value, rather it is calculated by using the timestamp of the slice plus the index offset of the value multiplied by the resolution.

The timestamp is the number of seconds since the UNIX Epoch (01-01-1970). The data value is parsed by the Python float() function and as such behaves in the same way for special strings such as 'inf'. Maximum and minimum values are determined by the Python interpreter’s allowable range for float values which can be found by executing:

python -c 'import sys; print sys.float_info'

Slices: Precision and Fragmentation

Ceres databases contain one or more slices, each with a specific data resolution and a timestamp to mark the beginning of the slice. Slices are ordered from the most recent timestamp to the oldest timestamp. Resolution of data is not considered when reading from a slice, only that when writing a slice with the finest precision configured for the node exists.

Gaps in data are handled in Ceres by padding slices with null datapoints. If the slice gap however is too big, then a new slice is instead created. If a Ceres node accumulates too many slices, read performance can suffer. This can be caused by intermittently reported data. To mitigate slice fragmentation there is a tolerance for how much space can be wasted within a slice file to avoid creating a new one. That tolerance level is determined by 'MAX_SLICE_GAP', which is the number of consecutive null datapoints allowed in a slice file.

If set very low, Ceres will waste less of the tiny bit disk space that this feature wastes, but then will be prone to performance problems caused by slice fragmentation, which can be pretty severe.

If set really high, Ceres will waste a bit more disk space. Although each null datapoint wastes 8 bytes, you must keep in mind your filesystem’s block size. If you suffer slice fragmentation issues, you should increase this or defrag the data more often. However you should not set it to be huge because then if a large but allowed gap occurs it has to get filled in, which means instead of a simple 8-byte write to a new file we could end up doing an (8 * MAX_SLICE_GAP)-byte write to the latest slice.

Rollup Aggregation

Expected features such as roll-up aggregation and data expiration are not provided by Ceres itself, but instead are implemented as maintenance plugins.

Such a rollup plugin exists in Ceres that aggregates data points in a way that is similar behavior of Whisper archives. Where multiple data points are collapsed and written to a lower precision slice, and data points outside of the set slice retentions are trimmed. By default, an average function is used, however alternative methods can be chosen by changing the metadata.

Retrieval Behavior

When data is retrieved (scoped by a time range), the first slice which has data within the requested interval is used. If the time period overlaps a slice boundary, then both slices are read, with their values joined together. Any missing data between them are filled with null data points.

There is currently no support in Ceres for handling slices with mixed resolutions in the same way that is done with Whisper archives.

Database Format

CeresSlice Data
Data Point+

Data types in Python’s struct format:

Point !d

Metadata for Ceres is stored in JSON format:

{“retentions”: [[30, 1440]], “timeStep”: 30, “xFilesFactor”: 0.5, “aggregationMethod”: “average”}

An Easy Way to Hide Files and Directories in Linux

Do you occasionally share your Linux desktop machine with family members, friends or perhaps with colleagues at your workplace, then you have a reason to hide certain private files as well as folders or directories. The question is how can you do this?

In this tutorial, we will explain an easy and effective way to hide files and directories and view hidden files/directories in Linux from the terminal and GUI.

As we’ll see below, hiding files and directories in Linux is so simple.

How to Hide Files and Directories in Linux

To hide a file or directory from the terminal, simply append a dot . at the start of its name as follows using the mv command.

$ ls
$ mv mv sync.ffs_db .sync.ffs_db
$ ls

Hide File in Linux Terminal

Using GUI method, the same idea applies here, just rename the file by adding a . at the start of its name as shown below.

Hide File in Linux Using File Manager

Once you have renamed it, the file will still be seen, move out of the directory and open it again, it will be hidden thereafter.

How to View Hide Files and Directories in Linux

To view hidden files, run the ls command with the -a flag which enables viewing of all files in a directory or -al flag for long listing.

$ ls -a
OR
$ ls -al

View Hidden Files in Linux Terminal

From a GUI file manager, go to View and check the option Show Hidden Files to view hidden files or directories.

View Hidden File Using File Manager

How to Compress Files and Directories with a Password

In order to add a little security to your hidden files, you can compress them with a password and then hide them from a GUI file manager as follows.

Select the file or directory and right click on it, then choose Compress from the menu list, after seeing the compression preferences interface, click on “Other options” to get the password option as shown in the screenshot below.

Once you have set the password, click on Create.

Compress Files with Password in Linux

From now on, each time anyone wants to open the file, they’ll be asked to provide the password created above.

Enter Password to View Files

Now you can hide the file by renaming it with a . as we explained before.

fswatch – Monitors Files and Directory Changes or Modifications in Linux

fswatch is a cross-platform, file change monitor that gets notification alerts when the contents of the specified files or directories are altered or modified.

It executes four types of monitors on different operating systems such as:

  1. A monitor build on the File System Events API of Apple OS X.
  2. A monitor based on kqueue, a notification interface present in FreeBSD 4.1 also supported on many *BSD systems, OS X inclusive.
  3. A monitor based on File Events Notification API of the Solaris kernel plus its spin-offs.
  4. A monitor based on inotify, a kernel subsystem that shows file system modifications to apps.
  5. A monitor based on ReadDirectoryChangesW, a Windows API that records alters to a directory.
  6. A monitor that regularly check that status of file system, keeps file modification times in memory, and manually determine file system changes (which works anywhere, where stat can be used).

Features of fswatch

  1. Supports several OS-specific APIs
  2. Allows recursive directory monitoring
  3. Performs path filtering using including and excluding regular expressions
  4. Supports customizable record format
  5. Additionally, it supports periodic idle events

How To Install fswatch in Linux Systems

Unfortunately, fswatch package is not available to install from the default system repositories in any Linux distributions. The only way to install the latest version of fswatch is to build from source tarball as show in the following installation instructions.

First grab the latest fswatch tarball using following wget command and install it as shown:

$ wget https://github.com/emcrisostomo/fswatch/releases/download/1.9.3/fswatch-1.9.3.tar.gz
$ tar -xvzf fswatch-1.9.3.tar.gz
$ cd fswatch-1.9.3
$ ./configure
$ make
$ sudo make install 

Important: Make sure you’ve GNU GCC (C and C++ Compiler) and Development Tools (build-essential on Debian/Ubuntu) installed on the system, before you compile fswatch from source. If not, install it using following command on your respective Linux distributions..

# yum group install 'Development Tools'		[On CentOS/RHEL]
# dnf group install 'Development Tools'		[On Fedora 22+ Versions]
$ sudo apt-get install build-essential          [On Debian/Ubuntu Versions]

On Debian/Ubuntu distributions, you might get following error while executing fswatch command..

fswatch: error while loading shared libraries: libfswatch.so.6: cannot open shared object file: No such file or directory

To fix it, you need to execute the command below, this will help refresh the links and cache to the dynamic libraries before you can start using fswatch.

$ sudo ldconfig

How do I use fswatch on Linux?

The general syntax for running fswatch is:

$ fswatch [option] [path]

On Linux, it is recommended that you use the default inotify monitor, you can list available monitors by employing the -M or - list-monitors option:

$ fswatch -M
$ fswatch --list-monitors

fswatch - List Monitors

The command below enables you to watch the changes in the current directory (/home/tecmint), with events being delivered to standard output every 4 seconds.

The -l or –-latency option allows you to set the latency in seconds, the default being 1 second.

$ fswatch -l 4 .

fswatch - Monitor Home Directory Changes

The next command monitors changes to the /var/log/auth.log file every 5 seconds:

$ fswatch -l 5 /var/log/auth.log

Using -t or --timestamp option prints the time stamp for every event, to print the time in UTC format, employ -u or --utf-time option. You can as well format time using -f or --format-time format option:

$ fswatch --timestamp /var/log/auth.log

Next, -x or --event-flags tells fswatch to print the event flags along side the event path. You can use –event-field-seperator option to print events using the particular separator.

$ fswatch --events-flags ~ /var/log/auth.log

To print the numeric value of an event indicating changes in your home directory and /var/log/auth.log file, use -n or --numeric option as below:

$ fswatch --numeric ~ /var/log/auth.log 

Perhaps you can look through the fswatch man page for detailed usage options and information:

$ man fswatch