Results 1 to 2 of 2

Thread: Internal FileSystem layer

  1. #1

    Default Internal FileSystem layer

    Hello,

    I want to ask why you need intermediate file system layer? As I saw in sources this is additional file system implementation over real file system. And this implementation hasn't journaling, only some cache capabilities. Why you need this overhead and real file system duplication?(taking in mind that now exist advanced NTFS on windows and ext4, ZFS ,etc on linux file systems) Because without journaling on this internal files system layer it is possible to crash internal file system structure.

    Thanks in advance.

    Regards,
    Saulius

  2. #2

    Default

    The file system layer provides a lightweight independent huge set of arranged linear spaces. This is the main responsibility of the internal file system. It has to provide a very lightweight sparse file management mechanism.

    The STS File System manage files, called RawFiles. Each RawFile is a sparse file, that provide an isolated linear space of bytes.

    On the upper layer, the STSdb engine is a tree of XTables. Each XTable is a transactional sparse radix tree. (This is the reason, that the XTable is much faster than any other B or B+ based tables.) Each XTable is placed over a single RawFile.

    The internal file system has no journaling, but each XTable is transactional. This is achieved thanks to the XTable behaviour over the back-ended RawFile and the RawFile arrangement over the physical file, holding the file system.

    In XTable every record modification is made not in-place. The engine never writes a modified record on the same place. This way, from the moment of table opening to the moment of table committing - the transaction session - we have both the original and the modified tree of the table content. The two trees of the XTable instance do share all unmodified sub-trees of records. So the transaction changes never touch the original records, it never touch the original XTable root! (As consequence of that, we achieve also the possibility to have an instant table snapshots and table variations.)

    When the XTable commit process is started, the modified tree starts storing itself bottom to top - from the new leafs to the new root. If there have been errors - the original data remain, the original root remain; if not - after saving the new root, we switch the active header of the XTable in the back-ended RawFile. That way, after all, all the transaction safety is done by storing or not the switch, indicating who is the active XTable header, respectively where in the RawFile is the XTable root.

    The internal file system is formatted on sectors by 512 bytes. And the above two headers of the XTable plus the switch, indicating who is the active one, are always placed in one sector - the first one in the RawFile. In turn, the NTFS, holding the file with the STS File System, is formatted on 4K, 8K etc. allocation units. So our internal sectors are aligned to the file allocation units of the OS file system. But the allocation units are strictly arranged into the physical HDD sectors. Consequently, our internal file system sector is multiple on to the physical HDD sector. Finally, the most of (modern) hard disks guarantee the atomic writes of a single physical sector. In that way we ensures the XTable ACID transaction!

    So, on one hand, we need the internal file system to have arranged sparse files. Of course, this is not critical for the database layer, we can always make it to work directly with NTFS files, instead of using the RawFiles (the NTFS also supports sparse files). But when the number of XTables exceed a several of millions, the bottleneck of the whole engine will be exactly the overhead of having such number of physical files. Of course, we can always group them to reduce the count of files, but it will be a half-solution.

    Having a STS File System provides a few big advantages:
    • we can have millions of XTables without overhead of having such number of real files (the database is designed to operate with such scales of data);
    • we can have BLOB fields in the XTables with practically unlimited size;
    • advanced users may combine using the database with using the internal file system. The file system is not hidden, and even it could be used independently from the database engine;
    • the database may be placed over a several files or hard disk drives;
    • in the future stage - we can totally bypass the real file system, by writing an appropriate driver...


    These are the main reasons, that we need an internal file system.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
2002 - 2014 STS Soft SC. All Rights reserved.
STSdb, Waterfall Tree and WTree are registered trademarks of STS Soft SC.