Exnode specification: Difference between revisions

From ReddNet
Jump to navigation Jump to search
No edit summary
No edit summary
Line 23: Line 23:
== Erasure codings ==
== Erasure codings ==
*L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.
*L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.
**These allocation can be read directly by LoRS.
**These blocks can be read directly by LoRS.
*L-Store also writes extra erasure allocations to recover lost allocations.
*L-Store also writes extra erasure blocks to recover lost blocks.
**the XOR-ing to generate erasure allocation is calculated on the depots
**the XOR-ing to generate erasure block is calculated on the depots
***nfu operation stored on the depot
***nfu operation stored on the depot
**erasure allocations are also stored in the exnode file
**erasure blocks are also stored in the exnode file
**erasure allocations are only useful to the data-management area and not the data-client (see diagram)
**erasure blocks are only useful to the data-management area and not the data-client (see diagram)
**erasure allocations constitute an extension of the base (LoRS) exnode concept (or schema)
**erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema)
*L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.
*L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.
**LoRS compatible (the base schema)
**LoRS compatible (the base schema)
**LoRS does not have access to L-Store's capability to recover an allocation
**LoRS does not have access to L-Store's capability to recover a block
*The L-Store data-client (Lstcp) has no allocation recovery capability.
*The L-Store data-client (Lstcp) has no block recovery capability.
*data recovery, data warming, etc all data-management tasks, not data-client task.
*data recovery, data warming, etc all data-management tasks, not data-client task.


Revision as of 11:41, 18 January 2008

The exnode specification is implied by current usage in L-Store and LoRS.

Example LoRS exnode xml file

Example L-Store exnode xml file

Schema

The exnode xml is fairly unstructured and the functionality of a schema is minimal.

Simplified xml file

  • (removed xmlns to view structure more clearly)

Simple schema

  • In this example schema
    • The schema shows that only the element "mapping" has required elements.
    • The element metadata's use of attributes is unrestricted.
    • The only re-used type is the IBP_capability (read,write,manage)
  • L-Store additions
    • element "function"
    • erasure metadata "parity slice size"
    • additional erasure encoding info to be added here.

Erasure codings

  • L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.
    • These blocks can be read directly by LoRS.
  • L-Store also writes extra erasure blocks to recover lost blocks.
    • the XOR-ing to generate erasure block is calculated on the depots
      • nfu operation stored on the depot
    • erasure blocks are also stored in the exnode file
    • erasure blocks are only useful to the data-management area and not the data-client (see diagram)
    • erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema)
  • L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.
    • LoRS compatible (the base schema)
    • LoRS does not have access to L-Store's capability to recover a block
  • The L-Store data-client (Lstcp) has no block recovery capability.
  • data recovery, data warming, etc all data-management tasks, not data-client task.

Data client API & interfaces

  • command line tools for file management
    • file movement
    • file replication, augment, trim
  • command line tools for namespace management
    • list files
    • create directories, etc
  • LoRS API
    • uses the base exnode schema (LoRS)
    • reads local exnode files
    • talks to LoDN with a combined http:/LoDN protocol
    • C/C++ Posix and posix-like API
  • Lstcp API (not developed yet)
    • does not parse exnodes?
    • talks to L-Server with the L-Store protocol
    • receives IBP commands individually from L-Server?
      • sends these commands directly to IBP depots?
  • typical C/C++ API needs (proposed)
    • simple offsets and lengths
    • non-blocked (vector) reads
Retrieved from "http://www.reddnet.org/mwiki/index.php?title=Exnode_specification&oldid=3088"