Exnode specification: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
(→Schema) |
||
(49 intermediate revisions by the same user not shown) | |||
Line 2: | Line 2: | ||
[[Example LoRS exnode xml file]] | [[Example LoRS exnode xml file]] | ||
* 1 namespace: exnode | |||
* 6 elements: exnode, mapping, metadata, read, write, manage | |||
* element metadata uses 2 attibutes: name, type | |||
** 5 values used for attribute "name:" alloc_length, alloc_offset, e2e_blocksize, exnode_offset, logical_length | |||
** 1 value used for attribute "type:" integer | |||
[[Example L-Store xml file]] | [[Example L-Store exnode xml file -- parity off]] | ||
[[Example L-Store exnode xml file -- with parity blocks]] | |||
*original implementation -- being modified | |||
== Schema == | |||
[[Simplified xml file]] ( | The exnode xml is fairly unstructured and the functionality of a schema is minimal. | ||
[[Simplified xml file]] | |||
*(removed xmlns to view structure more clearly) | |||
[[Simple schema]] | [[Simple schema]] | ||
*In this example schema | |||
**The schema shows that only the element "mapping" has required elements. | |||
**The element metadata's use of attributes is unrestricted. | |||
**The only re-used type is the IBP_capability (read,write,manage) | |||
*extra-schema assumptions | |||
**The exnode mapping order corresponds to the file block order | |||
*L-Store additions | |||
**element "function" (a tmp space holder to bypass a java bug) | |||
**erasure metadata attribute "name" with a value of "parity slice size" | |||
**element "parity" is the mapping for the parity block if parity slice size != -1 | |||
***the parity mapping immediately follows the data mapping | |||
== Erasure codings == | |||
*L-Store writes data in its original format (possibly encrypted, etc), just like LoRS. | |||
**These blocks can be read directly by LoRS. | |||
*L-Store also writes extra erasure blocks to recover lost blocks. | |||
**the XOR-ing to generate erasure block is calculated on the depots | |||
***nfu operation stored on the depot | |||
**erasure blocks are also stored in the exnode file | |||
**erasure blocks are only useful to the data-management area and not the data-client (see diagram) | |||
**erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema) | |||
*L-Store can write a LoRS compatible exnode file that excludes the erasure allocations. | |||
**LoRS compatible (the base schema) | |||
**LoRS does not have access to L-Store's capability to recover a block | |||
*The L-Store data-client (Lstcp) has no block recovery capability. | |||
*data recovery, data warming, etc all data-management tasks, not data-client task. | |||
== Data client API & interfaces == | |||
*command line tools for file management | |||
**file movement | |||
**file replication, augment, trim | |||
*command line tools for namespace management | |||
**list files | |||
**create directories, etc | |||
*LoRS API | |||
**uses the base exnode schema (LoRS) | |||
**reads local exnode files | |||
**talks to LoDN with a combined http:/LoDN protocol | |||
**C/C++ Posix and posix-like API | |||
*Lstcp API (not developed yet) | |||
**does not parse exnodes? | |||
**talks to L-Server with the L-Store protocol | |||
**receives IBP commands individually from L-Server? | |||
***sends these commands directly to IBP depots? | |||
*typical C/C++ API needs (proposed) | |||
**simple offsets and lengths | |||
**non-blocked (vector) reads |
Latest revision as of 11:49, 29 January 2008
The exnode specification is implied by current usage in L-Store and LoRS.
- 1 namespace: exnode
- 6 elements: exnode, mapping, metadata, read, write, manage
- element metadata uses 2 attibutes: name, type
- 5 values used for attribute "name:" alloc_length, alloc_offset, e2e_blocksize, exnode_offset, logical_length
- 1 value used for attribute "type:" integer
Example L-Store exnode xml file -- parity off
Example L-Store exnode xml file -- with parity blocks
- original implementation -- being modified
Schema
The exnode xml is fairly unstructured and the functionality of a schema is minimal.
- (removed xmlns to view structure more clearly)
- In this example schema
- The schema shows that only the element "mapping" has required elements.
- The element metadata's use of attributes is unrestricted.
- The only re-used type is the IBP_capability (read,write,manage)
- extra-schema assumptions
- The exnode mapping order corresponds to the file block order
- L-Store additions
- element "function" (a tmp space holder to bypass a java bug)
- erasure metadata attribute "name" with a value of "parity slice size"
- element "parity" is the mapping for the parity block if parity slice size != -1
- the parity mapping immediately follows the data mapping
Erasure codings
- L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.
- These blocks can be read directly by LoRS.
- L-Store also writes extra erasure blocks to recover lost blocks.
- the XOR-ing to generate erasure block is calculated on the depots
- nfu operation stored on the depot
- erasure blocks are also stored in the exnode file
- erasure blocks are only useful to the data-management area and not the data-client (see diagram)
- erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema)
- the XOR-ing to generate erasure block is calculated on the depots
- L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.
- LoRS compatible (the base schema)
- LoRS does not have access to L-Store's capability to recover a block
- The L-Store data-client (Lstcp) has no block recovery capability.
- data recovery, data warming, etc all data-management tasks, not data-client task.
Data client API & interfaces
- command line tools for file management
- file movement
- file replication, augment, trim
- command line tools for namespace management
- list files
- create directories, etc
- LoRS API
- uses the base exnode schema (LoRS)
- reads local exnode files
- talks to LoDN with a combined http:/LoDN protocol
- C/C++ Posix and posix-like API
- Lstcp API (not developed yet)
- does not parse exnodes?
- talks to L-Server with the L-Store protocol
- receives IBP commands individually from L-Server?
- sends these commands directly to IBP depots?
- typical C/C++ API needs (proposed)
- simple offsets and lengths
- non-blocked (vector) reads