https://www.reddnet.org/mwiki/api.php?action=feedcontributions&feedformat=atom&user=Dano 2026-05-25T00:42:26Z User contributions MediaWiki 1.39.3 https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3582 2008-07-02T14:07:38Z

<p>Dano: /* NFU checksum approach */</p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> ***This is problematic since a file can be replicated with different allocation boundaries leading to different top hash for the file.<br /> <br /> *It seems to be common in distributed file systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that <br /> **the checksum method becomes part of the IBP protocol itself <br /> **and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> ** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) or with mmaps for NFU servers integrated into IBP depot. <br /> *This may be slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3581 2008-07-02T14:02:42Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * Three components:<br /> ** NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> ** NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> ** NFU client -- initiates launch of an nfu Op and exchanges data with it.<br /> * NFU API -- Two iotypes categories for data exchange between nfu_op and nfu_client:<br /> ** IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** IBP_REF_[RD, WR, RDWR]<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfu Ops<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges <br /> *** this forces the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU iotypes</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3580 2008-07-02T14:01:53Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfu Op and exchanges data with it.<br /> * NFU API -- Two iotypes categories for data exchange between nfu_op and nfu_client:<br /> ** IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** IBP_REF_[RD, WR, RDWR]<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfu Ops<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges <br /> *** this forces the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU iotypes</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3579 2008-07-02T14:01:15Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfu Op and exchanges data with it.<br /> * NFU API -- Two iotypes for data exchange between nfu_op and nfu_client:<br /> ** IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** IBP_REF_[RD, WR, RDWR]<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfu Ops<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges <br /> *** this forces the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU iotypes</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3578 2008-07-02T14:00:07Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfu Op and exchanges data with it.<br /> * NFU API -- Two iotypes for data exchange between nfu_op and nfu_client:<br /> ** IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** IBP_REF_[RD, WR, RDWR]<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfu Ops<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3577 2008-07-02T13:58:52Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfu Op and exchanges data with it.<br /> * NFU API -- Two iotypes for data exchange between nfu_op and nfu_client:<br /> ** IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** IBP_REF_[RD, WR, RDWR]<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3576 2008-07-02T13:58:35Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfu Op and exchanges data with it.<br /> * NFU API -- Two iotypes for data exchange between nfu_op and nfu_client:<br /> ** 1. IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP_REF_[RD, WR, RDWR]<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3575 2008-07-02T13:57:51Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfu Op and exchanges data with it.<br /> * NFU API:<br /> ** Two iotypes for data exchange between nfu_op and nfu_client:<br /> ** 1. IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP_REF_[RD, WR, RDWR]<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3574 2008-07-02T10:20:45Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfuOp and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which would otherwise be processed by the nfuServer.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3573 2008-07-02T08:45:28Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfuOp and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, i.e. nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which otherwise are processed by the nfuServer and hidden from the nfuOp.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3572 2008-07-02T08:44:47Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfuOp and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> ** UTK NFU server mmaps allocations from readCaps, nfuOp has no direct knowledge of allocation file names.<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which otherwise are processed by the nfuServer and hidden from the nfuOp.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3571 2008-07-02T08:43:58Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfuOp and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> * compatibility issues:<br /> ** UTK NFU server mmaps allocations from readCaps, nfuOp has no direct knowledge of allocation file names.<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which otherwise are processed by the nfuServer and hidden from the nfuOp.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3570 2008-07-02T08:42:41Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfuOp and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps which otherwise are processed by the nfuServer and hidden from the nfuOp.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3569 2008-07-02T08:35:21Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot, runs NFU Ops<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- initiates launch of an nfuOp and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3568 2008-07-01T13:52:32Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps.<br /> *** this disables 1/2 of the NFU api</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3567 2008-07-01T13:51:59Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server<br /> * compatibility issues:<br /> ** Java NFU server currently has no way to run C nfuOps<br /> ** if nfu server is not part of the ibp depot, then all IBP_REF_* exchanges need to be changed to IBP_VAL_* exchanges forcing the nfuOp to process IBP caps.</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3566 2008-07-01T13:49:27Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> *** NFU Server (not nfuOp) processes IBP Caps<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3565 2008-07-01T13:48:30Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> * NFU API:<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3564 2008-07-01T13:47:49Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> * security issues<br /> ** Java NFU server has VM isolation of nfuOp<br /> ** UTK NFU server has option to run Cap as process to isolate nfuOp from server</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3563 2008-07-01T13:47:27Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string<br /> <br /> * Java NFU server has VM isolation of nfuOp<br /> * UTK NFU server has option to run Cap as process to isolate nfuOp from server</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3562 2008-07-01T13:46:22Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***E.G. _RD: client sends readCap to server, NFU server provides nfuOp a pointer to mmap of the allocation<br /> ***nfuOp does not receive readCap string</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3561 2008-07-01T13:41:29Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN, OUT, INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> ***IBP_REF_[RD, WR, RDWR]<br /> ***client sends an IBP Cap to server, NFU server returns pointer to mmap of the allocation</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3560 2008-07-01T13:41:09Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> ** 1. variable exchange<br /> ***IBP_VAL_[IN,OUT,INOUT]<br /> *** variable/values exchanged directly between nfu client and nfu Op<br /> ** 2. IBP Cap handling<br /> ***IBP_REF_[RD,WR,RDWR]<br /> ***client sends an IBP Cap to server, NFU server returns pointer to mmap of the allocation</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3559 2008-07-01T13:40:40Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> *** 1. variable exchange<br /> ****IBP_VAL_[IN,OUT,INOUT]<br /> variable/values exchanged directly between nfu client and nfu Op<br /> *** 2. IBP Cap handling<br /> ****IBP_REF_[RD,WR,RDWR]<br /> ****client sends an IBP Cap to server, NFU server returns pointer to mmap of the allocation</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3558 2008-07-01T13:39:03Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> *** 1. variable exchange -- data passed directly between NFU Client and NFU Op<br /> IBP_VAL_[IN,OUT,INOUT]<br /> *** 2. IBP Cap handling<br /> IBP_REF_[RD,WR,RDWR]<br /> client sends an IBP Cap to server, NFU server returns pointer to mmap of the allocation</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3557 2008-07-01T13:30:42Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> *** 1. variable exchange -- data passed directly between NFU Client and NFU Op<br /> NFU_VAL_IN<br /> *** 2. IBP Cap handling<br /> NFU_IBP_RD, client sends an IBP Cap to server, NFU Op does not receive cap, but a pointer to a mmap of the cap.</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3556 2008-07-01T13:28:53Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> **1. variable exchange -- data passed directly between NFU Client and NFU Op<br /> NFU_VAL_IN<br /> **2. IBP Cap handling<br /> NFU_IBP_RD, client sends an IBP Cap to server, NFU Op does not receive cap, but a pointer to a mmap of the cap.</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3555 2008-07-01T13:28:12Z

<p>Dano: </p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** Two types of data exchange:<br /> 1. variable exchange -- data passed directly between NFU Client and NFU Op<br /> NFU_VAL_IN<br /> 2. IBP Cap handling<br /> NFU_IBP_RD, client sends an IBP Cap to server, NFU Op does not receive cap, but a pointer to a mmap of the cap.</div>

Dano https://www.reddnet.org/mwiki/index.php?title=NFU_Specification&diff=3554 2008-07-01T13:24:45Z

<p>Dano: New page: Simple overview: * NFU server -- &quot;adjacent&quot; to IBP depot. * NFU Op -- lightweight program, written in C, or Java, depending on server. * NFU client -- launches nfuOps and exchanges data wi...</p> <hr /> <div>Simple overview:<br /> * NFU server -- &quot;adjacent&quot; to IBP depot.<br /> * NFU Op -- lightweight program, written in C, or Java, depending on server.<br /> * NFU client -- launches nfuOps and exchanges data with it.<br /> ** 2 types of data exchange:<br /> ** 1. variable exchange -- data passed directly between NFU Client and NFU Op<br /> NFU_VAL_IN<br /> ** 2. IBP Cap handling<br /> NFU_IBP_RD, client sends an IBP Cap to server, NFU Op does not receive cap, but a pointer to a mmap of the cap.</div>

Dano https://www.reddnet.org/mwiki/index.php?title=All_Hands_Meeting,_Feb_1,_2008&diff=3224 2008-01-31T23:44:48Z

<p>Dano: </p> <hr /> <div>==Agenda for All-Hands==<br /> <br /> Meeting Purpose: &quot;To decide upon a plan of action that will make REDDnet a success.&quot;<br /> <br /> The Meeting will be held at ACCRE at Vanderbilt University<br /> <br /> ===Morning Session 9-12:30===<br /> <br /> ====09:00-09:30 --- Meeting Kickoff====<br /> <br /> * Summary of the proposal - towards a vision for REDDnet - Terry<br /> * Putting together a plan of action - Paul <br /> <br /> ====09:30-10:00 --- Deployment and Operations====<br /> <br /> * [[Operations and Deployment|Plan of Work]] - Bobby<br /> * What should we expect of REDDnet sites? What should they expect of us?<br /> <br /> ====10:00-10:30 --- Applications====<br /> * Application Liaison Effort - Plan of Work (Carie Lee)<br /> * [[Demo Projects and Application Use]] - Plans (such as CMS) (Dan)<br /> <br /> ====10:30-10:45 --- Coffee Break====<br /> <br /> ====10:45-12:30 --- A Toolbox for Distributed Storage (Part 1)====<br /> * Data and Directory Services Discussion<br /> ** [[Directory and data management services APIs|Architectural Issues]] - Larry <br /> ** How we did things in the tools we developed at UTK - Chris<br /> ** TSSP - Hunter<br /> * IBP: [[Presentation information|Current Status and Proposed Extensions]] - Alan<br /> <br /> ===Lunch 12:30-01:30===<br /> * Lunch will be provided at the Vanderbilt Commons<br /> <br /> ===Afternoon Session 1:30-05:00===<br /> <br /> ====01:30-03:15 --- A Toolbox for Distributed Storage (Part 2)====<br /> <br /> * continue discussion listed above.<br /> <br /> ====03:15-03:30 --- Coffee Break====<br /> <br /> ====03:30-05:00 --- Close-Out Discussion and Assignments====<br /> * tasks to be done<br /> * prioritize tasks<br /> * build a software diagram? (straw man is at [[Protocol Standardization Efforts]])</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3216 2008-01-29T17:39:06Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> ***This is problematic since a file can be replicated with different allocation boundaries leading to different top hash for the file.<br /> <br /> *It seems to be common in distributed file systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that <br /> **the checksum method becomes part of the IBP protocol itself <br /> **and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> ** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) <br /> *This would be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Exnode_specification&diff=3215 2008-01-29T16:49:30Z

<p>Dano: /* Schema */</p> <hr /> <div>The exnode specification is implied by current usage in L-Store and LoRS.<br /> <br /> [[Example LoRS exnode xml file]]<br /> * 1 namespace: exnode<br /> * 6 elements: exnode, mapping, metadata, read, write, manage<br /> * element metadata uses 2 attibutes: name, type<br /> ** 5 values used for attribute &quot;name:&quot; alloc_length, alloc_offset, e2e_blocksize, exnode_offset, logical_length<br /> ** 1 value used for attribute &quot;type:&quot; integer<br /> <br /> [[Example L-Store exnode xml file -- parity off]]<br /> <br /> [[Example L-Store exnode xml file -- with parity blocks]]<br /> *original implementation -- being modified<br /> <br /> == Schema ==<br /> <br /> The exnode xml is fairly unstructured and the functionality of a schema is minimal.<br /> <br /> [[Simplified xml file]]<br /> *(removed xmlns to view structure more clearly)<br /> <br /> [[Simple schema]]<br /> *In this example schema<br /> **The schema shows that only the element &quot;mapping&quot; has required elements.<br /> **The element metadata's use of attributes is unrestricted.<br /> **The only re-used type is the IBP_capability (read,write,manage)<br /> *extra-schema assumptions<br /> **The exnode mapping order corresponds to the file block order<br /> *L-Store additions<br /> **element &quot;function&quot; (a tmp space holder to bypass a java bug)<br /> **erasure metadata attribute &quot;name&quot; with a value of &quot;parity slice size&quot;<br /> **element &quot;parity&quot; is the mapping for the parity block if parity slice size != -1<br /> ***the parity mapping immediately follows the data mapping<br /> <br /> == Erasure codings ==<br /> *L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.<br /> **These blocks can be read directly by LoRS.<br /> *L-Store also writes extra erasure blocks to recover lost blocks.<br /> **the XOR-ing to generate erasure block is calculated on the depots<br /> ***nfu operation stored on the depot<br /> **erasure blocks are also stored in the exnode file<br /> **erasure blocks are only useful to the data-management area and not the data-client (see diagram)<br /> **erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema)<br /> *L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.<br /> **LoRS compatible (the base schema)<br /> **LoRS does not have access to L-Store's capability to recover a block<br /> *The L-Store data-client (Lstcp) has no block recovery capability.<br /> *data recovery, data warming, etc all data-management tasks, not data-client task.<br /> <br /> == Data client API &amp; interfaces ==<br /> *command line tools for file management<br /> **file movement<br /> **file replication, augment, trim<br /> *command line tools for namespace management<br /> **list files<br /> **create directories, etc<br /> *LoRS API<br /> **uses the base exnode schema (LoRS)<br /> **reads local exnode files<br /> **talks to LoDN with a combined http:/LoDN protocol<br /> **C/C++ Posix and posix-like API<br /> *Lstcp API (not developed yet)<br /> **does not parse exnodes?<br /> **talks to L-Server with the L-Store protocol<br /> **receives IBP commands individually from L-Server?<br /> ***sends these commands directly to IBP depots?<br /> *typical C/C++ API needs (proposed)<br /> **simple offsets and lengths<br /> **non-blocked (vector) reads</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Exnode_specification&diff=3214 2008-01-29T16:47:37Z

<p>Dano: </p> <hr /> <div>The exnode specification is implied by current usage in L-Store and LoRS.<br /> <br /> [[Example LoRS exnode xml file]]<br /> * 1 namespace: exnode<br /> * 6 elements: exnode, mapping, metadata, read, write, manage<br /> * element metadata uses 2 attibutes: name, type<br /> ** 5 values used for attribute &quot;name:&quot; alloc_length, alloc_offset, e2e_blocksize, exnode_offset, logical_length<br /> ** 1 value used for attribute &quot;type:&quot; integer<br /> <br /> [[Example L-Store exnode xml file -- parity off]]<br /> <br /> [[Example L-Store exnode xml file -- with parity blocks]]<br /> *original implementation -- being modified<br /> <br /> == Schema ==<br /> <br /> The exnode xml is fairly unstructured and the functionality of a schema is minimal.<br /> <br /> [[Simplified xml file]]<br /> *(removed xmlns to view structure more clearly)<br /> <br /> [[Simple schema]]<br /> *In this example schema<br /> **The schema shows that only the element &quot;mapping&quot; has required elements.<br /> **The element metadata's use of attributes is unrestricted.<br /> **The only re-used type is the IBP_capability (read,write,manage)<br /> *extra-schema assumptions<br /> **The exnode mapping order corresponds to the file block order<br /> *L-Store additions<br /> **element &quot;function&quot; (a tmp space holder to bypass a java bug)<br /> **erasure metadata &quot;parity slice size&quot;<br /> **element &quot;parity&quot; is the mapping for the parity block if parity slice size != -1<br /> ***the parity mapping immediately follows the data mapping<br /> == Erasure codings ==<br /> *L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.<br /> **These blocks can be read directly by LoRS.<br /> *L-Store also writes extra erasure blocks to recover lost blocks.<br /> **the XOR-ing to generate erasure block is calculated on the depots<br /> ***nfu operation stored on the depot<br /> **erasure blocks are also stored in the exnode file<br /> **erasure blocks are only useful to the data-management area and not the data-client (see diagram)<br /> **erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema)<br /> *L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.<br /> **LoRS compatible (the base schema)<br /> **LoRS does not have access to L-Store's capability to recover a block<br /> *The L-Store data-client (Lstcp) has no block recovery capability.<br /> *data recovery, data warming, etc all data-management tasks, not data-client task.<br /> <br /> == Data client API &amp; interfaces ==<br /> *command line tools for file management<br /> **file movement<br /> **file replication, augment, trim<br /> *command line tools for namespace management<br /> **list files<br /> **create directories, etc<br /> *LoRS API<br /> **uses the base exnode schema (LoRS)<br /> **reads local exnode files<br /> **talks to LoDN with a combined http:/LoDN protocol<br /> **C/C++ Posix and posix-like API<br /> *Lstcp API (not developed yet)<br /> **does not parse exnodes?<br /> **talks to L-Server with the L-Store protocol<br /> **receives IBP commands individually from L-Server?<br /> ***sends these commands directly to IBP depots?<br /> *typical C/C++ API needs (proposed)<br /> **simple offsets and lengths<br /> **non-blocked (vector) reads</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Exnode_specification&diff=3213 2008-01-29T16:45:53Z

<p>Dano: </p> <hr /> <div>The exnode specification is implied by current usage in L-Store and LoRS.<br /> <br /> [[Example LoRS exnode xml file]]<br /> * 1 namespace: exnode<br /> * 6 elements: exnode, mapping, metadata, read, write, manage<br /> * element metadata uses 2 attibutes: name, type<br /> ** 5 attribute name values: alloc_length, alloc_offset, e2e_blocksize, exnode_offset, logical_length<br /> ** 1 attribute type value: integer<br /> <br /> [[Example L-Store exnode xml file -- parity off]]<br /> <br /> [[Example L-Store exnode xml file -- with parity blocks]]<br /> *original implementation -- being modified<br /> <br /> == Schema ==<br /> <br /> The exnode xml is fairly unstructured and the functionality of a schema is minimal.<br /> <br /> [[Simplified xml file]]<br /> *(removed xmlns to view structure more clearly)<br /> <br /> [[Simple schema]]<br /> *In this example schema<br /> **The schema shows that only the element &quot;mapping&quot; has required elements.<br /> **The element metadata's use of attributes is unrestricted.<br /> **The only re-used type is the IBP_capability (read,write,manage)<br /> *extra-schema assumptions<br /> **The exnode mapping order corresponds to the file block order<br /> *L-Store additions<br /> **element &quot;function&quot; (a tmp space holder to bypass a java bug)<br /> **erasure metadata &quot;parity slice size&quot;<br /> **element &quot;parity&quot; is the mapping for the parity block if parity slice size != -1<br /> ***the parity mapping immediately follows the data mapping<br /> == Erasure codings ==<br /> *L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.<br /> **These blocks can be read directly by LoRS.<br /> *L-Store also writes extra erasure blocks to recover lost blocks.<br /> **the XOR-ing to generate erasure block is calculated on the depots<br /> ***nfu operation stored on the depot<br /> **erasure blocks are also stored in the exnode file<br /> **erasure blocks are only useful to the data-management area and not the data-client (see diagram)<br /> **erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema)<br /> *L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.<br /> **LoRS compatible (the base schema)<br /> **LoRS does not have access to L-Store's capability to recover a block<br /> *The L-Store data-client (Lstcp) has no block recovery capability.<br /> *data recovery, data warming, etc all data-management tasks, not data-client task.<br /> <br /> == Data client API &amp; interfaces ==<br /> *command line tools for file management<br /> **file movement<br /> **file replication, augment, trim<br /> *command line tools for namespace management<br /> **list files<br /> **create directories, etc<br /> *LoRS API<br /> **uses the base exnode schema (LoRS)<br /> **reads local exnode files<br /> **talks to LoDN with a combined http:/LoDN protocol<br /> **C/C++ Posix and posix-like API<br /> *Lstcp API (not developed yet)<br /> **does not parse exnodes?<br /> **talks to L-Server with the L-Store protocol<br /> **receives IBP commands individually from L-Server?<br /> ***sends these commands directly to IBP depots?<br /> *typical C/C++ API needs (proposed)<br /> **simple offsets and lengths<br /> **non-blocked (vector) reads</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Exnode_specification&diff=3212 2008-01-29T16:44:16Z

<p>Dano: </p> <hr /> <div>The exnode specification is implied by current usage in L-Store and LoRS.<br /> <br /> [[Example LoRS exnode xml file]]<br /> * 1 namespace: exnode<br /> * 6 elements: exnode, mapping, metadata, read, write, manage<br /> * element metadata uses 2 attibutes: name, type<br /> ** 5 attribute name values: alloc_length, alloc<br /> <br /> [[Example L-Store exnode xml file -- parity off]]<br /> <br /> [[Example L-Store exnode xml file -- with parity blocks]]<br /> *original implementation -- being modified<br /> <br /> == Schema ==<br /> <br /> The exnode xml is fairly unstructured and the functionality of a schema is minimal.<br /> <br /> [[Simplified xml file]]<br /> *(removed xmlns to view structure more clearly)<br /> <br /> [[Simple schema]]<br /> *In this example schema<br /> **The schema shows that only the element &quot;mapping&quot; has required elements.<br /> **The element metadata's use of attributes is unrestricted.<br /> **The only re-used type is the IBP_capability (read,write,manage)<br /> *extra-schema assumptions<br /> **The exnode mapping order corresponds to the file block order<br /> *L-Store additions<br /> **element &quot;function&quot; (a tmp space holder to bypass a java bug)<br /> **erasure metadata &quot;parity slice size&quot;<br /> **element &quot;parity&quot; is the mapping for the parity block if parity slice size != -1<br /> ***the parity mapping immediately follows the data mapping<br /> == Erasure codings ==<br /> *L-Store writes data in its original format (possibly encrypted, etc), just like LoRS.<br /> **These blocks can be read directly by LoRS.<br /> *L-Store also writes extra erasure blocks to recover lost blocks.<br /> **the XOR-ing to generate erasure block is calculated on the depots<br /> ***nfu operation stored on the depot<br /> **erasure blocks are also stored in the exnode file<br /> **erasure blocks are only useful to the data-management area and not the data-client (see diagram)<br /> **erasure blocks constitute an extension of the base (LoRS) exnode concept (or schema)<br /> *L-Store can write a LoRS compatible exnode file that excludes the erasure allocations.<br /> **LoRS compatible (the base schema)<br /> **LoRS does not have access to L-Store's capability to recover a block<br /> *The L-Store data-client (Lstcp) has no block recovery capability.<br /> *data recovery, data warming, etc all data-management tasks, not data-client task.<br /> <br /> == Data client API &amp; interfaces ==<br /> *command line tools for file management<br /> **file movement<br /> **file replication, augment, trim<br /> *command line tools for namespace management<br /> **list files<br /> **create directories, etc<br /> *LoRS API<br /> **uses the base exnode schema (LoRS)<br /> **reads local exnode files<br /> **talks to LoDN with a combined http:/LoDN protocol<br /> **C/C++ Posix and posix-like API<br /> *Lstcp API (not developed yet)<br /> **does not parse exnodes?<br /> **talks to L-Server with the L-Store protocol<br /> **receives IBP commands individually from L-Server?<br /> ***sends these commands directly to IBP depots?<br /> *typical C/C++ API needs (proposed)<br /> **simple offsets and lengths<br /> **non-blocked (vector) reads</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3211 2008-01-29T15:57:37Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> ***This is problematic since a file can be replicated with different allocation boundaries leading to different top hash for the file.<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that <br /> **the checksum method becomes part of the IBP protocol itself <br /> **and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> ** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) <br /> *This would be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3210 2008-01-29T15:46:48Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> ***This is problematic since a file can be replicated with different allocation boundaries leading to different top hash for the file.<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> ** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) <br /> *This would be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3209 2008-01-29T15:14:15Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> ***This is problematic since a file can be replicated with different allocation sizes and boundaries.<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> ** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) <br /> *This would be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3208 2008-01-29T14:59:57Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> ** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) <br /> *This would be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3198 2008-01-28T22:43:57Z

<p>Dano: /* NFU checksum approach */</p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> ** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) <br /> *This would be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3197 2008-01-28T22:42:20Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.<br /> *This would be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3196 2008-01-28T22:41:24Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> == NFU checksum approach ==<br /> *It has also been suggested that a java-based NFU approach could accommodate any generic hashing approach that the end user wants to implement.<br /> *This will be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3195 2008-01-28T22:40:51Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> *Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.<br /> **The checksum method is exposed at least to the IBP client<br /> **This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.<br /> <br /> - Dan 1/24/08<br /> <br /> *It has also been suggested that a java-based NFU approach could accommodate any generic hashing approach that the end user wants to implement.<br /> *This will be much slower than an algorithm integrated into the depot.<br /> *A depot internal checksum doesn't stop the use of NFU checksums.<br /> **excessive NFU usage could hamper depot performance<br /> <br /> - Dan 1/28/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3194 2008-01-28T16:54:46Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks for checksums (subdividing an IBP allocation) is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> <br /> - Dan 1/24/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3171 2008-01-25T10:24:00Z

<p>Dano: /* Data Integrity */</p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks per allocation for checksums is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?<br /> <br /> - Dan 1/24/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3170 2008-01-25T10:23:42Z

<p>Dano: /* Data Integrity */</p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks per allocation for checksums is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> *Do the hashes go into the exnode, or are they only stored in the depot allocation tables.<br /> <br /> - Dan 1/24/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3169 2008-01-25T10:22:24Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks per allocation for checksums is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. <br /> *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> <br /> - Dan 1/24/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3168 2008-01-25T10:13:34Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks per allocation for checksums is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it. If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.<br /> <br /> - Dan 1/24/08</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Development_ideas&diff=3167 2008-01-25T10:10:01Z

<p>Dano: /* Validation along the entire data path */</p> <hr /> <div>= Suggested changes to existing protocol =<br /> <br /> == Re-order parameters in IBP_STATUS command ==<br /> The existing IBP v1.4 implementation is:<br /> <br /> :''version IBP_STATUS '''RID''' IBP_ST_INQ password TIMEOUT \n''<br /> :''version IBP_STATUS '''RID''' IBP_ST_CHANGE password TIMEOUT \n max_hard max_soft max_duration \n''<br /> :''version IBP_STATUS IBP_ST_RES TIMEOUT \n''<br /> <br /> Notice that two of the commands have a primary command, ''IBP_STATUS'', a resource ID (''RID''), followed by a sub-command (''IBP_ST_INQ, IBP_ST_CHANGE'') and the last version has no ''RID'', just a sub-command, ''IBP_ST_RES''. The current implementation can only be parsed by first reading the whole line in and then counting the number of arguments. The argument count is then used to determine which command is actually being issued. A more natural version of the commands would always have the sub-command immediately follow the IBP_STATUS command.<br /> <br /> == Change in ''RID'' format ==<br /> The current definition of an ''RID'' is an integer as defined in ''struct ibp_depot''. The definition of an integer is architecture dependent and hense not portable. An alternative definition would be to define the ''RID'' as a character string. This would provide flexibility in its implmentation and use. The current IBP client libraries already treat the ''RID'' as an opaque character string for all commands except ''IBP_Allocate()''.<br /> <br /> == Provide interface to IBP data structures ==<br /> No explicit interfaces are provided for any of the various IBP data structures. A more flexible approach would be to add API calls to manipulate these structures indirectly.<br /> <br /> == IBP_MCOPY current status ==<br /> The documentation for this command is sparse. It looks like numerous different multicast methods were implemented but there is very little documentation describing them. Should this command be dropped?<br /> <br /> == NFU ==<br /> There is very little documentation describing the NFU implementation in the current LoCI depot and the documentation provided has errors and is not fully supported. The concept of the NFU is very powerful and I wonder if it should be split out as a separate specification altogether. Hunter's Java implementation is quite elegant. In his implementation the NFU calls are actually Java JAR files stored as allocations. These allocation are then registered with the NFU manager with hooks for checksums for data integrity. Having the NFU call operating in a Java container is extremely appealing. Java can place the NFU call in a box to limit it's resource consumption(memory, cpu, threads, etc) making it much more difficult for an NFU call to inadvertently or maliciously take down the depot or NFU manager. Also because of Java's portability deploying new NFU calls becomes trivial.<br /> <br /> = Security =<br /> <br /> == Add support for SSL ==<br /> Self-explanatory<br /> <br /> == Auth/AuthZ for IBP_ALLOC command ==<br /> This command has the potential for abuse and could result in a a &quot;Denial of Space&quot; attack on the depot. If the concept of an &quot;account&quot; is added one could then come up with additional methods to share resources for example adding the concept of an account quota. It also provides a tracking mechanism on who is *creating* allocations.<br /> <br /> == Virtual Capabilities(vcap) ==<br /> The current implementation only allows a single set of caps for an allocation. So once a user has access to a cap it can never be revoked. Virtual caps is designed to solve this problem. The idea is a user presenting the IBP_MANAGE cap could request the depot issue a new set of caps with a shorter duration. These new vcaps could then be provided to a 3rd party. At any time the original cap owner can revoke access to the allocation by simply using the IBP_MANAGE command to delete the vcap. Another useful feature to consider is restricting the vcap to a specific byte range of the original cap.<br /> <br /> == IBP &quot;Accounts&quot; ==<br /> In order for several of these ideas to work a new set of commands would need to be added to manage the accounts.<br /> <br /> <br /> = Data Integrity =<br /> <br /> == Validation along the entire data path ==<br /> The current implementation allows for validating at the end points only. This is accomplished by having the data originator calculate a checksum before uploading the data. This checksum can be appended to the data uploaded data or it can be stored externally in the exnode. The consumer can then download the data, calculate the checksum, and compare it to what is stored. This approach is not well suited to live data streams since the raw data will have to be buffered until the consumer can download the data to verify it.<br /> <br /> An alternative approach would be to standardize on a checksum algorithm and have the client calculate the checksum as the data is being streamed to the depot while the depot simultaneously calculates the checksum as it receives the data. The sender would pass on it's checksum for validation by the receiver. Any discrepancy occurring during the network transfer would be immediately detected while the data is still in the senders original buffer. The depot could then store this checksum as part of the allocation for use later. Most OS will immediately detect a write failure but not necessarily bit rot when reading unless the disk is part of a RAID array. Likewise when a reader requests data the reverse process can occur. Namely the depot and receiver both calculate the checksum as the data is being sent. The depot would additionally compare the original checksum stored with what was just calculated in order to detect disk errors. If no errors occurred the depot would go ahead and send the checksum down to the receiver for validation. This process is computationally efficient since the data is never re-read. The checksum is just part of the transfer pipeline.<br /> <br /> Building this validation procedure into the protocol simplifies the data integrity higher level tools require. These checksums could be used by higher level tools to verify replicated copies and detect data changes. The checksums should be treated as opaque strings and could be accessed by additional IBP_MANAGE sub-commands:<br /> <br /> * IBP_GET_CHECKSUM - Return the allocations checksum<br /> * IBP_VALIDATE_CHECKSUM - Re-calculate the checksum<br /> <br /> Using single checksum for an entire allocation is not efficient if random I/O on an allocation is allowed. In this case changing a single byte of a 10MB allocation would require the re-processing of the entire allocation. Another option would be to specify that for every 64KB of data (I picked this out of the blue so feel free to suggest something different) a checksum is generated. This means each allocation could have multiple checksums. In this case if a single byte was changed only 64KB of data would have to be re-processed. If the checksum field on the client is treated as an opaque string then having 1 or multiple checksums is irrelevant. Both cases can be treated the same.<br /> <br /> see discussion tab for comments<br /> <br /> = Miscellanous =<br /> <br /> == Support UDP transfers ==<br /> What about usingthe UDT implementation since it can mimic FAST, web100, etc. TCP congestion control methods...</div>

Dano https://www.reddnet.org/mwiki/index.php?title=Talk:Development_ideas&diff=3151 2008-01-24T20:53:34Z

<p>Dano: </p> <hr /> <div>==Data Integrity==<br /> <br /> *The idea of 64k chunks per allocation for checksums is what's termed a hash list<br /> **http://en.wikipedia.org/wiki/Hash_list<br /> *which can be extended to a multi-level hash tree<br /> **http://en.wikipedia.org/wiki/Hash_tree<br /> <br /> *One could imagine a 3-level tree<br /> **a hash for each lowest-common-unit, say the commonly used 1K<br /> **a single top hash for each IBP allocation<br /> **a single top hash for the file<br /> **or more levels<br /> <br /> *It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.<br /> **Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost<br /> **http://en.wikipedia.org/wiki/Tiger_%28hash%29<br /> **http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash<br /> **http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes<br /> <br /> If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would have to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it.<br /> <br /> - Dan 1/24/08</div>

Dano