Talk:Development ideas - Revision history

Dano: /* NFU checksum approach */

2008-07-02T14:07:38Z

NFU checksum approach

← Older revision		Revision as of 09:07, 2 July 2008
Line 30:		Line 30:
	== NFU checksum approach ==		== NFU checksum approach ==
	*It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.		*It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.
	** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?)		** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?) or with mmaps for NFU servers integrated into IBP depot.
	*This ~~would~~ be ~~much~~ slower than an algorithm integrated into the depot.		*This may be slower than an algorithm integrated into the depot.
	*A depot internal checksum doesn't stop the use of NFU checksums.		*A depot internal checksum doesn't stop the use of NFU checksums.
	**excessive NFU usage could hamper depot performance		**excessive NFU usage could hamper depot performance

	- Dan 1/28/08		- Dan 1/28/08

Dano at 17:39, 29 January 2008

2008-01-29T17:39:06Z

← Older revision		Revision as of 12:39, 29 January 2008
Line 12:		Line 12:
	***This is problematic since a file can be replicated with different allocation boundaries leading to different top hash for the file.		***This is problematic since a file can be replicated with different allocation boundaries leading to different top hash for the file.

	*It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.		*It seems to be common in distributed file systems to use a binary hash tree rather than a hash list -- even on individual data blocks.
	**Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost		**Tiger Tree Hash TTH is currently the most? popular since it provides high security with a relatively low cpu cost
	**http://en.wikipedia.org/wiki/Tiger_%28hash%29		**http://en.wikipedia.org/wiki/Tiger_%28hash%29

Dano at 15:57, 29 January 2008

2008-01-29T15:57:37Z

← Older revision		Revision as of 10:57, 29 January 2008
Line 21:		Line 21:
	*If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.		*If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.
	*Do the hashes go into the exnode, or are they only stored in the depot allocation tables?		*Do the hashes go into the exnode, or are they only stored in the depot allocation tables?
	*Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.		*Since the IBP client needs to do the checksumming it needs to be emphasized that
			**the checksum method becomes part of the IBP protocol itself
			**and is not a hidden internal operation.
	**The checksum method is exposed at least to the IBP client		**The checksum method is exposed at least to the IBP client
	**This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.		**This may have significant implications to the scalability of the IBP protocol and these scalability implications may need to be explicitly addressed.

Dano at 15:46, 29 January 2008

2008-01-29T15:46:48Z

← Older revision		Revision as of 10:46, 29 January 2008
Line 10:		Line 10:
	**a single top hash for each IBP allocation		**a single top hash for each IBP allocation
	**a single top hash for the file		**a single top hash for the file
	***This is problematic since a file can be replicated with different allocation ~~sizes and~~ boundaries.		***This is problematic since a file can be replicated with different allocation boundaries leading to different top hash for the file.

	*It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.		*It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.

Dano at 15:14, 29 January 2008

2008-01-29T15:14:15Z

← Older revision		Revision as of 10:14, 29 January 2008
Line 10:		Line 10:
	**a single top hash for each IBP allocation		**a single top hash for each IBP allocation
	**a single top hash for the file		**a single top hash for the file
	**~~or more levels~~		***This is problematic since a file can be replicated with different allocation sizes and boundaries.

	*It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.		*It seems to be common in distributed systems to use a binary hash tree rather than a hash list -- even on individual data blocks.

Dano at 14:59, 29 January 2008

2008-01-29T14:59:57Z

← Older revision		Revision as of 09:59, 29 January 2008
Line 19:		Line 19:

	If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it.		If the IBP protocol were to be extended to support a single specific checksum method it would possibly be something like TTH. In this example, this would mean TTH usage becomes part of the IBP protocol specification and both the IBP client and IBP depot would be required to implement it. The next level tools and APIs like LoRS and libxio and lstcp would not be required to know anything about IBP internal checksums, but they could optionally use it.
	*If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes ~~and~~ issue.		*If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes an issue.
	*Do the hashes go into the exnode, or are they only stored in the depot allocation tables?		*Do the hashes go into the exnode, or are they only stored in the depot allocation tables?
	*Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.		*Since the IBP client needs to do the checksumming it needs to be emphasized that the checksum method becomes part of the IBP protocol itself and is not a hidden internal operation.

Dano: /* NFU checksum approach */

2008-01-28T22:43:57Z

NFU checksum approach

← Older revision		Revision as of 17:43, 28 January 2008
Line 28:		Line 28:
	== NFU checksum approach ==		== NFU checksum approach ==
	*It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.		*It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.
			** the NFU operation would read data from disk via localhost (can NFU talk to the depot internally?)
	*This would be much slower than an algorithm integrated into the depot.		*This would be much slower than an algorithm integrated into the depot.
	*A depot internal checksum doesn't stop the use of NFU checksums.		*A depot internal checksum doesn't stop the use of NFU checksums.

Dano at 22:42, 28 January 2008

2008-01-28T22:42:20Z

← Older revision		Revision as of 17:42, 28 January 2008
Line 27:		Line 27:
	- Dan 1/24/08		- Dan 1/24/08
	== NFU checksum approach ==		== NFU checksum approach ==
	*It has also been suggested that ~~a java-based NFU approach~~ could accommodate any generic hashing approach that the end user wants to implement.		*It has also been suggested that NFUs could accommodate any generic hashing approach that the end user wants to implement.
	*This ~~will~~ be much slower than an algorithm integrated into the depot.		*This would be much slower than an algorithm integrated into the depot.
	*A depot internal checksum doesn't stop the use of NFU checksums.		*A depot internal checksum doesn't stop the use of NFU checksums.
	**excessive NFU usage could hamper depot performance		**excessive NFU usage could hamper depot performance

	- Dan 1/28/08		- Dan 1/28/08

Dano at 22:41, 28 January 2008

2008-01-28T22:41:24Z

← Older revision		Revision as of 17:41, 28 January 2008
Line 26:		Line 26:

	- Dan 1/24/08		- Dan 1/24/08
			== NFU checksum approach ==
	*It has also been suggested that a java-based NFU approach could accommodate any generic hashing approach that the end user wants to implement.		*It has also been suggested that a java-based NFU approach could accommodate any generic hashing approach that the end user wants to implement.
	*This will be much slower than an algorithm integrated into the depot.		*This will be much slower than an algorithm integrated into the depot.

Dano at 22:40, 28 January 2008

2008-01-28T22:40:51Z

@@ Line 21: / Line 21: @@
 *If checksumming were optional, additional IBP commands to turn it on and off would be needed, and interoperability with non-checksumming depots becomes and issue.
 *Do the hashes go into the exnode, or are they only stored in the depot allocation tables?
 - Dan 1/24/08