Development ideas - Revision history

Hunterh: /* Suggested changes to existing protocol */

2008-02-01T19:38:31Z

Suggested changes to existing protocol

← Older revision		Revision as of 14:38, 1 February 2008
Line 21:		Line 21:
	== NFU ==		== NFU ==
	There is very little documentation describing the NFU implementation in the current LoCI depot and the documentation provided has errors and is not fully supported. The concept of the NFU is very powerful and I wonder if it should be split out as a separate specification altogether. Hunter's Java implementation is quite elegant. In his implementation the NFU calls are actually Java JAR files stored as allocations. These allocation are then registered with the NFU manager with hooks for checksums for data integrity. Having the NFU call operating in a Java container is extremely appealing. Java can place the NFU call in a box to limit it's resource consumption(memory, cpu, threads, etc) making it much more difficult for an NFU call to inadvertently or maliciously take down the depot or NFU manager. Also because of Java's portability deploying new NFU calls becomes trivial.		There is very little documentation describing the NFU implementation in the current LoCI depot and the documentation provided has errors and is not fully supported. The concept of the NFU is very powerful and I wonder if it should be split out as a separate specification altogether. Hunter's Java implementation is quite elegant. In his implementation the NFU calls are actually Java JAR files stored as allocations. These allocation are then registered with the NFU manager with hooks for checksums for data integrity. Having the NFU call operating in a Java container is extremely appealing. Java can place the NFU call in a box to limit it's resource consumption(memory, cpu, threads, etc) making it much more difficult for an NFU call to inadvertently or maliciously take down the depot or NFU manager. Also because of Java's portability deploying new NFU calls becomes trivial.

			== Resource Discovery ==
			Added call to receive the list of resource ID's

			--[[User:Hunterh\|Hunterh]] 13:38, 1 February 2008 (CST)

	= Security =		= Security =

Dano: /* Validation along the entire data path */

2008-01-25T10:10:01Z

Validation along the entire data path

← Older revision		Revision as of 05:10, 25 January 2008
Line 40:		Line 40:

	== Validation along the entire data path ==		== Validation along the entire data path ==
	~~Dan has added a bunch of text on this subject elsewhere so I'm not going to go into detail.~~

	The current implementation allows for validating at the end points only. This is accomplished by having the data originator calculate a checksum before uploading the data. This checksum can be appended to the data uploaded data or it can be stored externally in the exnode. The consumer can then download the data, calculate the checksum, and compare it to what is stored. This approach is not well suited to live data streams since the raw data will have to be buffered until the consumer can download the data to verify it.		The current implementation allows for validating at the end points only. This is accomplished by having the data originator calculate a checksum before uploading the data. This checksum can be appended to the data uploaded data or it can be stored externally in the exnode. The consumer can then download the data, calculate the checksum, and compare it to what is stored. This approach is not well suited to live data streams since the raw data will have to be buffered until the consumer can download the data to verify it.

Line 52:		Line 50:

	Using single checksum for an entire allocation is not efficient if random I/O on an allocation is allowed. In this case changing a single byte of a 10MB allocation would require the re-processing of the entire allocation. Another option would be to specify that for every 64KB of data (I picked this out of the blue so feel free to suggest something different) a checksum is generated. This means each allocation could have multiple checksums. In this case if a single byte was changed only 64KB of data would have to be re-processed. If the checksum field on the client is treated as an opaque string then having 1 or multiple checksums is irrelevant. Both cases can be treated the same.		Using single checksum for an entire allocation is not efficient if random I/O on an allocation is allowed. In this case changing a single byte of a 10MB allocation would require the re-processing of the entire allocation. Another option would be to specify that for every 64KB of data (I picked this out of the blue so feel free to suggest something different) a checksum is generated. This means each allocation could have multiple checksums. In this case if a single byte was changed only 64KB of data would have to be re-processed. If the checksum field on the client is treated as an opaque string then having 1 or multiple checksums is irrelevant. Both cases can be treated the same.

			see discussion tab for comments

	= Miscellanous =		= Miscellanous =

Tacketar: /* Validation along the entire data path */

2008-01-23T16:11:00Z

Validation along the entire data path

← Older revision		Revision as of 11:11, 23 January 2008
Line 44:		Line 44:
	The current implementation allows for validating at the end points only. This is accomplished by having the data originator calculate a checksum before uploading the data. This checksum can be appended to the data uploaded data or it can be stored externally in the exnode. The consumer can then download the data, calculate the checksum, and compare it to what is stored. This approach is not well suited to live data streams since the raw data will have to be buffered until the consumer can download the data to verify it.		The current implementation allows for validating at the end points only. This is accomplished by having the data originator calculate a checksum before uploading the data. This checksum can be appended to the data uploaded data or it can be stored externally in the exnode. The consumer can then download the data, calculate the checksum, and compare it to what is stored. This approach is not well suited to live data streams since the raw data will have to be buffered until the consumer can download the data to verify it.

	An alternative approach would be to standardize on a checksum algorithm and have the client calculate the checksum as the data is being streamed to the depot while the depot simultaneously calculates the checksum as it receives the data. Any discrepancy occurring during the network transfer would be immediately detected while the data is still in the senders original buffer. The depot could then store this checksum as part of the allocation for use later. Most OS will immediately detect a write failure but not necessarily bit rot when reading unless the disk is part of a RAID array. Likewise when a reader requests data the reverse process can occur. Namely the depot and receiver both calculate the checksum as the data is being sent. The depot would additionally compare the original checksum stored with what was just calculated in order to detect disk errors. If no errors occurred the depot would go ahead and send the checksum down to the receiver for validation. This process is computationally efficient since the data is never re-read. The checksum is just part of the transfer pipeline.		An alternative approach would be to standardize on a checksum algorithm and have the client calculate the checksum as the data is being streamed to the depot while the depot simultaneously calculates the checksum as it receives the data. The sender would pass on it's checksum for validation by the receiver. Any discrepancy occurring during the network transfer would be immediately detected while the data is still in the senders original buffer. The depot could then store this checksum as part of the allocation for use later. Most OS will immediately detect a write failure but not necessarily bit rot when reading unless the disk is part of a RAID array. Likewise when a reader requests data the reverse process can occur. Namely the depot and receiver both calculate the checksum as the data is being sent. The depot would additionally compare the original checksum stored with what was just calculated in order to detect disk errors. If no errors occurred the depot would go ahead and send the checksum down to the receiver for validation. This process is computationally efficient since the data is never re-read. The checksum is just part of the transfer pipeline.

	Building this validation procedure into the protocol simplifies the data integrity higher level tools require. These checksums could be used by higher level tools to verify replicated copies and detect data changes. The checksums should be treated as opaque strings and could be accessed by additional IBP_MANAGE sub-commands:		Building this validation procedure into the protocol simplifies the data integrity higher level tools require. These checksums could be used by higher level tools to verify replicated copies and detect data changes. The checksums should be treated as opaque strings and could be accessed by additional IBP_MANAGE sub-commands:
Line 52:		Line 52:

	Using single checksum for an entire allocation is not efficient if random I/O on an allocation is allowed. In this case changing a single byte of a 10MB allocation would require the re-processing of the entire allocation. Another option would be to specify that for every 64KB of data (I picked this out of the blue so feel free to suggest something different) a checksum is generated. This means each allocation could have multiple checksums. In this case if a single byte was changed only 64KB of data would have to be re-processed. If the checksum field on the client is treated as an opaque string then having 1 or multiple checksums is irrelevant. Both cases can be treated the same.		Using single checksum for an entire allocation is not efficient if random I/O on an allocation is allowed. In this case changing a single byte of a 10MB allocation would require the re-processing of the entire allocation. Another option would be to specify that for every 64KB of data (I picked this out of the blue so feel free to suggest something different) a checksum is generated. This means each allocation could have multiple checksums. In this case if a single byte was changed only 64KB of data would have to be re-processed. If the checksum field on the client is treated as an opaque string then having 1 or multiple checksums is irrelevant. Both cases can be treated the same.


	= Miscellanous =		= Miscellanous =

Tacketar at 01:38, 23 January 2008

2008-01-23T01:38:09Z

← Older revision		Revision as of 20:38, 22 January 2008
Line 16:		Line 16:
	No explicit interfaces are provided for any of the various IBP data structures. A more flexible approach would be to add API calls to manipulate these structures indirectly.		No explicit interfaces are provided for any of the various IBP data structures. A more flexible approach would be to add API calls to manipulate these structures indirectly.

	== IBP_MCOPY current status ===		== IBP_MCOPY current status ==
	The documentation for this command is sparse. It looks like numerous different multicast methods were implemented but there is very little documentation describing them. Should this command be dropped?		The documentation for this command is sparse. It looks like numerous different multicast methods were implemented but there is very little documentation describing them. Should this command be dropped?

Tacketar at 01:37, 23 January 2008

2008-01-23T01:37:53Z

← Older revision		Revision as of 20:37, 22 January 2008
Line 11:		Line 11:

	== Change in ''RID'' format ==		== Change in ''RID'' format ==

	The current definition of an ''RID'' is an integer as defined in ''struct ibp_depot''. The definition of an integer is architecture dependent and hense not portable. An alternative definition would be to define the ''RID'' as a character string. This would provide flexibility in its implmentation and use. The current IBP client libraries already treat the ''RID'' as an opaque character string for all commands except ''IBP_Allocate()''.		The current definition of an ''RID'' is an integer as defined in ''struct ibp_depot''. The definition of an integer is architecture dependent and hense not portable. An alternative definition would be to define the ''RID'' as a character string. This would provide flexibility in its implmentation and use. The current IBP client libraries already treat the ''RID'' as an opaque character string for all commands except ''IBP_Allocate()''.

	== Provide interface to IBP data structures ==		== Provide interface to IBP data structures ==
			No explicit interfaces are provided for any of the various IBP data structures. A more flexible approach would be to add API calls to manipulate these structures indirectly.

			== IBP_MCOPY current status ===
			The documentation for this command is sparse. It looks like numerous different multicast methods were implemented but there is very little documentation describing them. Should this command be dropped?

	~~No explicit interfaces are~~ provided ~~for any~~ of the ~~various IBP~~ data ~~structures~~. A more ~~flexible approach would be~~ to ~~add API~~ calls ~~to manipulate these structures indirectly~~.		== NFU ==
			There is very little documentation describing the NFU implementation in the current LoCI depot and the documentation provided has errors and is not fully supported. The concept of the NFU is very powerful and I wonder if it should be split out as a separate specification altogether. Hunter's Java implementation is quite elegant. In his implementation the NFU calls are actually Java JAR files stored as allocations. These allocation are then registered with the NFU manager with hooks for checksums for data integrity. Having the NFU call operating in a Java container is extremely appealing. Java can place the NFU call in a box to limit it's resource consumption(memory, cpu, threads, etc) making it much more difficult for an NFU call to inadvertently or maliciously take down the depot or NFU manager. Also because of Java's portability deploying new NFU calls becomes trivial.

	= Security =		= Security =

Tacketar at 01:23, 23 January 2008

2008-01-23T01:23:00Z

← Older revision		Revision as of 20:23, 22 January 2008
Line 50:		Line 50:


			= Miscellanous =

			== Support UDP transfers ==
			What about usingthe UDT implementation since it can mimic FAST, web100, etc. TCP congestion control methods...
	= ~~Miscellanous~~ =

Tacketar at 01:20, 23 January 2008

2008-01-23T01:20:24Z

@@ Line 21: / Line 21: @@
 == Add support for SSL ==
 == Auth/AuthZ for IBP_ALLOC command ==
@@ Line 34: / Line 35: @@
 = Data Integrity =
-== Data ==
+== Validation along the entire data path ==
 = Miscellanous =

Tacketar at 00:49, 23 January 2008

2008-01-23T00:49:17Z

@@ Line 20: / Line 20: @@
 = Security =
 = Data Integrity =
 = Miscellanous =

Tacketar at 23:18, 22 January 2008

2008-01-22T23:18:20Z

← Older revision		Revision as of 18:18, 22 January 2008
Line 17:		Line 17:

	No explicit interfaces are provided for any of the various IBP data structures. A more flexible approach would be to add API calls to manipulate these structures indirectly.		No explicit interfaces are provided for any of the various IBP data structures. A more flexible approach would be to add API calls to manipulate these structures indirectly.

			= Security =

			= Data Integrity =

			= Miscellanous =

Tacketar at 23:17, 22 January 2008

2008-01-22T23:17:01Z

← Older revision		Revision as of 18:17, 22 January 2008
Line 15:		Line 15:

	== Provide interface to IBP data structures ==		== Provide interface to IBP data structures ==

			No explicit interfaces are provided for any of the various IBP data structures. A more flexible approach would be to add API calls to manipulate these structures indirectly.