Aug 16, 2007 - Revision history

Terry at 13:07, 23 August 2007

2007-08-23T13:07:33Z

← Older revision		Revision as of 08:07, 23 August 2007
Line 46:		Line 46:
	*Could StoreCore easily be made that accessible through something other than a Java call?		*Could StoreCore easily be made that accessible through something other than a Java call?
	This discussion revealed the need for a real resource discovery protocol. We need a standard for resource discovery. ACCRE is going to contact Nevoa and discuss how a more portable protocol can be derived from the current StoreCore protocol. L-Bone is completely metadata based system; StoreCore is completely hand tagged, based on a logical volume model. The question is whether StoreCore model is general enough for many applications. The L-Bone is currently deprecated, or getting that way, but the model it uses – keeping metrics and making dynamic choices – may be good for some applications, e.g. for getting “Depots close to me” for improved data logistics.		This discussion revealed the need for a real resource discovery protocol. We need a standard for resource discovery. ACCRE is going to contact Nevoa and discuss how a more portable protocol can be derived from the current StoreCore protocol. L-Bone is completely metadata based system; StoreCore is completely hand tagged, based on a logical volume model. The question is whether StoreCore model is general enough for many applications. The L-Bone is currently deprecated, or getting that way, but the model it uses – keeping metrics and making dynamic choices – may be good for some applications, e.g. for getting “Depots close to me” for improved data logistics.
			=== Depots on the Teragrid===
			TeraGrid deployment? Judging from our discussions with John Cobb, what they seemed to want initially is fully authenticated identity, where everything is logged. Micah has made suggestions about how they could do something lighter weight. Discussion of Micah’s suggestion for how to do it with allocate, especially using third parties to help with authentication.

	===Upcoming Events===		===Upcoming Events===
	# Library of Congress meeting, Sept. 17		# Library of Congress meeting, Sept. 17
			# AmericaView Fall meeting, Sept. 19th and 20th.
	# Meeting in San Diego, Oct. 8th.		# Meeting in San Diego, Oct. 8th.
	# Supercomputing 07		# Supercomputing 07

	~~===Milenium Copyright Issues===~~

	~~===Deployment (Bobby)===~~
	~~#should be short~~

	~~===L-Store/Nevoa Software Development (Larry/Hunter)===~~
	~~# resource 0 problem progress?~~
	~~# any other issues~~

	~~===All other business===~~
	~~#Micah's proposal for addressing certain security issues that might inhibit work with ORNL/TeraGrid.~~

Terry: /* User Reports */

2007-08-23T13:03:21Z

User Reports

← Older revision		Revision as of 08:03, 23 August 2007
Line 39:		Line 39:

	Stage 2: We want to be able to show the use of untethered data and move away from the model that requires moving the data whenever you want to compute on it. In other words, we want much better data logistics for CMS. The main issue is getting the CMS applications to work with exNodes; Dan is working with the ROOT developers on this. Most jobs read 4 to 10% of the file. What’s killing performance is the latency, but they should be able to prefetch the data the application needs. The current synchronous interface is not satisfactory on this front. Huadong has developed an asynchronous interface and Alan is going to look at this work in preparation for trying to create an asynchronous interface that will do the job.		Stage 2: We want to be able to show the use of untethered data and move away from the model that requires moving the data whenever you want to compute on it. In other words, we want much better data logistics for CMS. The main issue is getting the CMS applications to work with exNodes; Dan is working with the ROOT developers on this. Most jobs read 4 to 10% of the file. What’s killing performance is the latency, but they should be able to prefetch the data the application needs. The current synchronous interface is not satisfactory on this front. Huadong has developed an asynchronous interface and Alan is going to look at this work in preparation for trying to create an asynchronous interface that will do the job.


	* '''FACIT (Terry/others)'''		* '''FACIT (Terry/others)'''
			Terry described interactions with UCSB, including the difficulty that the person at UCSB was encountering in configuring his depots re the L-Bone. This led to a discussion of providing a common way to do resource management and discovery. The thought was that the person at UCSB (Scott Smith) should be using store core. But this raised some questions:
			*How can he use StoreCore with the depot list?
			*Can the LoRS tools be modified to use StoreCore?
			*Could StoreCore easily be made that accessible through something other than a Java call?
			This discussion revealed the need for a real resource discovery protocol. We need a standard for resource discovery. ACCRE is going to contact Nevoa and discuss how a more portable protocol can be derived from the current StoreCore protocol. L-Bone is completely metadata based system; StoreCore is completely hand tagged, based on a logical volume model. The question is whether StoreCore model is general enough for many applications. The L-Bone is currently deprecated, or getting that way, but the model it uses – keeping metrics and making dynamic choices – may be good for some applications, e.g. for getting “Depots close to me” for improved data logistics.

	===Upcoming Events===		===Upcoming Events===

Terry: /* User Reports */

2007-08-23T12:54:46Z

User Reports

← Older revision		Revision as of 07:54, 23 August 2007
Line 34:		Line 34:
	We're still waiting on the Library to get their system set up on their end.		We're still waiting on the Library to get their system set up on their end.
	* '''CMS/RHIC (Dan)'''		* '''CMS/RHIC (Dan)'''
	~~Need programs~~ up and running to receive data. This ~~requires~~ SRM and GridFTP ~~compatability~~. Dan has ~~hooked up LoRS via a plugin in to GridFTP. The CMS jobs that come from Vampire will run on local depots.~~		In order to get going with CMS, Vandy needs certain middleware, including SRM and GridFTP, up and running to receive data. This means making our LN infrastructure compatible with SRM and GridFTP. Dan has made progress on a number of fronts.
	~~The goal is to get hooked up into CMS infrastructure w/o anyone having to care that we’re using L-Store~~ on ~~the back end. We’re building~~ a ~~back end for GridFTP.~~
	~~We and to be able to register and receive the data.~~
	~~Stage 2: They want to be able to show the use of unteathered data. Want to move away from the model that requires moving the data whenever you want to compute on it.~~
	~~Main issue is with the exNodes.~~
	~~Working with the ROOT developers. Most jobs read 4 to 10%~~ of ~~the file. What’s killing performance is the latency, but they should be able to prefetch the data you need. The current IBP protocol doesn’t allow you~~
	~~Huadong may have code for an IBP client that will be required~~.

			Dan has hooked up LoRS via a plugin in to GridFTP. The CMS jobs that come from Vampire will run on local depots. The goal is to get hooked up into CMS infrastructure w/o anyone having to care that we’re using L-Store on the back end. If we get this working well, it will mean that others in the community be able to start using L-Store seamlessly. We are now able to register and receive the data.

			Stage 2: We want to be able to show the use of untethered data and move away from the model that requires moving the data whenever you want to compute on it. In other words, we want much better data logistics for CMS. The main issue is getting the CMS applications to work with exNodes; Dan is working with the ROOT developers on this. Most jobs read 4 to 10% of the file. What’s killing performance is the latency, but they should be able to prefetch the data the application needs. The current synchronous interface is not satisfactory on this front. Huadong has developed an asynchronous interface and Alan is going to look at this work in preparation for trying to create an asynchronous interface that will do the job.

Terry: /* Agenda for today's call */

2007-08-23T12:44:11Z

Agenda for today's call

← Older revision		Revision as of 07:44, 23 August 2007
Line 29:		Line 29:
	There was a discussion of the need to accelerate the development of our "view" technology, so that, as a part of the demonstration, the audience could see what LN technology, such as L-Store, was doing under the covers. LoRS View has proved to be effective for this purpose in the past, but it hasn't been adapted to work with L-Store and needs some updating in any case. Larry has examined the possibility of porting or otherwise leveraging LoRS View for this purpose. PR and his group are going to give it a try. There was a discussion of using the Google maps API to do the display. Larry has distributed what he had on the project, which included Chris's description of how LoRS View does what it does.		There was a discussion of the need to accelerate the development of our "view" technology, so that, as a part of the demonstration, the audience could see what LN technology, such as L-Store, was doing under the covers. LoRS View has proved to be effective for this purpose in the past, but it hasn't been adapted to work with L-Store and needs some updating in any case. Larry has examined the possibility of porting or otherwise leveraging LoRS View for this purpose. PR and his group are going to give it a try. There was a discussion of using the Google maps API to do the display. Larry has distributed what he had on the project, which included Chris's description of how LoRS View does what it does.

			* '''VOIC/Peru (Santi)'''
			Not much to report here, except that our depot in Lima didn't go down during the recent earthquake.
			* '''Vanderbilt TV News Archive (Alan?)'''
			We're still waiting on the Library to get their system set up on their end.
			* '''CMS/RHIC (Dan)'''
			Need programs up and running to receive data. This requires SRM and GridFTP compatability. Dan has hooked up LoRS via a plugin in to GridFTP. The CMS jobs that come from Vampire will run on local depots.
			The goal is to get hooked up into CMS infrastructure w/o anyone having to care that we’re using L-Store on the back end. We’re building a back end for GridFTP.
			We and to be able to register and receive the data.
			Stage 2: They want to be able to show the use of unteathered data. Want to move away from the model that requires moving the data whenever you want to compute on it.
			Main issue is with the exNodes.
			Working with the ROOT developers. Most jobs read 4 to 10% of the file. What’s killing performance is the latency, but they should be able to prefetch the data you need. The current IBP protocol doesn’t allow you
			Huadong may have code for an IBP client that will be required.



	* '''VOIC/Peru (Santi)'''
	* '''Vanderbilt TV News Archive (Alan?)'''
	* '''CMS/RHIC (Dan)'''
	* '''FACIT (Terry/others)'''		* '''FACIT (Terry/others)'''

Terry: /* User Reports */

2007-08-23T12:41:35Z

User Reports

← Older revision		Revision as of 07:41, 23 August 2007
Line 27:		Line 27:
	In terms of software, they need to look at the new L-Store code and API. The Vandy team reported that there were few, if any, changes to the API, so the TexasView software should still run. It was noted that there was enough in place to start using available depots. All the Vandy depots are available for them to test with. Santi will post information about using this infrastructure, which includes StoreCore and management software on the list. Duration is set to infinity. The Texas nodes have been ordered. Capricorn has it and will ship them to Vandy, probably end of next week.		In terms of software, they need to look at the new L-Store code and API. The Vandy team reported that there were few, if any, changes to the API, so the TexasView software should still run. It was noted that there was enough in place to start using available depots. All the Vandy depots are available for them to test with. Santi will post information about using this infrastructure, which includes StoreCore and management software on the list. Duration is set to infinity. The Texas nodes have been ordered. Capricorn has it and will ship them to Vandy, probably end of next week.

	Google maps API to do the display:		There was a discussion of the need to accelerate the development of our "view" technology, so that, as a part of the demonstration, the audience could see what LN technology, such as L-Store, was doing under the covers. LoRS View has proved to be effective for this purpose in the past, but it hasn't been adapted to work with L-Store and needs some updating in any case. Larry has examined the possibility of porting or otherwise leveraging LoRS View for this purpose. PR and his group are going to give it a try. There was a discussion of using the Google maps API to do the display. Larry has distributed what he had on the project, which included Chris's description of how LoRS View does what it does.

Terry: /* User Reports */

2007-08-23T12:33:25Z

User Reports

← Older revision		Revision as of 07:33, 23 August 2007
Line 20:		Line 20:

	===User Reports===		===User Reports===
	* '''AmericaView (PR)'''		* '''AmericaView (PR)''' - TexasView is planning to demo both at the Fall I2 meeting, America View fall conference (Sept. 19th and 20th) Elements of the plan include the following:
	TexasView is planning to demo both at the Fall I2 meeting, America View fall conference (Sept. 19th and 20th) Elements of the plan include the following:		**Plan is to do what they did last year, but with more data. They need a few TB of space for this.
	**Plan is to do what they did last year, but with more data. They need a few TB of space for this.		**They want use data from other states, which makes it all the more critical that they get a more stable and reliable storage infrastructure underneath them so that they won’t be able to upload repeatedly. Repeated uploads just won't be feasible if other people's data is being used.
	** They want use data from other states, which makes it all the more critical that they get a more stable and reliable storage infrastructure underneath them so that they won’t be able to upload repeatedly. Repeated uploads just won't be feasible if other people's data is being used.		**They’ll show speed of download. What they need is a stable place to store the demo data, and they want their new depots in place for this reason.
	**They’ll show speed of download.
	What they need is a stable place to store the demo data, and they want their new depots in place for this reason.

	In terms of software, they need to look at the new L-Store code and API. The Vandy team reported that there were few, if any, changes to the API, so the TexasView software should still run. It was noted that there was enough in place to start using available depots. All the Vandy depots are available for them to test with. Santi will post information about using this infrastructure, which includes StoreCore and management software on the list. Duration is set to infinity. The Texas nodes have been ordered. Capricorn has it and will ship them to Vandy, probably end of next week.		In terms of software, they need to look at the new L-Store code and API. The Vandy team reported that there were few, if any, changes to the API, so the TexasView software should still run. It was noted that there was enough in place to start using available depots. All the Vandy depots are available for them to test with. Santi will post information about using this infrastructure, which includes StoreCore and management software on the list. Duration is set to infinity. The Texas nodes have been ordered. Capricorn has it and will ship them to Vandy, probably end of next week.

Terry: /* User Reports */

2007-08-23T04:01:25Z

User Reports

← Older revision		Revision as of 23:01, 22 August 2007
Line 26:		Line 26:
	**They’ll show speed of download.		**They’ll show speed of download.
	What they need is a stable place to store the demo data, and they want their new depots in place for this reason.		What they need is a stable place to store the demo data, and they want their new depots in place for this reason.
	In terms of software, they need to look at the new L-Store code and API. The Vandy team reported that there were few, if any, changes to the API, so the TexasView software should still run.
	It was noted that there was enough in place to start using available depots. All the Vandy depots are available for them to test with. Duration is set to infinity.		In terms of software, they need to look at the new L-Store code and API. The Vandy team reported that there were few, if any, changes to the API, so the TexasView software should still run. It was noted that there was enough in place to start using available depots. All the Vandy depots are available for them to test with. Santi will post information about using this infrastructure, which includes StoreCore and management software on the list. Duration is set to infinity. The Texas nodes have been ordered. Capricorn has it and will ship them to Vandy, probably end of next week.
	~~Their depots~~ have been ordered. Capricorn has it and will ship them to Vandy, probably end of next week.
	~~Santi will post information about using this infrastructure on the list.~~
	Google maps API to do the display:		Google maps API to do the display:

Terry: /* User Reports */

2007-08-23T03:58:49Z

User Reports

← Older revision		Revision as of 22:58, 22 August 2007
Line 20:		Line 20:

	===User Reports===		===User Reports===
	# AmericaView (PR)		* '''AmericaView (PR)'''
	# VOIC/Peru (Santi)		TexasView is planning to demo both at the Fall I2 meeting, America View fall conference (Sept. 19th and 20th) Elements of the plan include the following:
	# Vanderbilt TV News Archive (Alan?)		**Plan is to do what they did last year, but with more data. They need a few TB of space for this.
	# CMS/RHIC (Dan)		** They want use data from other states, which makes it all the more critical that they get a more stable and reliable storage infrastructure underneath them so that they won’t be able to upload repeatedly. Repeated uploads just won't be feasible if other people's data is being used.
	# FACIT (Terry/others)		**They’ll show speed of download.
			What they need is a stable place to store the demo data, and they want their new depots in place for this reason.
			In terms of software, they need to look at the new L-Store code and API. The Vandy team reported that there were few, if any, changes to the API, so the TexasView software should still run.
			It was noted that there was enough in place to start using available depots. All the Vandy depots are available for them to test with. Duration is set to infinity.
			Their depots have been ordered. Capricorn has it and will ship them to Vandy, probably end of next week.
			Santi will post information about using this infrastructure on the list.
			Google maps API to do the display:




			* '''VOIC/Peru (Santi)'''
			* '''Vanderbilt TV News Archive (Alan?)'''
			* '''CMS/RHIC (Dan)'''
			* '''FACIT (Terry/others)'''

	===Upcoming Events===		===Upcoming Events===

Terry: /* Annual Report Status (Paul) */

2007-08-23T03:49:26Z

Annual Report Status (Paul)

← Older revision		Revision as of 22:49, 22 August 2007
Line 17:		Line 17:

	===Annual Report Status (Paul)===		===Annual Report Status (Paul)===
			Paul is just getting this started and will be calling on the collaborators for contributions.

	===User Reports===		===User Reports===

Terry: /* LoDN Status (Micah) */

2007-08-23T03:47:27Z

LoDN Status (Micah)

← Older revision		Revision as of 22:47, 22 August 2007
Line 11:		Line 11:

	===LoDN Status (Micah)===		===LoDN Status (Micah)===
			LoDN is about ready to distribute; expect it to be deployed here at UT this weekend for a little more in service testing. Some of the main improvements include
			* More predictable warming – where to put copies
			* A lot of work has been done on security. ssh-http connections, the same certificate for signing the code and using the service.
			Should be packaged up and ready for distribution by Aug. 30th.

	===Annual Report Status (Paul)===		===Annual Report Status (Paul)===