ReddNet - User contributions [en]

Tests/LoDN

2009-09-01T17:47:07Z

Sellersc: This page summaries the current state of LoDN and testing of LoDN

LoDN Test Page

LoCI tools and services

2008-04-21T02:20:59Z

Sellersc: Updated formatting

Christopher Sellers and Harold Gonzales

April 21, 2008

Outline

LoDN 
LibStdIO 
LoDNFS 
LEncoder 
PyIBP 
ScreenCasts

Architectural Stack

<code><pre>
LibSTDIO LoDNFS

LoDN

LoRS

Exnode LBone End2End

IBP
</pre></code>

LoDN Purpose
* Exnode Repository
* Exnode Management Services
* Exnode “warming”

LoDN Usefulness
* User Configurable
* Web Interface
* File system-like approach for management
* Exnode Persistence

User Configurable
* Exnode accessible (published/unpublished)
* Distribution of exnode across depots

Web Interface
* LoDN service is provided over http[s] through a web server
* All data and actions are passed to LoDN through http[s].
* This makes it accessible and viewable via a browser.
* Provides a simple interface protocol for writing client code.

File system-like appearance
* It provides heirachical directory structure
* Traverse, create and remove directories
* Exnodes appear as files within each directory
* Each exnode has properties: sizes, accessibility

LoDN URI
* lodn://hostname[:port]/username/path-to-exnode

Exnode Persistence
* Warming
* Imported exnodes fall under the control of the warmer
* Users can control how exnodes are warmed and distributed

Warmer
* Periodically refreshes each allocation
* Replicates data across the user defined "sites"
* One allocation is replicated and maintained per site
* Controls sharing and distribution of data

Warmer Control
* The warmer operates based on user defined configuration files
* Users can specify their own sites for their exnodes
* The warmer operates on exnodes based on the "nearest" configuration file.
* Hierarchical arranged, per exnode and per directory and per account
* Warmer files can be managed through web interface.

<DEMO of LoDN and Warmer file>

LibStdIO
* Based on the stdio interface
* Serves as drop in replacement
* Offers the following functions:
fopen, fdopen, fileno, ferror, setvbuf, fprintf, fread, putc, fputc, fputs, fwrite,
ftell, fseek, fclose, fflush, fgetc, fgets, feof, clearerr, fstat, ftruncate

* The only change needed is to replace the stdio.h header file with libStdIO.h and to recompile with LoRS libraries.
* It does preprocessor replacement, to replace all of the above functions with their std_ equivalents in libStdIO.

* It works with the LoDN URI and is compatible with regular stdio.

* When a file is opened, libStdIO determines the type based on the URI scheme. For every file descriptor, there is an internal data structure of function pointers for the io functions to use. For standard files, the regular io functions are assigned. For LoDN URI, the LoRS (lors_*) functions are assigned in the data structure. When the std_* functions are called, this data structure is used to call the correction io function.

Ex: Application --> std_fread() --> lors_fread() or fread()

* When a LoDN URI is opened, libStdIO uses the http[s] protocol to export the exnode from LoDN and to the local file system for use.
* When a LoDN URI is closed, libStdIO uses the http[s] protocol to import the exnode to LoDN. - The logistical networking based io functions are implemented with the LoRS API with local buffering.

* In addition to the LoDN URI, environmental variables are used to pass information to libStdIO including username (LODN_USER), password (LODN_PASSWORD), ssl method (LODN_SSL_METHOD), and depots (LORS_DEPOTS).

* Example Applications:
HDF5, grace, gnuplot and more

<LibStDIO Demo>

LoDNFS over Fuse

Purpose
* Many application expect lower Posix IO and filesystem interface if StdIO can be used.
* Every application has to be individual ported.

Fuse
* Filesystem in userspace or FUSE allows non-privileged users to create their own file systems without the need to write any kernel code.
* This is achieved by running the file system code in user space, while the FUSE module only provides a "bridge" to the actual kernel interfaces.
* In this case the FUSE interface handles the acquisition of the exnode and the collection of data from the depots, leaving the user free to use the mounted directory as they would a normal filesystem.
* The use of FUSE eliminates the need for applications to be ported, but also enforces filesystem semantics and necessitates the mounting of the desired lodn directories.
* A major advantage of fuse is that it allows us to not only use a command line interface but allows for the full range of applications of any mounted partition.

Implementation
* Fuse provides a set of functions to implement.
* The kernel performs filesystem requests through a pseudo device and fuse module that calls these functions.
* These functions work similar to the lors_* functions in libStdIO in that they use http[s] to communicate with LoDN and the LoRS API for operating on the exnode.

* Though still in development this tool allows for greater ease of use, a wider variety of applications, and a new level of convenience for the LoDN user community.

< LoDNFS Demo >

LEncoder: a logistical computing video transcoding service

* The NFU provides a mechanism for computation within the logisitcal network
* There exists a potential for computationally complex operations to be distributed and executed via many NFUs
* An example of such an operation is video transcoding ( the transformation of a video file by manipulating bitrate, codec, etc)
* LEncoder (based on Mplayer's MEncoder program) will introduce a facility for video transcoding within the logistical network
* encoding operations will read data from and write output to ibp capabilities
* A client program written using PyIBP will segment the data in the original file, distribute it across depots, invoke LEncoder on those segments, download the finished pieces, and coalesce the output file
* Current used planned for the transcoding of hdtv for the new ibpvoHD service

PyIBP

* Provides bindings to the C IBP client API in Python.
* In development
* Rapid prototyping for new tools and applications.
** New ideas can be rapidly tried and tested in a high level language.
** Adjustments and changes can be made quickly in Python before writing the production code in lower and more efficient languages like C.
* Python is heavily used by the scientific community for applications and gluing components together. This allows them to tie in another component (IBP) without rewriting their programs in C.
* For example, the client program for LEncoder is planned to be implemented with PyIBP.

Screen casts

* Provides a series of introductions to the various components of Logistical Networking.
* Targeted at different audiences (Users and Developers).
* The goal is to provide a template that explains to new users fundamentals of LN

* 3 main topics:
# LN intro (10 min) - Everyone
# LoDN (10 min) - Non tech users
# LibStdIO/Fuse - Application Developers

* In development due in Mid Summer

LoCI tools and services

2008-04-20T23:10:30Z

Sellersc: Initial Import of Page

LoCI tools and services

Christopher Sellers and Harold Gonzales

April 21, 2008

Outline

LoDN 
LibStdIO 
LoDNFS 
LEncoder 
PyIBP 
ScreenCasts

Architectural Stack
<nowiki>
LibSTDIO LoDNFS

LoDN

LoRS

Exnode LBone End2End

IBP
</nowiki>

LoDN Purpose
* Exnode Repository
* Exnode Management Services
* Exnode “warming”

LoDN Usefulness
* User Configurable
* Web Interface
* File system-like approach for management
* Exnode Persistence

User Configurable
* Exnode accessible (published/unpublished)
* Distribution of exnode across depots

Web Interface
* LoDN service is provided over http[s] through a web server
* All data and actions are passed to LoDN through http[s].
* This makes it accessible and viewable via a browser.
* Provides a simple interface protocol for writing client code.

File system-like appearance
* It provides heirachical directory structure
* Traverse, create and remove directories
* Exnodes appear as files within each directory
* Each exnode has properties: sizes, accessibility

LoDN URI
* lodn://hostname[:port]/username/path-to-exnode

Exnode Persistence
* Warming
* Imported exnodes fall under the control of the warmer
* Users can control how exnodes are warmed and distributed

Warmer
* Periodically refreshes each allocation
* Replicates data across the user defined "sites"
* One allocation is replicated and maintained per site
* Controls sharing and distribution of data

Warmer Control
* The warmer operates based on user defined configuration files
* Users can specify their own sites for their exnodes
* The warmer operates on exnodes based on the "nearest" configuration file.
* Hierarchical arranged, per exnode and per directory and per account
* Warmer files can be managed through web interface.

<DEMO of LoDN and Warmer file>

LibStdIO
* Based on the stdio interface
* Serves as drop in replacement
* Offers the following functions:
fopen, fdopen, fileno, ferror, setvbuf, fprintf, fread, putc, fputc, fputs, fwrite,
ftell, fseek, fclose, fflush, fgetc, fgets, feof, clearerr, fstat, ftruncate

* The only change needed is to replace the stdio.h header file with libStdIO.h and to recompile with LoRS libraries.
* It does preprocessor replacement, to replace all of the above functions with their std_ equivalents in libStdIO.

* It works with the LoDN URI and is compatible with regular stdio.

* When a file is opened, libStdIO determines the type based on the URI scheme. For every file descriptor, there is an internal data structure of function pointers for the io functions to use. For standard files, the regular io functions are assigned. For LoDN URI, the LoRS (lors_*) functions are assigned in the data structure. When the std_* functions are called, this data structure is used to call the correction io function.

Ex: Application --> std_fread() --> lors_fread() or fread()

* When a LoDN URI is opened, libStdIO uses the http[s] protocol to export the exnode from LoDN and to the local file system for use.
* When a LoDN URI is closed, libStdIO uses the http[s] protocol to import the exnode to LoDN. - The logistical networking based io functions are implemented with the LoRS API with local buffering.

* In addition to the LoDN URI, environmental variables are used to pass information to libStdIO including username (LODN_USER), password (LODN_PASSWORD), ssl method (LODN_SSL_METHOD), and depots (LORS_DEPOTS).

* Example Applications:
HDF5, grace, gnuplot and more

<LibStDIO Demo>

LoDNFS over Fuse

Purpose
* Many application expect lower Posix IO and filesystem interface if StdIO can be used.
* Every application has to be individual ported.

Fuse
* Filesystem in userspace or FUSE allows non-privileged users to create their own file systems without the need to write any kernel code.
* This is achieved by running the file system code in user space, while the FUSE module only provides a "bridge" to the actual kernel interfaces.
* In this case the FUSE interface handles the acquisition of the exnode and the collection of data from the depots, leaving the user free to use the mounted directory as they would a normal filesystem.
* The use of FUSE eliminates the need for applications to be ported, but also enforces filesystem semantics and necessitates the mounting of the desired lodn directories.
* A major advantage of fuse is that it allows us to not only use a command line interface but allows for the full range of applications of any mounted partition.

Implementation
* Fuse provides a set of functions to implement.
* The kernel performs filesystem requests through a pseudo device and fuse module that calls these functions.
* These functions work similar to the lors_* functions in libStdIO in that they use http[s] to communicate with LoDN and the LoRS API for operating on the exnode.

* Though still in development this tool allows for greater ease of use, a wider variety of applications, and a new level of convenience for the LoDN user community.

< LoDNFS Demo >

LEncoder: a logistical computing video transcoding service

* The NFU provides a mechanism for computation within the logisitcal network
* There exists a potential for computationally complex operations to be distributed and executed via many NFUs
* An example of such an operation is video transcoding ( the transformation of a video file by manipulating bitrate, codec, etc)
* LEncoder (based on Mplayer's MEncoder program) will introduce a facility for video transcoding within the logistical network
* encoding operations will read data from and write output to ibp capabilities
* A client program written using PyIBP will segment the data in the original file, distribute it across depots, invoke LEncoder on those segments, download the finished pieces, and coalesce the output file
* Current used planned for the transcoding of hdtv for the new ibpvoHD service

PyIBP

* Provides bindings to the C IBP client API in Python.
* In development
* Rapid prototyping for new tools and applications.
** New ideas can be rapidly tried and tested in a high level language.
** Adjustments and changes can be made quickly in Python before writing the production code in lower and more efficient languages like C.
* Python is heavily used by the scientific community for applications and gluing components together. This allows them to tie in another component (IBP) without rewriting their programs in C.
* For example, the client program for LEncoder is planned to be implemented with PyIBP.

< PyIBP Demo >

Screen casts

* Provides a series of introductions to the various components of Logistical Networking.
* Targeted at different audiences (Users and Developers).
* The goal is to provide a template that explains to new users fundamentals of LN

* 3 main topics:
# LN intro (10 min) - Everyone
# LoDN (10 min) - Non tech users
# LibStdIO/Fuse - Application Developers

* In development due in Mid Summer

REDDnet@I2: REDDnet Activities meeting, 21April08

2008-04-20T22:47:21Z

Sellersc: Created a link to LoCI tools and services page.

==Coordinates==
'''April 28, 8:00am-5:00pm, [http://www.marriott.com/hotels/travel/wasgw-crystal-gateway-marriott/ Crystal Gateway Marriott], Washington, DC.''' This meeting is held in cooperation with the [http://events.internet2.edu/2008/spring-mm/ Spring 2008 Internet2 Member Meeting], which will be held April 21-23, 2008. The meeting will be held in the [http://www.reddnet.org/mwiki/index.php/Image:Marriott2ndFloorDiagram.png Alexandria room].
== Overview and main goals of the the meeting ==
Over the course of the next 6 to 9 months, REDDnet will become a operational infrastructure, working in pre-production (i.e. "beta) mode, but none the less supporting application communities of early adopters and experimental users. The main purpose of this meeting is to map out, discuss, and agree on the steps necessary for success in this effort. Longer term plans for essential software development, service deployment, and community support will also be explored.

The morning session, which will be restricted to the REDDnet PIs and the core LN software development community and REDDnet operations group, will focus on issues surrounding the REDDnet roll out and mid-term software and service development and deployment. In the afternoon session, we will be joined by participants and interested parties from REDDnet's application communities; the goal of the session is to listen to and understand their requirements, concerns, and questions, so that REDDnet objectives and milestones line up as well as possible with their goals. The success of REDDnet largely depends on the success of these application communities in using it.

'''[[REDDnet@I2-Apr08 meeting results]]: Agreed upon objectives, milestones, task assignments, and other notes from the meeting.'''

=Agenda=
'''Continental Breakfast (7:30am-8:30am)'''
== Morning: Software Development and REDDnet Operations (8:30a - 12:00p)==
'''8:30-8:45''': Greetings (Paul and Terry)

'''8:45-9:30''': [[State of REDDnet infrastructure - April08]]
* Deployment: Current state, plans for next 6 months, issues raised. (Bobby)
* Summary of ongoing tests: ACCRE depot tests, LoDN/warming with ACCRE depot, VTNA, recursive augment for FACIT, CMS experiments (Alan,Santi, Chris, Harold)
* Request tracker for problem resolution (Matt)
*Issues?
** Protocols for infrastructure testing vs. application use
** Monitoring and management
** Instrumentation/monitoring? For example, PerfSonar?
** What about sites wanting to opt in?

'''9:30-10:00''' REDDnet Communications

'''10:00-10:15''' Morning Break

'''10:15-12:00''' [[Software and Service Development - April08]]
*Current LN middleware for REDDnet:
** L-Store (Santi)
** [[LoCI tools and services]] (Chris and Harold)
** Nevoa software suite
** 3rd party
**Phoebus?
* Future

Software Development - Since our resources for software development are limited at this point, the critical question here would seem to be this: "What new software development (including extensions or modifications of existing code) needs to occur in order for REDDnet to succeed over the next 6-9 months and be prepared for next phase.
* Interoperability testing:
**IBP depots - LoCI, ACCRE, Nevoa
**exNodes - L-Store and LoDN
**LoRS/LoDN and the ACCRE depot; L-Store and LoCI or Nevoa depot?
* Future modifications to/standardization of efforts
** IBP
** exNode
** exMSP
** TSSP

'''12:00-1:30 LUNCH'''

==Afternoon: Planning for Application Communities (1:30p - 5:00p)==
The afternoon session will focus on hearing about REDDnet's ongoing work with application communities. Representatives from some of those communities will be their, so we'll also hear from them. A more detailed agenda is in process, but below are some of the items to be presented and/or discussed. The main objective is to ascertain what lies on the critical path for operations, support, software development and standardization in order to enable our application groups to make progress. Application communities and projects to be discussed include the following:

* America View (P. R. Blackwell)- P.R. will give a brief overview of what they're doing to set the stage for an opportunity currently coming into view with the EDC and the USGS. The goal is to plan a demonstration of REDDnet capabilities for EDC early this summer.
* Federated Archive Cyberinfrastructure Testbed (FACIT) and broader collaboration with the Library or Congress and their NDIIPP program
* Vanderbilt Opthalmic Imaging Center -- Remote Third World Health Screening (Paul/Santi)
* Fusion Energy (Chris)
* CMS/OSG
* Collaboration with SuraGrid

=Attendees=
''Full Day''
*Vanderbilt: Paul Sheldon, Dan Engh, Alan Tackett, Santi de Ledesma, Bobby Brown, Matt Binkley, David Mathews,...
*University of Tennessee: Terry Moore, Chris Sellers, Harold Gonzales, Micah Beck (remotely)
*Stephen F. Austin: PR Blackwell
*University of Deleware: Martin Swany
*[http://www.muni.cz/ Masaryk University (Czech Republic)]: Lukáš Hejtmánek

''Afternoon: Application Communities''
*Library of Congress (Jane Mandelbaum, ...)
*SURA (Gary Crane, John Paul Robinson - UAB)
*UCSB (Remotely)
*Eastern Suffolk BOCES (Joe Isabelle)

How we did things in the tools we developed at UTK

2008-02-01T06:42:13Z

Sellersc: How LoCI has done things

How we did things in the tools we developed at UTK

Christopher Sellers

Feb 1, 2008

Outline

LoDN
LibStdIO
LoDNFS

Architectural Stack

LibSTDIO LoDNFS

LoDN

LoRS

Exnode LBone End2End

IBP

LoDN Purpose
- Exnode Repository
- Exnode Management Services
- Exnode “warming”

LoDN Usefulness
- User Configurable
- Web Interface
- File system-like approach for management
- Exnode Persistence

User Configurable
- Exnode accessible (published/unpublished)
- Distribution of exnode across depots

Web Interface
- LoDN service is provided over http[s] through a web server
- All data and actions are passed to LoDN through http[s].
- This makes it accessible and viewable via a browser.
- Provides a simple interface protocol for writing client code.

File system-like appearance
- It provides heirachical directory structure
- Traverse, create and remove directories
- Exnodes appear as files within each directory
- Each exnode has properties: sizes, accessibility

LoDN URI
- lodn://hostname[:port]/username/path-to-exnode

<DEMO of LoDN>

Exnode Persistence
- Warming
- Imported exnodes fall under the control of the warmer
- Users can control how exnodes are warmed and distributed

Warmer
- Periodically refreshes each allocation
- Replicates data across the user defined "sites"
- One allocation is replicated and maintained per site
- Controls sharing and distribution of data

Warmer Control
- The warmer operates based on user defined configuration files
- Users can specify their own sites for their exnodes
- The warmer operates on exnodes based on the "nearest" configuration file.
- Hierarchical arranged, per exnode and per directory
- Configuration files can be managed through web interface.

<DEMO of Warmer and control>

LibStdIO
- Based on the stdio interface
- Serves as drop in replacement
- Offers the following functions:
fopen, fdopen, fileno, ferror, setvbuf, fprintf, fread, putc, fputc, fputs, fwrite,
ftell, fseek, fclose, fflush, fgetc, fgets, feof, clearerr, fstat, ftruncate

- The only change needed is to replace the stdio.h header file with libStdIO.h and to recompile with LoRS libraries.

- It works with the LoDN URI and is compatible with regular stdio.
- It does preprocessor replacement, to replace all of the above functions with their std_ equivalents in libStdIO.
- When a file is opened, libStdIO determines the type based on the URI scheme. For every file descriptor, there is an internal data structure of function pointers for the io functions to use. For standard files, the regular io functions are assigned. For LoDN URI, the LoRS (lors_*) functions are assigned in the data structure. When the std_* functions are called, this data structure is used to call the correction io function.

Ex: Application --> std_fread() --> lors_fread() or fread()

- When a LoDN URI is opened, libStdIO uses the http[s] protocol to export the exnode from LoDN and to the local file system for use.
- When a LoDN URI is closed, libStdIO uses the http[s] protocol to import the exnode to LoDN. - The logistical networking based io functions are implemented with the LoRS API with local buffering.

- In addition to the LoDN URI, environmental variables are used to pass information to libStdIO including username (LODN_USER), password (LODN_PASSWORD), ssl method (LODN_SSL_METHOD), and depots (LORS_DEPOTS).

- Example Applications:
HDF5, grace, gnuplot and more.

#include <stdlib.h>
#include <unistd.h>
#include "libStdIO.h"

/***---Main---***/
int main(int argc, char *argv[])
{
/* Declarations */
FILE *file;
char buffer[4096];
int amtRead;
int i;

for(i=1; i<argc; i++)
{
if((file = fopen(argv[i], "r")) == NULL)
{
fprintf(stderr, "Error opening %s\n", argv[i]);
continue;
}

while((amtRead = fread(buffer, 1, sizeof(buffer), file)) > 0)
{
write(1, buffer, amtRead);
}

fclose(file);
}

/* Return successfully to the parent process */
return EXIT_SUCCESS;
}

<LibStDIO Demo>

Questions?

LoDNFS over Fuse

Purpose
- Many application expect lower Posix IO and filesystem interface if StdIO can be used.
- Every application has to be individual ported.

Fuse
- Filesystem in userspace or FUSE allows non-privileged users to create their own file systems without the need to write any kernel code.
- This is achieved by running the file system code in user space, while the FUSE module only provides a "bridge" to the actual kernel interfaces.
- In this case the FUSE interface handles the acquisition of the exnode and the collection of data from the depots, leaving the user free to use the mounted directory as they would a normal filesystem.
- The use of FUSE eliminates the need for applications to be ported, but also enforces filesystem semantics and necessitates the mounting of the desired lodn directories.
- A major advantage of fuse is that it allows us to not only use a command line interface but allows for the full range of applications of any mounted partition.

Implementation
- Fuse provides a set of functions to implement.
- The kernel performs filesystem requests through a pseudo device and fuse module that calls these functions.
- These functions work similar to the lors_* functions in libStdIO in that they use http[s] to communicate with LoDN and the LoRS API for operating on the exnode.

- Though still in development this tool allows for greater ease of use, a wider variety of applications, and a new level of convenience for the LoDN user community.

< LoDNFS Demo>

Questions?

How we did things in the tools we developed at UTK

2008-02-01T06:34:43Z

Sellersc: How LoCI has done things

All Hands Meeting, Feb 1, 2008

2008-02-01T05:39:32Z

Sellersc: /* 10:45-12:30 --- A Toolbox for Distributed Storage (Part 1) */

==Agenda for All-Hands==

Meeting Purpose: "To decide upon a plan of action that will make REDDnet a success."

The Meeting will be held at ACCRE at Vanderbilt University

===Morning Session 9-12:30===

====09:00-09:30 --- Meeting Kickoff====

* Summary of the proposal - towards a vision for REDDnet - Terry
* Putting together a plan of action - Paul

====09:30-10:00 --- Deployment and Operations====

* [[Operations and Deployment|Plan of Work]] - Bobby
* What should we expect of REDDnet sites? What should they expect of us?

====10:00-10:30 --- Applications====
* Application Liaison Effort - Plan of Work (Carie Lee)
* [[Demo Projects and Application Use]] - Plans (such as CMS) (Dan)

====10:30-10:45 --- Coffee Break====

====10:45-12:30 --- A Toolbox for Distributed Storage (Part 1)====
* Data and Directory Services Discussion
** [[Directory and data management services APIs|Architectural Issues]] - Larry
** [[How we did things in the tools we developed at UTK]] - Chris
** [[Check Point|TSSP Satus]] - Hunter
* IBP: [[Presentation information|Current Status and Proposed Extensions]] - Alan

===Lunch 12:30-01:30===
* Lunch will be provided at the Vanderbilt Commons

===Afternoon Session 1:30-05:00===

====01:30-03:15 --- A Toolbox for Distributed Storage (Part 2)====

* continue discussion listed above.

====03:15-03:30 --- Coffee Break====

====03:30-05:00 --- Close-Out Discussion and Assignments====
* tasks to be done
* prioritize tasks
* build a software diagram? (straw man is at [[Protocol Standardization Efforts]])