November 21, 2006 L-Store Planning Meeting (VU): Difference between revisions
Jump to navigation
Jump to search
(16 intermediate revisions by 2 users not shown) | |||
Line 4: | Line 4: | ||
== Meeting Goal == | == Meeting Goal == | ||
Develop an initial strategy for development and releases of L-Store software. There was | Develop an initial strategy for development and releases of L-Store software. There was | ||
also some discussion of planned hardware deployment for the near term. | also some discussion of planned hardware deployment for the near term. | ||
== L-Store Releases and Features == | |||
=== L-Store Release 0.8 --- December 4 REDDnet@I2 meeting === | |||
* API documentation will be provided | |||
** L-Store | ** with the caveat that it is subject to change | ||
** may change due to input from REDDnet community | |||
* L-Store Functionality: | |||
** | ** <i>mkdir | ||
** | ** ls | ||
** | ** upload | ||
* <b>L-Store Release 0.9</ | ** download | ||
* <b>L-Store Release 1.0 | ** command line help</I> | ||
** storage fault tolerance via weaver erasure codes (upload and recovery) | |||
* Demonstration similar to SC06 (but on a smaller scale... 20 Gbs?) will be made at REDDnet@I2 meeting. | |||
* Clients will be made available (from REDDnet website) for installation by attendees at REDDnet@I2 | |||
** for user testing and application development | |||
** files written to depots should be considered scratch | |||
* Server and Depot software will not be made available at this point. | |||
* Bugs: a dozen or so minor (non-data-threatening) bugs were discovered at SC06. As many of these as can be fixed will be, the rest will be fixed by "0.9" release. | |||
* ACCRE will work with Med Center Staff (Leibler/Proteomics) to begin the process of migrating them over to using L-Store instead of ADIC | |||
* NevoaExplorer interface (Hunter) | |||
** This interface will work with existing depots | |||
** Software will be demo-ed at REDDnet@I2 | |||
** Full integration with L-Store will likely not be complete at this stage | |||
** Hunter is making a generic layer that will be agnostic to protocol (NFS, CIFS, L-Store) | |||
* <b>Omissions</B> | |||
** No Physical Delete | |||
** No Authentication and Authorization | |||
** No production depots publically available, but public depots for scratch files and testing will be available. | |||
=== L-Store Release 0.9 --- End of January 2007 === | |||
* Client available on REDDnet webpage | |||
* New Functionality | |||
** FUSE filesystem interface for Linux - allows you to mount L-Store storage as a native drive | |||
** Physical <i>delete</i> command | |||
** <i> stat</I> command | |||
** <I> ls-l </I> | |||
** Remaining SC06 bugs will be fixed, plus any new ones discovered | |||
* The meta-data server for the publicly available depots will be set up on a mirrored-raid device for fault tolerance | |||
** "production" public depots will be available | |||
* Full L-Store/NevoaExplorer interoperability will be implemented | |||
* Server/Depot software will be made available to REDDnet collaborators on request | |||
* Full scale testing of ADIC replacement for Proteomics will begin | |||
* <b>Omissions</B> | |||
** No Authentication and Authorization | |||
=== L-Store Release 1.0 --- May/June 2007 === | |||
* User Manual and Server Manual will be made available | |||
* Actual release date will be re-evaluated in January of 2007 | |||
* Fault tolerance | |||
** recover from lost metadata (we are not sure what the mechanism will be) | |||
** If a depot permanently dies, the missing file data will be automatically reconstituted for those files where possible. Unrecoverable files will be marked as such. | |||
* Switch Proteomics over to L-Store, replace ADIC system. | |||
* Real Server and Depot installer | |||
* Authentication and Security features enabled | |||
** need to spell out more specifically what | |||
* Load balancing on the meta-data servers will be implemented | |||
* Integration of StorCor into L-Store | |||
* complete L-Store monitoring will be ready taking advantage of new Java implementation of IBP | |||
== IBP Development??? == | |||
* Hunter will have two interns from Sun, he will have them work on the IBP software, converting it to Java. | |||
** Larry has requested input on what is done | |||
** will provide a lot more depot monitoring | |||
* libxio replacement | |||
== Near-Term Hardware Deployment == | |||
* Validation, Verification, and Performance Testing Depot | * Validation, Verification, and Performance Testing Depot | ||
** We will deploy 20 of the Capricorn Depots in a single system that can be used for validation, verification, and performance testing. | ** We will deploy about 20 of the Capricorn Depots in a single system that can be used for validation, verification, and performance testing. | ||
** This system will be for developers use only, and it can be taken down at any time. Software can be updated at any time. | ** This system will be for developers use only, and it can be taken down at any time. Software can be updated at any time. | ||
** Nightly, the L-Store development release code will be built, installed, and tested with Clyde. This may take a few months to get going smoothly. | ** Nightly, the L-Store development release code will be built, installed, and tested with Clyde. This may take a few months to get going smoothly. | ||
* User System | ** this will be available to REDDnet users at the REDDnet Chicago meeting and afterwards, until public depots are available. | ||
** The remainder of the systems | * Public Campus (LAN) System | ||
** approximately 10 of the Capricorn Depots will be deployed for use by VU campus researchers | |||
** A version of this will be made available with the release of L-Store 0.9 (end of January 2007). | |||
* Public User (WAN) System | |||
** The remainder of the systems will be deployed as a user facility, combined with the existing REDDnet depots. | |||
** A version of this will be made available with the release of L-Store 0.9 (end of January 2007). |
Latest revision as of 17:00, 3 December 2006
Attending
- Hagewood (Nevoa)
- Brown, Dawson, de Ledesma, Sheldon, Tackett (VU)
Meeting Goal
Develop an initial strategy for development and releases of L-Store software. There was also some discussion of planned hardware deployment for the near term.
L-Store Releases and Features
L-Store Release 0.8 --- December 4 REDDnet@I2 meeting
- API documentation will be provided
- with the caveat that it is subject to change
- may change due to input from REDDnet community
- L-Store Functionality:
- mkdir
- ls
- upload
- download
- command line help
- storage fault tolerance via weaver erasure codes (upload and recovery)
- Demonstration similar to SC06 (but on a smaller scale... 20 Gbs?) will be made at REDDnet@I2 meeting.
- Clients will be made available (from REDDnet website) for installation by attendees at REDDnet@I2
- for user testing and application development
- files written to depots should be considered scratch
- Server and Depot software will not be made available at this point.
- Bugs: a dozen or so minor (non-data-threatening) bugs were discovered at SC06. As many of these as can be fixed will be, the rest will be fixed by "0.9" release.
- ACCRE will work with Med Center Staff (Leibler/Proteomics) to begin the process of migrating them over to using L-Store instead of ADIC
- NevoaExplorer interface (Hunter)
- This interface will work with existing depots
- Software will be demo-ed at REDDnet@I2
- Full integration with L-Store will likely not be complete at this stage
- Hunter is making a generic layer that will be agnostic to protocol (NFS, CIFS, L-Store)
- Omissions
- No Physical Delete
- No Authentication and Authorization
- No production depots publically available, but public depots for scratch files and testing will be available.
L-Store Release 0.9 --- End of January 2007
- Client available on REDDnet webpage
- New Functionality
- FUSE filesystem interface for Linux - allows you to mount L-Store storage as a native drive
- Physical delete command
- stat command
- ls-l
- Remaining SC06 bugs will be fixed, plus any new ones discovered
- The meta-data server for the publicly available depots will be set up on a mirrored-raid device for fault tolerance
- "production" public depots will be available
- Full L-Store/NevoaExplorer interoperability will be implemented
- Server/Depot software will be made available to REDDnet collaborators on request
- Full scale testing of ADIC replacement for Proteomics will begin
- Omissions
- No Authentication and Authorization
L-Store Release 1.0 --- May/June 2007
- User Manual and Server Manual will be made available
- Actual release date will be re-evaluated in January of 2007
- Fault tolerance
- recover from lost metadata (we are not sure what the mechanism will be)
- If a depot permanently dies, the missing file data will be automatically reconstituted for those files where possible. Unrecoverable files will be marked as such.
- Switch Proteomics over to L-Store, replace ADIC system.
- Real Server and Depot installer
- Authentication and Security features enabled
- need to spell out more specifically what
- Load balancing on the meta-data servers will be implemented
- Integration of StorCor into L-Store
- complete L-Store monitoring will be ready taking advantage of new Java implementation of IBP
IBP Development???
- Hunter will have two interns from Sun, he will have them work on the IBP software, converting it to Java.
- Larry has requested input on what is done
- will provide a lot more depot monitoring
- libxio replacement
Near-Term Hardware Deployment
- Validation, Verification, and Performance Testing Depot
- We will deploy about 20 of the Capricorn Depots in a single system that can be used for validation, verification, and performance testing.
- This system will be for developers use only, and it can be taken down at any time. Software can be updated at any time.
- Nightly, the L-Store development release code will be built, installed, and tested with Clyde. This may take a few months to get going smoothly.
- this will be available to REDDnet users at the REDDnet Chicago meeting and afterwards, until public depots are available.
- Public Campus (LAN) System
- approximately 10 of the Capricorn Depots will be deployed for use by VU campus researchers
- A version of this will be made available with the release of L-Store 0.9 (end of January 2007).
- Public User (WAN) System
- The remainder of the systems will be deployed as a user facility, combined with the existing REDDnet depots.
- A version of this will be made available with the release of L-Store 0.9 (end of January 2007).