November 21, 2006 L-Store Planning Meeting (VU)

From ReddNet
Jump to navigation Jump to search


  • Hagewood (Nevoa)
  • Brown, Dawson, de Ledesma, Sheldon, Tackett (VU)

Meeting Goal

Develop an initial strategy for development and releases of L-Store software. There was also some discussion of planned hardware deployment for the near term.

L-Store Releases and Features

L-Store Release 0.8 --- December 4 REDDnet@I2 meeting

  • API documentation will be provided
    • with the caveat that it is subject to change
    • may change due to input from REDDnet community
  • L-Store Functionality:
    • mkdir
    • ls
    • upload
    • download
    • command line help
    • storage fault tolerance via weaver erasure codes (upload and recovery)
  • Demonstration similar to SC06 (but on a smaller scale... 20 Gbs?) will be made at REDDnet@I2 meeting.
  • Clients will be made available (from REDDnet website) for installation by attendees at REDDnet@I2
    • for user testing and application development
    • files written to depots should be considered scratch
  • Server and Depot software will not be made available at this point.
  • Bugs: a dozen or so minor (non-data-threatening) bugs were discovered at SC06. As many of these as can be fixed will be, the rest will be fixed by "0.9" release.
  • ACCRE will work with Med Center Staff (Leibler/Proteomics) to begin the process of migrating them over to using L-Store instead of ADIC
  • NevoaExplorer interface (Hunter)
    • This interface will work with existing depots
    • Software will be demo-ed at REDDnet@I2
    • Full integration with L-Store will likely not be complete at this stage
    • Hunter is making a generic layer that will be agnostic to protocol (NFS, CIFS, L-Store)
  • Omissions
    • No Physical Delete
    • No Authentication and Authorization
    • No production depots publically available, but public depots for scratch files and testing will be available.

L-Store Release 0.9 --- End of January 2007

  • Client available on REDDnet webpage
  • New Functionality
    • FUSE filesystem interface for Linux - allows you to mount L-Store storage as a native drive
    • Physical delete command
    • stat command
    • ls-l
    • Remaining SC06 bugs will be fixed, plus any new ones discovered
  • The meta-data server for the publicly available depots will be set up on a mirrored-raid device for fault tolerance
    • "production" public depots will be available
  • Full L-Store/NevoaExplorer interoperability will be implemented
  • Server/Depot software will be made available to REDDnet collaborators on request
  • Full scale testing of ADIC replacement for Proteomics will begin
  • Omissions
    • No Authentication and Authorization

L-Store Release 1.0 --- May/June 2007

  • User Manual and Server Manual will be made available
  • Actual release date will be re-evaluated in January of 2007
  • Fault tolerance
    • recover from lost metadata (we are not sure what the mechanism will be)
    • If a depot permanently dies, the missing file data will be automatically reconstituted for those files where possible. Unrecoverable files will be marked as such.
  • Switch Proteomics over to L-Store, replace ADIC system.
  • Real Server and Depot installer
  • Authentication and Security features enabled
    • need to spell out more specifically what
  • Load balancing on the meta-data servers will be implemented
  • Integration of StorCor into L-Store
  • complete L-Store monitoring will be ready taking advantage of new Java implementation of IBP

IBP Development???

  • Hunter will have two interns from Sun, he will have them work on the IBP software, converting it to Java.
    • Larry has requested input on what is done
    • will provide a lot more depot monitoring
  • libxio replacement

Near-Term Hardware Deployment

  • Validation, Verification, and Performance Testing Depot
    • We will deploy about 20 of the Capricorn Depots in a single system that can be used for validation, verification, and performance testing.
    • This system will be for developers use only, and it can be taken down at any time. Software can be updated at any time.
    • Nightly, the L-Store development release code will be built, installed, and tested with Clyde. This may take a few months to get going smoothly.
    • this will be available to REDDnet users at the REDDnet Chicago meeting and afterwards, until public depots are available.
  • Public Campus (LAN) System
    • approximately 10 of the Capricorn Depots will be deployed for use by VU campus researchers
    • A version of this will be made available with the release of L-Store 0.9 (end of January 2007).
  • Public User (WAN) System
    • The remainder of the systems will be deployed as a user facility, combined with the existing REDDnet depots.
    • A version of this will be made available with the release of L-Store 0.9 (end of January 2007).