Proposal due date is Tuesday, February 18, 2014
The CORAL Request for Proposal (RFP) has been approved by DOE for release and publication on this Web site on January 6, 2014. Interested offerors are advised to discard all preceding draft RFP documents that were published on this Web site prior to January 6, 2014. DOE directed changes to the draft RFP, and those changes are reflected in RFP documents now available on this Web site. Interested offerors are advised to base their proposal responses on RFP documents now available on this Web site and any subsequent RFP amendments.
Interested offerors must submit all communication (questions, comments, etc.) about the CORAL RFP to the LLNS Contract Administrator, Gary Ward, whose contact information is provided below. The CORAL RFP market survey phase is complete; therefore, interested offerors are no longer permitted direct communication with the CORAL technical community except for regular business activities that do not pertain to the CORAL RFP or as otherwise directed by the LLNS Contract Administrator.
Interested offerors are advised to monitor this Web site for potential CORAL RFP amendments and other CORAL RFP information updates. LLNS may notify interested offerors (who have previously contacted LLNS and expressed an interest in the CORAL RFP) of updated CORAL RFP information via e-mail; however, LLNS is under no obligation to do so. It is the responsibility of all interested offerors to monitor this Web site for current CORAL RFP information.
CORAL Benchmark questions—and only benchmark-related questions—may be submitted via e-mail to email@example.com.
|LLNS Contract Administrator
Amendment 1, CORAL RFP B604142 (January 21, 2014)
Q1: In UMT, the README file distributed in the source distribution defines the FOM calculation inconsistent with the definition used in both the source code (SuOlsonTest.cc:221) and in the UMT Summary File (top of page 3). Which is correct?
A1: The version in the source code and summary file is correct (FOM = number_Unknowns/cumulativeWorkTime*cumulativeIterationCount). The parentheses in the README file version that change the order of operator precedence were incorrectly introduced and will be updated.
Q2: With respect to FOCI, how does this apply to the anticipated LLNL awards?
A2: It is anticipated the LLNL NRE awards will require Q-cleared subcontractor personnel to, in part, support porting key DOE applications to the CORAL system and improving the performance of DOE applications on the CORAL system. The need is for Subcontractor personnel to have access to / work on LLNL codes, which are classified. It is anticipated the LLNL build award will require Q-cleared subcontractor personnel for, in part, on-site analyst(s) support. The analyst(s) work will require on-going access to the system once it moves to a classified area. Having Q-cleared subcontractor personnel also improves the installation/initial shakeout/acceptance testing process.
Q3: What funding requests have been made to date concerning the CORAL program? What is the status of those requests?
A3: The Department of Energy is aware of budgetary amounts identified in the RFP letter. Funding for CORAL will depend, in part, on to-be-determined annual appropriated funds from Congress.
Q4: When is a decision regarding the class advance waiver anticipated? Will it occur prior to the due date for proposals?
A4: It is anticipated DOE will make a decision prior to the proposal due date.
Q5: Re PEPPI, 1.1: How will the identified evaluation factors of performance features, diversity, supplier attributes, and price be weighted in evaluating proposals? Within each of those four evaluation factors, how will the subfactors identified in the PEPPI, 1.3-1.6 be weighted?
A5: RFP evaluation criteria are not weighted.
Q6: The RFP requests copies of SEC Form 10-K filings. Is a link to the electronically filed 10-K sufficient or does the lab require the 10-K filing to be downloaded to the proposal CD-ROM (this will be very voluminous)? The RFP also requests tax returns for at least the past two years. Will the lab accept summary-level information, consisting of the first several pages of the filing, as opposed to the complete filings with all attachments and schedules?
A6: Providing the link within a proposal is sufficient. Summary-level information is sufficient. Additional supporting information may be subsequently requested, if deemed necessary.
Q7: The RFP does not specify where in the proposal volumes offerors should identify background IP and/or any IP exceptions or exclusions. In fact, some of the general terms and conditions seem to indicate that such exceptions and exclusions should not be submitted with proposals. See LLNL General Provisions, "Documents of Seller" and ORNL General Provisions, 1.5, "Acceptance of Terms and Conditions." Please clarify the timing and content of submissions of any exceptions to afford sufficient time to address the protections required for background patents, technical data, and computer software.
A7: Offerors may address IP positions in Proposal Volume 5, "Other Documents." Offerors may also include a brief reference to its IP position within Proposal Volume 2, "Business Proposal," directing the reader to its complete IP position within Proposal Volume 5. Offerors should carefully consider the nature and scope of any proposed exceptions to RFP terms and conditions, as such matters could potentially significantly delay or prevent an award(s). Any exceptions considered essential to an Offeror should be identified in its proposal response.
Q8: ORNL - Are Offerors required to propose a Lease-to-Own Option in their CORAL proposal or would ORNL be amenable to pursuing a third party leasing-only approach like ANL? Would there be any impact on proposal evaluation if an Offeror elected not to offer a Lease to Own option? If ORNL uses third party leasing, how will the third party leasing arrangement work? What documents will govern the relationships between ORNL, the awardee, and the third party lessor?
A8: Offerors must be able to offer financing in order to be considered for award. Offerors must propose per the CORAL Draft ORNL Lease Agreement. UT-Battelle is willing to consider a second alternative that is similar to the ANL approach. A proposal that does not include a lease-to-own option would not be viewed favorably. In order to implement third party leasing, UT-Battelle would exercise the Third Party Leasing Option in the CORAL Draft ORNL Lease Agreement with the awardee. Once the CORAL system is accepted at ORNL, UT-Battelle will instruct the third party lessor to make the payment to the awardee. The value of the payment will be based on system price in the CORAL Draft ORNL Lease Agreement. The third party lessor will then pay the awardee the designated amount. Once the awardee has been paid, title to the system would pass to the third party lessor. The awardee will continue to provide maintenance and support as described in the CORAL Draft ORNL Lease Agreement. UT-Battelle will make payments to the third party lessor until the lease is paid in full. The "CORAL Draft ORNL Lease Agreement" is the governing document between UT-Battelle and the awardee. UT-Battelle will use documents similar to the Master Lease to Ownership Agreement and Sample Lease to Ownership Order that ANL has posted in the CORAL RFP. The Master Lease to Ownership Agreement and Sample Lease to Ownership Order will be the contractual documents with the third party lessor.
Q9: ANL - How will ANL's intended third party leasing arrangement work? What documents will govern the relationships between ANL, the awardee, and the third party lessor? For example, are the Master Lease-To-Ownership Agreement and Sample Lease-to-Ownership Order documents that will be signed between ANL and the third party lessor, or between ANL and the Offeror?
(a) If the former, what document(s) will reflect the relationship between ANL and the Offeror?
(b) If the latter, what document(s) will effect the transfer of the system from the Offeror to the third party?
Whatever documentation will govern the relationship between ANL and the Offeror, there are several provisions in the LLNL Specific Build sample subcontract and in the Draft ORNL Lease Agreement that have yet to be adopted by ANL. Specifically, the provisions found in ARTICLE 16 – SIERRA GO/NO-GO DECISION of the LLNL Specific Build sample subcontract and Article III, Agreement Conditions, § L, Go/No Go Decision Clause in the Draft ORNL Lease Agreement are not included in any of the provided ANL documentation. Will ANL agree to revise its agreements to include these provisions?
A9: Offerors must be able to offer a lease to ownership plan, in accordance with the terms and conditions of the "Master Lease to Ownership Agreement" and "Lease to Ownership Order" in order to be considered for award, and Argonne reserves the right to later solicit and obtain third party financing dependent on market conditions, etc., prior to delivery of the equipment. ANL/Awardee documents to include: Terms and Conditions, Moderate Risk Work Supplemental Conditions, Purchase Order, Statement of Work, Lease to Ownership Order, and Master Lease to Ownership Agreement. ANL/Third Party Lessor documents to include: Lease to Ownership Order, and Master Lease to Ownership Agreement. In order to effect a transfer of the system from the Offeror to the third party a modification will be done to the Purchase Order and Lease to Ownership Order, as necessary. The Go/No Go Decisions will be negotiated and added to the SOW after a vendor has been selected. Any other provisions may be added after vendor selection if deemed necessary.
Q10: Will certified cost or pricing data be required for the NRE awards?
A10: Certified cost or pricing data will be obtained only if LLNS determines that none of the exceptions or other reasons stated below apply:
• The prices agreed upon are based on adequate price competition;
• A commercial item is being acquired or a subcontract is modified to acquire commercial items; or
• The prices agreed upon are based on prices set by law or regulation
Note that the CORAL RFP (Livermore NRE-Specific Components) includes documents pertaining to CAS, FAR Table 15-2, and Certificate of Current Cost or Pricing Data.
Q11: In response to Offeror Question 9 about leasing arrangements with ANL, LLNL responded in part: "Offerors must be able to offer a lease to ownership plan, in accordance with the terms and conditions of the "Master Lease to Ownership Agreement" and "Lease to Ownership Order" in order to be considered for award, and Argonne reserves the right to later solicit and obtain third party financing dependent on market conditions, etc., prior to delivery of the equipment." The cover letter to the CORAL RFP, however, states that "ANL intends to acquire its machines using third party Lease-To-Own financing." Has ANL changed its approach to leasing such that it now intends to pursue offeror-financed leasing as an initial option while reserving the right to use third party financing if ANL so chooses? Must Offerors propose offeror-financed lease-to-own terms for ANL as for ORNL?
A11: ORNL Response: Offerors must be able to offer financing in order to be considered for award. Offerors must propose per the CORAL Draft ORNL Lease Agreement.
A11: ANL Response: Offerors must be able to offer a lease to ownership plan in accordance with the terms and conditions of the "Master Lease to Ownership Agreement" and "Lease to Ownership Order" in order to be considered for award and Argonne reserves the right to later solicit and obtain third party financing dependent on market conditions, etc., prior to delivery of the equipment.
Q12: Where in the proposal submission are Offerors to include their pricing terms for lease financing for ORNL and ANL?
A12: The Price Schedule spreadsheet will be modified (to include tabs for ORNL and Argonne lease pricing terms) and reissued.
Q13: Referring to RFP Section 5.1.8, does the reference to NVRAM apply to EEPROM used to store manufacturing data? If so, is it sufficient for a vendor to assert that unprivileged users cannot access this NVRAM?
A13: No, it is not sufficient to state that unprivileged users do not have access to EEPROMs. We would like a way to clear all parts of the chip that can be written to.
Q14: The 5% allowable variation in spectral element count for the nekbone benchmark does not provide enough flexibility in the case of varying system parameters. Can the range be extended?
A14: The range of allowable deviation in the Nekbone benchmark spectral element count has been increased from 5 to 15%. The Nekbone benchmark summary has been updated to reflect this.
Q15: Can the FOM for CAM-SE be modified to reflect an increase in NE?
A15: No. The FOM for CAM-SE is simulated years per day (SYPD). Increasing the number of elements to some arbitrary amount has no real scientific justification. The metric for this throughput app was chosen to reflect actual usage as part of an ensemble calculation. The ability to increase NE was provided to allow some room for tuning, but the key measure of performance remains throughput.
Q16: In the benchmark results file, should the number of strings for the large configuration be 2,400,000,000 (not 2,400,000)? Also, there are not cells to report all three test problem sizes, how should we report those?
A16: The HASH tab in the results spreadsheet has been updated to reflect the correct size for the large problem, and cells have been added to support reporting all three sizes and optional FOMs as additional information.
Q17: Does the working budget range of $155M specified in the RFP for each machine build subcontract include maintenance and support costs?
A17: To help guide Offerors, we present additional details of the expected budgets for the CORAL subcontracts. LLNL expects a total maximum budget of $170M to apply to the CORAL subcontracts. This budget is expected to cover LLNL's share of the NRE subcontracts (amount to be determined), the LLNL Build subcontract, parallel file system, and 5 years of maintenance. ORNL and ANL each expect a maximum budget of $155M to cover their respective Build agreements and their parallel file systems. Maintenance for the ORNL and ANL systems will be paid out of a separate budget, their annual operational budgets. The Office of Science expects to contribute funds for the NRE subcontracts. The amount is expected to be no greater than $30M. All three labs have the option to purchase a parallel file system as part of their respective Build subcontract/agreements. However if a lab chooses to buy the parallel file system from another vendor, the Build subcontract/agreement budget will be decremented by the amount being reserved for the parallel file system.
|Lab/Office||Expected CORAL Budget||Scope Covered by the Budget||Scope Covered by Other Budgets|
|ANL||$155M maximum||Build Contract, parallel file system||5-year maintenance|
|LLNL||$170M maximum||Build Contract, parallel file system, NRE contracts, and 5-year maintenance||–|
|ORNL||$155M maximum||Build Contract, parallel file system||5-year maintenance|
|DOE Office of Science||$30M maximum||NRE contracts||–|
Q18: We're seeing an issue in LCALS in the VOL3D_CALC function with an array bounds overwrite. Is this a bug?
A18: This issue has been fixed in the most recent LCALS source update on the CORAL benchmark page. See the Change Log entry for 01/31 for details.
Q19: Parts of UMT2013 appear to have thread safety issues. Also, MPI+OpenMP versions of UMT2013 built with recent versions of certain compilers will abort.
A19: Three changes were made that added threadprivate() directives to several module variables that will resolve these issues, as well as stabilize the iteration count between variants of the run. The changes are detailed in the updated UMT2013 README file. Because of the late date of this change, Offerers need not recalculate their FOM for UMT2013 but should specify in a note if they do.
Q20: Where do I download the larger NAMD datasets?
A20: They are available via FTP.
password: <your email address>
ftp> cd pub/People/rloy/NAMD-benchmark
250 CWD command successful
229 Entering Extended Passive Mode (|||50662|)
150 Opening ASCII mode data connection for file list
-rw-r----- 1 ftp ftp 6856306605 May 21 2013 namd_test_100M.tar.gz
-rw-r----- 1 ftp ftp 131892992 May 21 2013 namd_test_1M.tar.gz
-rw-r----- 1 ftp ftp 742618100 May 21 2013 namd_test_20M.tar.gz
-rw-r----- 1 ftp ftp 396117013 May 21 2013 namd_test_3M.tar.gz
-rw-r----- 1 ftp ftp 64715247 May 21 2013 namd_test_small.tar.gz
226 Transfer complete
Q21: How do I tell if a NAMD run is successful?
A21: Check for the following:
1. NAMD outputs at least one line containing "Info: Benchmark time"
2. Energy is non-increasing
3. It reaches "End of program"
4. There are no overt error messages
Q22: How do I determine the NAMD FOM?
A22: NAMD benchmark output generally appears after initialization and convergence of the load balancing, when the simulation enters steady-state time-stepping. It is not necessarily at the very end of the output. The "Info: Benchmark time" lines are usually output in groups of 3, the FOM data is taken from the last occurrence of the last group.The FOM is ns/day, the inverse of the "days/ns" field on that line. Using an example from one of the sample data set directories, if these were the "Info: Benchmark time" lines in the output:
Info: Benchmark time: 8192 CPUs 0.0305275 s/step 0.176664 days/ns 130.5 MB memory
Info: Benchmark time: 8192 CPUs 0.0297227 s/step 0.172007 days/ns 130.5 MB memory
Info: Benchmark time: 8192 CPUs 0.0299303 s/step 0.173208 days/ns 130.5 MB memory
then taking the last one, the FOM is (1/0.173208) = 5.77
Q23: Regarding Argonne Master LTO, Section 3, second sentence. Please explain "adjusted for accrual of progress payment." When and why would Argonne make a progress payment prior to acceptance of the equipment?
A23: It would be to make a "down" payment, thus lessening the total amount financed under the lease.
Q24: Regarding Oak Ridge Draft ORNL Lease Agreement, Article II (D) (1), Early Payment, please explain advanced lease payments, and what type "credit" might be expected by ORNL.
A24: UT-Battelle requires the flexibility to make payments that are expected to reduce costs and the term of that lease. As an example, assume that the agreed upon lease terms describe fixed monthly payments over a fixed term. UT-Battelle may choose to pay an amount equal to an entire year of payments as a single payment.; The resulting interest savings would be applied as a credit at the end of the payment schedule. The Lessor must describe (a) any changes to the remaining payment stream, (b) the reduction in total interest paid over the full term of the lease, and (c) the reduction in the term of the lease. The actions described here do not allow for a recapitalization of the lease.
Q25: We expect the performance/benchmark materials for Volume 7 to exceed the capacity of a CD. Would a DVD be acceptable?
A25: Yes, that will be fine.
Q26: Given that the LLNL NRE subcontract will be a firm-fixed-price subcontract with milestone payments, please confirm that the awardee's cost-share component for purposes of the forthcoming DOE patent waiver will be based on the awardee's proposed fair and reasonable price, and will not require future tracking of actual incurred costs by the awardee.
A26: The CORAL NRE subcontracts will be firm-fixed-price with milestone payments. As such, incurred costs will not be a factor in determining payment to the Subcontractor, and the Subcontractor will generally not be required to provide incurred cost information after award. Offerors are advised that, if LLNS determines none of the conditions exist for exemption from certified cost or pricing data, then certified cost or pricing data will be required and the NRE awards would be subject to Cost Accounting Standards (CAS) and customary post-award audit. The 40% Subcontractor cost-share (mentioned in the CORAL RFP letter) determination will be made by LLNS based a review of the successful offeror's good-faith-estimate (i.e., proposal) to perform the NRE work, and any subsequent supporting information LLNS may request prior to award. The Subcontractor will not be required to provide incurred cost information after award for the express purpose of supporting/verifying its initial proposed share in the NRE development expense.
Q27: The tab for MPIBenchmarks in the benchmark results spreadsheet does not clearly indicate where to enter the data. Where should we include the numerical results?
A27: This was an inadvertent oversight. Please insert a column before the "Notes" column to enter the numbers for each row (measurement).